The overwhelming majority of biodiversity harbored within microbial communities is represented among their large proportion of rare taxa. We know almost nothing about these microorganisms aside from a small sequence from their ribosomal RNA genes. However, there is accumulating evidence that rare taxa sometimes contribute to community stability by rapidly responding to environmental changes. Given the ongoing stressors of global changes on our planet, there is a critical need to determine the specific roles that rare taxa play for community stability. Here, we aim to extend knowledge of what often is investigated as a static property (rarity and prevalence) towards its true dynamic nature. We discuss a statistical method for uncovering dynamics of persistent, rare taxa that occasionally become more prominent in their communities ("conditionally rare"). Using this method, we quantified conditionally rare taxa in time series from a variety of ecosystems. We discovered that conditionally rare taxa were present in all habitats, and that they disproportionately contributed to temporal changes in diversity when they were most abundant. We offer a case study in deciphering the ecology of conditionally rare taxa, informed by time series observations before, during and after an ecosystem manipulation. Our results suggest that persistent but rare microbial taxa contribute to high within-sample (alpha) diversity, and help to maintain community stability after disturbance.