Cell polarity is regulated by evolutionarily conserved polarity factors whose precise higher-order organization at the cell cortex is largely unknown. Here we image frontally the cortex of live fission yeast cells using time-lapse and super-resolution microscopy. Interestingly, we find that polarity factors are organized in discrete cortical clusters resolvable to ~50-100 nm in size, which can form and become cortically enriched by oligomerization. We show that forced co-localization of the polarity factors Tea1 and Tea3 results in polarity defects, suggesting that the maintenance of both factors in distinct clusters is required for polarity. However, during mitosis, their co-localization increases, and Tea3 helps to retain the cortical localization of the Tea1 growth landmark in preparation for growth reactivation following mitosis. Thus, regulated spatial segregation of polarity factor clusters provides a means to spatio-temporally control cell polarity at the cell cortex. We observe similar clusters in Saccharomyces cerevisiae and Caenorhabditis elegans cells, indicating this could be a universal regulatory feature.