Memory B cells (MBCs) generate rapid antibody responses upon secondary encounter with a pathogen. Here, we investigated the kinetics, avidity and cross-reactivity of serum antibodies and MBCs in 155 SARS-CoV-2 infected and vaccinated individuals over a 16-month timeframe. SARS-CoV-2-specific MBCs and serum antibodies reached steady-state titers with comparable kinetics in infected and vaccinated individuals. Whereas MBCs of infected individuals targeted both pre- and postfusion Spike (S), most vaccine-elicited MBCs were specific for prefusion S, consistent with the use of prefusion-stabilized S in mRNA vaccines. Furthermore, a large fraction of MBCs recognizing postfusion S cross-reacted with human betacoronaviruses. The avidity of MBC-derived and serum antibodies increased over time resulting in enhanced resilience to viral escape by SARS-CoV-2 variants, including Omicron BA.1 and BA.2 sub-lineages, albeit only partially for BA.4 and BA.5 sublineages. Overall, the maturation of high-affinity and broadly-reactive MBCs provides the basis for effective recall responses to future SARS-CoV-2 variants.