SARS-CoV-2 infection elicits a robust B cell response, resulting in the generation of long-lived plasma cells and memory B cells. Here, we aimed to determine the effect of COVID-19 severity on the memory B cell response and characterize changes in the memory B cell compartment between recovery and five months post-symptom onset. Using high-parameter spectral flow cytometry, we analyzed the phenotype of memory B cells with reactivity against the SARS-CoV-2 spike protein or the spike receptor binding domain (RBD) in recovered individuals who had been hospitalized with non-severe (n=8) or severe (n=5) COVID-19. One month after symptom onset, a substantial proportion of spike-specific IgG + B cells showed an activated phenotype. In individuals who experienced non-severe disease, spike-specific IgG + B cells showed increased expression of markers associated with durable B cell memory, including T-bet, FcRL5, and CD11c, which was not observed after severe disease. Five months post-symptom onset, the majority of spike-specific memory B cells had a resting phenotype and the percentage of spike-specific T-bet + IgG + memory B cells decreased to baseline levels. Collectively, our results suggest that the memory B cell response elicited during non-severe COVID-19 may be of higher quality than the response after severe disease.