Comorbid medical illnesses, such as obesity and diabetes, are associated with more severe COVID-19, hospitalization, and death. However, the role of the immune system in mediating these clinical outcomes has not been determined. We used multi-parameter flow cytometry and systems serology to comprehensively profile the functions of T cells and antibodies targeting spike, nucleocapsid, and envelope proteins in a convalescent cohort of COVID-19 subjects who were either hospitalized (n=20) or not hospitalized (n=40). To avoid confounding, subjects were matched by age, sex, ethnicity, and date of symptom onset. Surprisingly, we found that the magnitude and functional breadth of virus-specific CD4 T cell and antibody responses were consistently higher among hospitalized subjects, particularly those with medical comorbidities. However, an integrated analysis identified more coordination between polyfunctional CD4 T-cells and antibodies targeting the S1 domain of spike among subjects that were not hospitalized. These data reveal a functionally diverse and coordinated response between T cells and antibodies targeting SARS-CoV-2 which is reduced in the presence of comorbid illnesses that are known risk factors for severe COVID-19. Our data suggest that isolated measurements of the magnitudes of spike-specific immune responses are likely insufficient to anticipate vaccine efficacy in high-risk populations.