Cancer patients have increased morbidity and mortality from Coronavirus Disease 2019 (COVID-19), but the underlying immune mechanisms are unknown. In a cohort of 100 cancer patients hospitalized for COVID-19 at the University of Pennsylvania Health System, we found that patients with hematologic cancers had a significantly higher mortality relative to patients with solid cancers after accounting for confounders including ECOG performance status and active cancer status. We performed flow cytometric and serologic analyses of 106 cancer patients and 113 non-cancer controls from two additional cohorts at Penn and Memorial Sloan Kettering Cancer Center. Patients with solid cancers exhibited an immune phenotype similar to non-cancer patients during acute COVID-19 whereas patients with hematologic cancers had significant impairment of B cells and SARS-CoV-2-specific antibody responses. High dimensional analysis of flow cytometric data revealed 5 distinct immune phenotypes. An immune phenotype characterized by CD8 T cell depletion was associated with a high viral load and the highest mortality of 71%, among all cancer patients. In contrast, despite impaired B cell responses, patients with hematologic cancers and preserved CD8 T cells had a lower viral load and mortality. These data highlight the importance of CD8 T cells in acute COVID-19, particularly in the setting of impaired humoral immunity. Further, depletion of B cells with anti-CD20 therapy resulted in almost complete abrogation of SARS-CoV-2-specific IgG and IgM antibodies, but was not associated with increased mortality compared to other hematologic cancers, when adequate CD8 T cells were present. Finally, higher CD8 T cell counts were associated with improved overall survival in patients with hematologic cancers. Thus, CD8 T cells likely compensate for deficient humoral immunity and influence clinical recovery of COVID-19. These observations have important implications for cancer and COVID-19-directed treatments, immunosuppressive therapies, and for understanding the role of B and T cells in acute COVID-19.