Although T cells are likely players in SARS-CoV-2 immunity, little is known about the phenotypic features of SARS-CoV-2-specific T cells associated with recovery from severe COVID-19. We analyzed T cells from longitudinal specimens of 34 COVID-19 patients with severities ranging from mild (outpatient) to critical culminating in death. Relative to patients that succumbed, individuals that recovered from severe COVID-19 harbored elevated and increasing numbers of SARS-CoV-2-specific T cells capable of homeostatic proliferation. In contrast, fatal COVID-19 displayed elevated numbers of SARS-CoV-2-specific regulatory T cells and a time-dependent escalation in activated bystander CXCR4+ T cells. Together with the demonstration of increased proportions of inflammatory CXCR4+ T cells in the lungs of severe COVID-19 patients, these results support a model whereby lung-homing T cells activated through bystander effects contribute to immunopathology, while a robust, non-suppressive SARS-CoV-2-specific T cell response limits pathogenesis and promotes recovery from severe COVID-19.