We lack the understanding of why HIV-infected individuals in South Africa progress to AIDS. We hypothesised that in end-stage disease there is a shifting dynamic between T cell imposed immunity and viral immune escape, which, through both compensatory and reverting viral mutations, results in increased viral fitness, elevated plasma viral loads and disease progression. We explored how T cell responses, viral adaptation and viral fitness inter-relate in South African cohorts recruited from Bloemfontein, the Free State (n = 278) and Durban, KwaZulu-Natal (n = 775). Immune responses were measured by γ-interferon ELISPOT assays. HLA-associated viral polymorphisms were determined using phylogenetically corrected techniques, and viral replication capacity (VRC) was measured by comparing the growth rate of gag-protease recombinant viruses against recombinant NL4-3 viruses. We report that in advanced disease (CD4 counts <100 cells/µl), T cell responses narrow, with a relative decline in Gag-directed responses (p<0.0001). This is associated with preserved selection pressure at specific viral amino acids (e.g., the T242N polymorphism within the HLA-B*57/5801 restricted TW10 epitope), but with reversion at other sites (e.g., the T186S polymorphism within the HLA-B*8101 restricted TL9 epitope), most notably in Gag and suggestive of "immune relaxation". The median VRC from patients with CD4 counts <100 cells/µl was higher than from patients with CD4 counts ≥ 500 cells/µl (91.15% versus 85.19%, p = 0.0004), potentially explaining the rise in viral load associated with disease progression. Mutations at HIV Gag T186S and T242N reduced VRC, however, in advanced disease only the T242N mutants demonstrated increasing VRC, and were associated with compensatory mutations (p = 0.013). These data provide novel insights into the mechanisms of HIV disease progression in South Africa. Restoration of fitness correlates with loss of viral control in late disease, with evidence for both preserved and relaxed selection pressure across the HIV genome. Interventions that maintain viral fitness costs could potentially slow progression.