CD8+ cytotoxic T lymphocyte (CTL)-mediated immune responses to HIV contribute to viral control in vivo. Epitopes encoded by alternative reading frame (ARF) peptides may be targeted by CTLs as well, but their frequency and in vivo relevance are unknown. Using host genetic (human leukocyte antigen [HLA]) and plasma viral sequence information from 765 HIV-infected subjects, we identified 64 statistically significant (q<0.2) associations between specific HLA alleles and sequence polymorphisms in alternate reading frames of gag, pol, and nef that did not affect the regular frame protein sequence. Peptides spanning the top 20 HLA-associated imprints were used to test for ex vivo immune responses in 85 HIV-infected subjects and showed responses to 10 of these ARF peptides. The most frequent response recognized an HLA-A*03-restricted +2 frame-encoded epitope containing a unique A*03-associated polymorphism at position 6. Epitope-specific CTLs efficiently inhibited viral replication in vitro when viruses containing the wild-type sequence but not the observed polymorphism were tested. Mutating alternative internal start codons abrogated the CTL-mediated inhibition of viral replication. These data indicate that responses to ARF-encoded HIV epitopes are induced during natural infection, can contribute to viral control in vivo, and drive viral evolution on a population level.