Following integration, HIV-1 in most cases produces active infection events; however, in some rare instances, latent infection events are established. The latter have major clinical implications, as latent infection allows the virus to persist despite antiretroviral therapy. Both the cellular factors and the viral elements that potentially determine whether HIV-1 establishes active or latent infection events remain largely elusive. We detail here the contribution of different long terminal repeat (LTR) sequences for the establishment of latent HIV-1 infection. Using a panel of full-length replication-competent virus constructs that reflect naturally occurring differences of HIV-1 subtype-specific LTRs and targeted LTR mutants, we found the primary ability of HIV-1 to establish latent infection in this system to be controlled by a four-nucleotide (nt) AP-1 element just upstream of the NF-κB element in the viral promoter. Deletion of this AP-1 site mostly deprived HIV-1 of the ability to establish latent HIV-1 infection. Extension of this site to a 7-nt AP-1 sequence massively promoted latency establishment, suggesting that this promoter region represents a latency establishment element (LEE). Given that these minimal changes in a transcription factor binding site affect latency establishment to such large extent, our data support the notion that HIV-1 latency is a transcription factor restriction phenomenon.