Transcriptional silencing of HIV in CD4 T cells generates a reservoir of latently infected cells that can reseed infection after interruption of therapy. As such, these cells represent the principal barrier to curing HIV infection, but little is known about their characteristics. To further our understanding of the molecular mechanisms of latency, we characterized a primary cell model of HIV latency in which infected cells adopt heterogeneous transcriptional fates. In this model, we observed that latency is a stable, heritable state that is transmitted through cell division. Using Assay of Transposon-Accessible Chromatin sequencing (ATACseq) we found that latently infected cells exhibit greatly reduced proviral accessibility, indicating the presence of chromatin-based structural barriers to viral gene expression. By quantifying the activity of host cell transcription factors, we observe elevated activity of Forkhead and Kruppel-like factor transcription factors (TFs), and reduced activity of AP-1, RUNX and GATA TFs in latently infected cells. Interestingly, latency reversing agents with different mechanisms of action caused distinct patterns of chromatin reopening across the provirus. We observe that binding sites for the chromatin insulator CTCF are highly enriched in the differentially open chromatin of infected CD4 T cells. Furthermore, depletion of CTCF inhibited HIV latency, identifying this factor as playing a key role in the initiation or enforcement of latency. These data indicate that HIV latency develops preferentially in cells with a distinct pattern of TF activity that promotes a closed proviral structure and inhibits viral gene expression. Furthermore, these findings identify CTCF as a novel regulator of HIV latency.