Repressive chromatin modifications are thought to compact chromatin to silence transcription. However, it is unclear how chromatin structure changes during silencing and epigenetic memory formation. We measured gene expression and chromatin structure in single cells after recruitment and release of repressors at a reporter gene. Chromatin structure is heterogeneous, with open and compact conformations present in both active and silent states. Recruitment of repressors associated with epigenetic memory produces chromatin compaction across 10-20 kilobases, while reversible silencing does not cause compaction at this scale. Chromatin compaction is inherited, but changes molecularly over time from histone methylation (H3K9me3) to DNA methylation. The level of compaction at the end of silencing quantitatively predicts epigenetic memory weeks later. Similarly, chromatin compaction at the Nanog locus predicts the degree of stem-cell fate commitment. These findings suggest that the chromatin state across tens of kilobases, beyond the gene itself, is important for epigenetic memory formation.