Chromatin is reprogrammed after fertilization to produce a totipotent zygote with the potential to generate a new organism. The maternal genome inherited from the oocyte and the paternal genome provided by sperm coexist as separate haploid nuclei in the zygote. How these two epigenetically distinct genomes are spatially organized is poorly understood. Existing chromosome conformation capture-based methods are not applicable to oocytes and zygotes owing to a paucity of material. To study three-dimensional chromatin organization in rare cell types, we developed a single-nucleus Hi-C (high-resolution chromosome conformation capture) protocol that provides greater than tenfold more contacts per cell than the previous method. Here we show that chromatin architecture is uniquely reorganized during the oocyte-to-zygote transition in mice and is distinct in paternal and maternal nuclei within single-cell zygotes. Features of genomic organization including compartments, topologically associating domains (TADs) and loops are present in individual oocytes when averaged over the genome, but the presence of each feature at a locus varies between cells. At the sub-megabase level, we observed stochastic clusters of contacts that can occur across TAD boundaries but average into TADs. Notably, we found that TADs and loops, but not compartments, are present in zygotic maternal chromatin, suggesting that these are generated by different mechanisms. Our results demonstrate that the global chromatin organization of zygote nuclei is fundamentally different from that of other interphase cells. An understanding of this zygotic chromatin 'ground state' could potentially provide insights into reprogramming cells to a state of totipotency.