XRCC4 plays a crucial role in the non-homologous end joining pathway that maintains genome stability. In this two-stage case-control study with 1,764 non-BRCA1/2 breast cancer patients and 1,623 cancer-free controls, we investigated the contribution of genetic variants of XRCC4 to breast cancer susceptibility in Chinese women. We identified a recessive missense variant, rs3734091 (c.739G>T, p.Ala247Ser), of XRCC4 that was significantly associated with an increased risk of breast cancer (odds ratio [OR] = 3.92, P = 0.007), particularly with the risk of developing triple-negative breast cancer (OR = 18.65, P < 0.0001). This p.Ala247Ser variant disturbed the nuclear localization of XRCC4 in cells homozygous for the rs3734091-T allele but not in heterozygous cells at both the cellular and tissue levels. In heterozygous cells, wild-type XRCC4 facilitated the nuclear localization of the XRCC4A247S mutant, thus compensating for the impaired localization of XRCC4A247S. This provided a biological mechanism by which rs3734091 conferred an increased susceptibility to non-BRCA1/2 breast cancer exclusively under a recessive model. Further functional analyses revealed that p.Ala247Ser impaired the DNA damage repair capacity and ultimately perturbed genomic stability. Taken together, our findings document the role of XRCC4 in non-BRCA1/2 breast cancer predisposition and reveal its underlying biological mechanism of action.