Accumulating evidence suggests that breast cancer metastatic progression is modified by germline polymorphism, although specific modifier genes have remained largely undefined. In the current study, we employ the MMTV-PyMT transgenic mouse model and the AKXD panel of recombinant inbred mice to identify AT-rich interactive domain 4B (Arid4b; NM_194262) as a breast cancer progression modifier gene. Ectopic expression of Arid4b promoted primary tumor growth in vivo as well as increased migration and invasion in vitro, and the phenotype was associated with polymorphisms identified between the AKR/J and DBA/2J alleles as predicted by our genetic analyses. Stable shRNA-mediated knockdown of Arid4b caused a significant reduction in pulmonary metastases, validating a role for Arid4b as a metastasis modifier gene. ARID4B physically interacts with the breast cancer metastasis suppressor BRMS1, and we detected differential binding of the Arid4b alleles to histone deacetylase complex members mSIN3A and mSDS3, suggesting that the mechanism of Arid4b action likely involves interactions with chromatin modifying complexes. Downregulation of the conserved Tpx2 gene network, which is comprised of many factors regulating cell cycle and mitotic spindle biology, was observed concomitant with loss of metastatic efficiency in Arid4b knockdown cells. Consistent with our genetic analysis and in vivo experiments in our mouse model system, ARID4B expression was also an independent predictor of distant metastasis-free survival in breast cancer patients with ER+ tumors. These studies support a causative role of ARID4B in metastatic progression of breast cancer.