Prolonged culture of embryonic stem cells (ESCs) leads them to adopt embryonal carcinoma cell features, creating enormous dangers for their further application. The mechanism involved in ESC stability has not, however, been extensively studied. We previously reported that SMAD family member 3 (Smad3) has an important role in maintaining mouse ESC stability, as depletion of Smad3 results in cancer cell-like properties in ESCs and Smad3-/- ESCs are prone to grow large, malignant teratomas. To understand how Smad3 contributes to ESC stability, we performed microarray analysis to compare the transcriptome of wild-type and Smad3-/- ESCs. We found that Rif1 (RAP1-associated protein 1), a factor important for genomic stability, is significantly upregulated in Smad3-/- ESCs. The expression level of Rif1 needs to be tightly controlled in ESCs, as a low level of Rif1 is associated with ESC differentiation, but a high level of Rif1 is linked to ESC transformation. In ESCs, Oct4 activates Rif1, whereas Smad3 represses its expression. Oct4 recruits Smad3 to bind to Rif1 promoter, but Smad3 joining facilitates the loading of a polycomb complex that generates a repressive epigenetic modification on Rif1 promoter, and thus maintains the expression of Rif1 at a proper level in ESCs. Interestingly, Rif1 short hairpin RNA (shRNA)-transduced Smad3-/- ESCs showed less malignant properties than the control shRNA-transduced Smad3-/- ESCs, suggesting a critical role of Rif1 in maintaining the stability of ESCs during proliferation.