Hybrid rice breeding using cytoplasmic male sterility/fertility restoration (CMS/Rf) systems plays an important role in ensuring global food security. Two backcross inbred line (BIL) populations derived from either Xieqingzao B (XB)//XB/Dongxiang wild rice (DWR) (XXD) or XB//DWR/XB (XDX) were used to detect quantitative trait loci (QTLs) for fertility restoration of Dwarf wild abortive- (DA-), Indonesia Paddy- (ID-), and DWR-type CMS in rice. Lines with ID- and DA-type CMS were testcrossed with both the XXD- and XDX-BILs, while the line with DWR-type CMS was testcrossed with the XDX-BILs only. A total of 16 QTLs for fertility restoration of CMS systems were identified, including three for DWR-type CMS, six for DA-type CMS, and seven for ID-type CMS. All of the additive alleles in the QTLs were derived from Oryza rufipogon. Eleven QTLs were clustered in five chromosomal regions, indicating that common Rf loci restored different CMS systems, and the favorable O. rufipogon alleles could be used to develop restorer lines for various CMS types by marker-assisted selection.