Jiedu Xiaozheng Yin decoction (JXY) is a type of Chinese traditional medicine, which has been used to treat various types of cancer. The present study explored the mechanisms underlying the anticancer activity of JXY. The effects of ethyl acetate extraction of JXY (EE-JXY) were evaluated on the HepG2 human hepatoma cell line in vitro and in vivo. Following treatment of the HepG2 cells with EE-JXY for 24 h, cell viability, apoptosis, mitochondrial membrane potential, caspase enzyme activity and the expression levels of apoptotic-associated proteins (Bcl-2 and Bax) were detected by MTT, flow cytometry, ELISA and western blotting respectively. In addition, HepG2 cells were subcutaneously transplanted into BALB/c nude mice, and the tumor bearing mice were treated with either EE-JXY (0.06 g/kg) or normal saline for 21 days. Tumor volume and weight were measured and recorded. The apoptotic index, and the expression levels of Bax and cytochrome c were determined with immunohistochemical staining. Treatment with EE-JXY inhibited the proliferation of HepG2 cells, and reduced cell viability in a dose-and time-dependent manner. Furthermore, EE-JXY induced HepG2 cell apoptosis, as demonstrated by a loss of plasma membrane asymmetry and externalization of phosphatidylserine, collapse of mitochondrial membrane potential, activation of caspase-9 and caspase-3, and an increased ratio of pro-apoptotic Bax to anti-apoptotic Bcl-2. Furthermore, EE-JXY inhibited tumor growth and increased the apoptotic index of tumors in tumor-bearing mice. In conclusion, the results of the present study suggest that JXY inhibits HepG2 cell proliferation through mitochondrion-mediated apoptosis, which may partially explain its anticancer activity.