Matrine is an alkaloid extracted from a Chinese herb Sophora flavescens Ait, which has shown chemopreventive potential against various cancers. In this study, we evaluated the anticancer efficacy of a novel derivative of matrine, (6aS, 10S, 11aR, 11bR, 11cS)-10- methylamino-dodecahydro- 3a,7a-diazabenzo (de) (MASM), against human hepatocellular carcinoma (HCC) cells and their corresponding sphere cells in vitro and in vivo. Human HCC cell lines (Hep3B and Huh7) were treated with MASM. Cell proliferation was assessed using CCK8 and colony assays; cell apoptosis and cell cycle distributions were examined with flow cytometry. The expression of cell markers and signaling molecules was detected using Western blot and qRT-PCR analyses. A sphere culture technique was used to enrich cancer stem cells (CSC) in Hep3B and Huh7 cells. The in vivo antitumor efficacy of MASM was evaluated in Huh7 cell xenograft model in BALB/c nude mice, which were administered MASM (10 mg·kg-1·d-1, ig) for 3 weeks. After the treatment was completed, tumor were excised and weighed. A portion of tumor tissue was enzymatically dissociated to obtain a single cell suspension for the spheroid formation assays. MASM (2, 10, 20 μmol/L) dose-dependently inhibited the proliferation of HCC cells, and induced apoptosis, which correlated with a reduction in Bcl-2 expression and an increase in PARP cleavage. MASM also induced cell cycle arrest in G0/G1 phase, which was accompanied by increased p27 and decreased Cyclin D1 expression. Interestingly, MASM (2, 10, and 20 μmol/L) drastically reduced the EpCAM+/CD133+ cell numbers, suppressed the sphere formation, inhibited the expression of stem cell marker genes and promoted the expression of mature hepatocyte markers in the Hep3B and Huh7 spheroids. Additionally, MASM dose-dependently suppressed the PI3K/AKT/mTOR and AKT/GSK3β/β-catenin signaling pathways in Hep3B and Huh7 cells. In Huh7 xenograft bearing nude mice, MASM administration significantly inhibited Huh7 xenograft tumor growth and markedly reduced the number of surviving cancer stem-like cells in the tumors. MASM administration also reduced the expression of stem cell markers while increasing the expression of mature hepatocyte markers in the tumor tissues. The novel derivative of matrine, MASM, markedly suppresses HCC tumor growth through multiple mechanisms, and it may be a promising candidate drug for the treatment of hepatocellular carcinoma.