The mechanisms of action of the anticancer agent perillyl alcohol (POH), presently in Phase II clinical trials, were investigated in advanced rat mammary carcinomas. Gross and ultrastructural morphology of POH-mediated tumor regression indicated that apoptosis accounted for the marked reduction in the epithelial compartment. Characterization of cell growth and death indices revealed that apoptosis was induced within 48 h of chemotherapy, before the induction of cytostasis. RNA expression studies, based on a multiplexed-nuclease protection assay, demonstrated that cell cycle- and apoptosis-related genes were differentially expressed within 48 h of POH treatment; p21(Cip1/WAF1), bax, bad, and annexin I were induced; cyclin E and cyclin-dependent kinase 2 were repressed; and bcl-2 and p53 were unchanged. Next, a potential role for transforming growth factor beta (TGF-beta) signaling in POH-mediated carcinoma regression was explored. RNA expression studies, again based on a multiplexed-nuclease protection assay, showed that TGF-beta-related genes were induced and temporally regulated during POH treatment: (a) c-jun and c-fos were transiently induced within 12 h of chemotherapy; (b) TGF-beta1 was induced within 24 h of chemotherapy; (c) the mannose 6-phosphate/insulin-like growth factor II receptor and the TGF-beta type I and II receptors were induced within 48 h of chemotherapy; and (d) smad3 was induced during active carcinoma regression. In situ protein expression studies, based on fluorescence-immunohistochemistry in concert with confocal microscopy, confirmed up-regulation and demonstrated colocalization of TGF-beta1, the mannose 6-phosphate/insulin-like growth factor II receptor, the TGF-beta type I and II receptors, and Smad2/Smad3 in epithelial cells. Nuclear localization of Smad2/Smad3 indicated that the TGF-beta signaling pathway was activated in regressing carcinomas. Subpopulations of Smad2/Smad3-positive and apoptotic nuclei colocalized, indicating a role for Smads in apoptosis. Thus, Smads may serve as a potential biomarker for anticancer activity. Importantly, none of the POH-mediated anticancer activities were observed in normal mammary gland.