Factors that drive prostate cancer progression remain poorly defined, thus hindering the development of new therapeutic strategies. Disseminated tumors are treated through regimens that ablate androgen signaling, as prostate cancer cells require androgen for growth and survival. However, recurrent, incurable tumors that have bypassed the androgen requirement ultimately arise. This study reveals that the Brm ATPase, a component of selected SWI/SNF complexes, has significant antiproliferative functions in the prostate that protect against these transitions. First, we show that targeted ablation of Brm is causative for the development of prostatic hyperplasia in mice. Second, in vivo challenge revealed that Brm-/- epithelia acquire the capacity for lobe-specific, castration-resistant cellular proliferation. Third, investigation of human specimens revealed that Brm mRNA and protein levels are attenuated in prostate cancer. Fourth, Brm down-regulation was associated with an increased proliferative index, consistent with the mouse model. Lastly, gene expression profiling showed that Brm loss alters factors upstream of E2F1; this was confirmed in murine models, wherein Brm loss induced E2F1 deregulation in a tissue-specific manner. Combined, these data identify Brm as a major effector of serum androgen-induced proliferation in the prostate that is disrupted in human disease, and indicate that loss of Brm confers a proliferative advantage in prostate cancer.