Previous studies suggest that enzymes involved in the androgen metabolic pathway are susceptibility factors for prostate cancer. Estrogen metabolites functioning as genotoxins have also been proposed as risk factors. In this study, we systematically tested the hypothesis that common genetic variations for those enzymes involved in the androgen and estrogen metabolic pathways increase risk for sporadic and familial prostate cancer. From these two pathways, 46 polymorphisms (34 single nucleotide polymorphisms, 10 short tandem repeat polymorphisms, and 2 null alleles) in 25 genes were tested for possible associations. Those genes tested included PRL, LHB, CYP11A1, HSD3B1, HSD3B2, HSD17B2, CYP17, SRD5A2, AKR1C3, UGT2B15, AR, SHBG, and KLK3 from the androgen pathway and CYP19, HSD17B1, CYP1A1, CYP1A2, CYP1B1, COMT, GSTP1, GSTT1, GSTM1, NQO1, ESR1, and ESR2 from the estrogen pathway. A case-control study design was used with two sets of cases: familial cases with a strong prostate cancer family history (n = 438 from 178 families) and sporadic cases with a negative prostate cancer family history (n = 499). The controls (n = 493) were derived from a population-based collection. Our results provide suggestive findings for an association with either familial or sporadic prostate cancer with polymorphisms in four genes: AKR1C3, HSD17B1, NQO1, and GSTT1. Additional suggestive findings for an association with clinical variables (disease stage, grade, and/or node status) were observed for single nucleotide polymorphisms in eight genes: HSD3B2, SRD5A2, SHBG, ESR1, CYP1A1, CYP1B1, GSTT1, and NQO1. However, none of the findings were statistically significant after appropriate corrections for multiple comparisons. Given that the point estimates for the odds ratio for each of these polymorphisms are <2.0, much larger sample sizes will be required for confirmation.