In conjunction with a pooled analysis of risk factors for advanced adenomas [adenomas with severe dysplasia, carcinoma in situ (CIS), and intramucosal carcinoma], we undertook a reliability study on the pathological diagnosis of advanced adenomas. We assessed intraobserver agreement (using Kappa (kappa) as the measure of agreement) across two time periods 10 years apart with a single pathologist and interobserver agreement (using Kappa) between two pathologists rating the same slides concurrently. The study pathologists were blinded to the original case classification. We used the slides of 190 colorectal adenomatous polyp cases (104 originally diagnosed as advanced adenomas, 86 adenomas without advanced lesions) from a colonoscopy-based case-control study conducted in New York City between 1986 and 1988. We also assessed conditional agreement for 71 slides of advanced adenomas from four adenoma case-control studies conducted in different geographic regions of the United States in the 1990s. Intra- and interobserver agreement was only fair to moderate on the classification of both histological type (villous, tubulovillous, and tubular: intraobserver kappa = 0.28; 95% confidence interval (CI), 0.17-0.39; interobserver kappa = 0.48; 95% CI, 0.33-0.62) and degree of dysplasia (none/mild, moderate, severe, CIS, and intramucosal: intraobserver kappa = 0.20; 95% CI, 0.12-0.28; interobserver kappa = 0.42; 95% CI, 0.29-0.55). Using broader, rather than finer, classifications for degree of dysplasia substantially improved the reliability (interobserver agreement for high-grade dysplasia (including severe dysplasia, CIS, and intramucosal carcinoma) versus low-grade dysplasia: kappa = 0.69; 95% CI, 0.55-0.83). These findings suggest that future epidemiological studies of advanced adenomas should use broad categories, such as high-grade versus low-grade dysplasia, include central review of all slides, and take measurement error into account in sample size calculations.