We assessed the correlation between a self-report questionnaire and an expert-rating including an initial interview and a longitudinal evaluation on the diagnosis of binge eating disorder (BED) in a sample of 100 obese women participating in a treatment program for weight reduction. The level of diagnostic agreement between patient-rating and expert-rating with regard to the presence or absence of BED was modest, with a kappa value of .57. According to Shrout, Spitzer, and Fleiss (Archives of General Psychiatry, 44, 172-177, 1987) this represents fair to good agreement beyond chance. The self-report instrument did not produce higher estimates of the frequency of BED in this selected sample of treatment seekers than the expert-rating, as observed in studies on the epidemiology of bulimia nervosa in community samples. The questionnaire identified 40 cases of BED, the expert-rating 43 cases. The results indicate that the disagreement between self-report and interview was mainly due to discordances in three of the diagnostic criteria of BED--namely loss of control, marked distress regarding binge eating, and the frequency requirement of two binge eating episodes per week for a 6-month period. Inconsistencies between subjects and clinicians with regard to the definition of an overeating episode and with regard to the behavioral indicators of loss of control did not lead to differences between self-report and observer-rating in the final diagnosis of BED.