Introduction: In problem-based medical curricula, consideration should be given to the level of difficulty of patient cases used for training and assessment. The Amsterdam Clinical Challenge Scale (ACCS) has been developed to assess the degree of difficulty of patient cases in a systematic and reproducible manner. To determine the reliability of the instrument two research questions were addressed: (1) How many judges are required, on the basis of the total score of the ACCS, to obtain a reliable estimate of the difficulty of a single case? (2) How many cases and/or how many judges are needed to reach an acceptable level of reliability of the total score of the ACCS?
Method: Four judges scored 36 patient scripts reflecting a wide range of patient problems encountered in general practice. Each script was scored four times. In the reliability analysis, the generalizability theory was applied.
Results: The results show that the judges did, indeed, use the whole range of difficulty ratings. When the ACCS is applied to a single case, eight or more judges are needed to reach an acceptable level of reliability. When more cases are involved, fewer judges are needed; for 10 or more cases one judge will be sufficient.
Conclusions: Given the typical length, for example of an objective structured clinical examination, the ACCS makes it possible to provide a reliable estimate of the level of difficulty of such a test with only a limited number of judges.