Context: The unbiased selection of images representing a spectrum of diagnostic difficulty is an important first step in designing effective assessment and teaching interventions for X-ray interpretation.
Objectives: This study aimed to develop a scale that would reliably differentiate more difficult X-rays from those that are easier to interpret.
Methods: After pilot testing, an X-ray difficulty scale (XRDS) was developed. Raters of different learner levels from two universities were presented with 20 chest X-rays (CXRs) and asked to read them and then to answer the scale questions that would help to differentiate the level of difficulty of interpretation of each film. Reliability of the scale was evaluated. Face validity of the scale was assessed and the construct validity of two hypotheses was tested.
Results: The final scale consisted of five questions in which a given X-ray could score from--10 (most difficult) to + 10 (easiest to interpret) by a single rater. Raters included 53 medical students, 10 paediatric residents and 10 emergency staff. The scale demonstrated excellent internal consistency (r = 0.94), inter-rater reliability (r = 0.95) and overall reliability (r = 0.90) in medical students. Construct validity testing demonstrated good correlation (r = 0.72) between diagnostic accuracy and mean XRDS score. Mean scores on the scale were significantly lower (indicating that CXRs were more difficult to interpret) for students than for resident and staff doctors (P < 0.0001).
Conclusions: The scale developed in this study serves as a reliable and valid tool for categorising CXRs according to diagnostic difficulty, which reduces the bias inherent in the process of selecting radiographs by expert opinion alone.