ROC Curve for Extremely Subtle Lung Nodules on Chest Radiographs Confirmed by CT Scan

Rationale and objectives: The aim of the present study was to test the hypothesis that when a radiologist does not perceive an abnormality in images that contain either extremely subtle abnormalities or no abnormalities, the radiologist cannot distinguish these two types of images and the receiver operating characteristic (ROC) curve reflects that performance.

Materials and methods: This retrospective study was conducted with approval of our institutional review board. Four general radiologists participated in an observer performance study of 100 chest images, each of which had a 5 × 5 cm region of interest (ROI) drawn (50 containing a lung nodule, and 50 did not, based on computed tomography [CT] confirmation). About half of the lung nodules were extremely subtle. The readers reported their confidence that a nodule was present within the ROI, from which empirical and maximum-likelihood "proper" binormal and conventional binormal ROC curves were estimated. The readers also reported whether they saw an abnormality that could be a nodule within the ROI.

Results: Empirical ROC curves deviated from typical ROC-curve shapes, and a portion of the curve leading to the northeast corner of the ROC space had relatively steep and constant slopes. The readers reported not seeing anything suggestive of a lung nodule in this portion of the ROC curve, which also corresponded to cases that either contained extremely subtle nodules or normal cases. The average area under the ROC curves (mean ± standard deviation) was 0.66 ± 0.02 for proper binormal, 0.62 ± 0.02 for conventional binormal, and 0.60 ± 0.03 for trapezoidal ROC curves.

Conclusions: When a radiologist does not perceive an abnormality in images that contain either extremely subtle abnormalities or no abnormalities, the ROC curve (or a portion thereof) is characterized by a straight line, which is not consistent with conventional ROC theories.