Intra- and interobserver agreement in evaluation of image-defined risk factors on computed tomography in pediatric neuroblastoma

Background: Image-defined risk factors (IDRFs) were introduced to provide a consensus approach for pre-treatment risk stratification on computed tomography (CT) and magnetic resonance imaging (MRI) in patients with neuroblastoma.

Objective: To assess the intra- and inter-reader agreement of radiologists in identifying IDRFs on CT.

Materials and methods: Approval for this retrospective study was granted by our institutional research ethics board with a waiver of consent. CT studies of pediatric patients with neuroblastoma were assessed by seven radiologists in two rounds. Each CT was accompanied by a standard form to indicate presence or absence of IDRFs for each patient. At least a 4-week period between rounds, randomization, and relabeling of the CT studies was required to minimize recall bias. Finally, three of the seven radiologists conducted a subsequent consensus reading to determine true positive IDRFs in the cohort. Fleiss' kappa statistic was used to evaluate readers' agreements and Pearson's correlation assessed the correlation between years of experiences of radiologists and their performance in accurately (intra-reader agreement) detecting IDRFs.

Results: A total of 31 children with a median age of 2.1 years (interquartile range (IQR) 1.1, 3.0; range 0-18; male=21 (67.7%)) were included and 251 total positive IDRFs were identified on their CT scans. The location of the primary tumor was in the neck in 1 patient (3.2%), within the chest in 2 patients (6.5%), within the abdomen in 27 patients (87.1%), and in the pelvis in 1 patient (3.2%). In determining IDRFs, the inter-reader agreement among radiologists was substantial: 0.65 (95% CI 0.60, 0.69) and the intra-reader agreement for each radiologist was substantial to near perfect, ranging from 0.67 (0.60, 0.70) to 0.86 (0.82, 0.90). The correlation between the number of years of experience of radiologists and their performances in accurately detecting IDRFs (their intra-reader agreements) was respectively low (r=45, P=0.30) for abdomino-pelvic IDRFs and high for organs' infiltration IDRFs (r=0.74, P=0.05).

Conclusion: The determination of IDRFs on CT is reproducible with significant agreement among radiologists. The two IDRF items with the lowest overall inter-reader agreements were "diaphragm infiltration" and "mesenteric infiltration."