Inter-reader agreement of the PI-QUAL score for prostate MRI quality in the NeuroSAFE PROOF trial

Objectives: The Prostate Imaging Quality (PI-QUAL) score assesses the quality of multiparametric MRI (mpMRI). A score of 1 means all sequences are below the minimum standard of diagnostic quality, 3 implies that the scan is of sufficient diagnostic quality, and 5 means that all three sequences are of optimal diagnostic quality. We investigated the inter-reader reproducibility of the PI-QUAL score in patients enrolled in the NeuroSAFE PROOF trial.

Methods: We analysed the scans of 103 patients on different MR systems and vendors from 12 different hospitals. Two dedicated radiologists highly experienced in prostate mpMRI independently assessed the PI-QUAL score for each scan. Interobserver agreement was assessed using Cohen's kappa with standard quadratic weighting (κw) and percent agreement.

Results: The agreement for each single PI-QUAL score was strong (κw = 0.85 and percent agreement = 84%). A similar agreement (κw = 0.82 and percent agreement = 84%) was observed when the scans were clustered into three groups (PI-QUAL 1-2 vs PI-QUAL 3 vs PI-QUAL 4-5). The agreement in terms of diagnostic quality for each single sequence was highest for T2-weighted imaging (92/103 scans; 89%), followed by dynamic contrast-enhanced sequences (91/103; 88%) and diffusion-weighted imaging (80/103; 78%).

Conclusion: We observed strong reproducibility in the assessment of PI-QUAL between two radiologists with high expertise in prostate mpMRI. At present, PI-QUAL offers clinicians the only available tool for evaluating and reporting the quality of prostate mpMRI in a systematic manner but further refinements of this scoring system are warranted.

Key points: • Inter-reader agreement for each single Prostate Imaging Quality (PI-QUAL) score (i.e., PI-QUAL 1 to PI-QUAL 5) was strong, with weighted kappa = 0.85 (95% confidence intervals: 0.51 - 1) and percent agreement = 84%. • Interobserver agreement was strong when the scans were clustered into three groups according to the ability (or not) to rule in and to rule out clinically significant prostate cancer (i.e., PI-QUAL 1-2 vs PI-QUAL 3 vs PI-QUAL 4-5), with weighted kappa = 0.82 (95% confidence intervals: 0.68 - 0.96) and percent agreement = 84%. • T2-weighted acquisitions were the most compliant with the Prostate Imaging Reporting and Data System (PI-RADS) v. 2.0 technical recommendations and were the sequences of highest diagnostic quality for both readers in 95/103 (92%) scans, followed by dynamic contrast enhanced acquisition with 81/103 (79%) scans and lastly by diffusion-weighted imaging with 79/103 (77%) scans.