Inter- and Intra-rater Reliability of A Grading System for Congenital Diaphragmatic Hernia Defect Size

Background: The Congenital Diaphragmatic Hernia Study Group (CDHSG) registry is a multi-institutional tool to track outcomes of patients with CDH. The CDHSG asks surgeons to categorize diaphragmatic defect sizes as type A-D based on published guidelines. The reported size of the defect has been correlated with patient outcomes, but the reliability of this system has never been studied. Our goal was to evaluate the inter- and intra-rater reliability of the CDHSG grading system.

Materials and methods: Forty-six operative notes from CDH patients that underwent surgical repair at a single institution were collected and cropped to include only the information necessary to grade the hernia defect based on the CDHSG guidelines. The defects were graded by nine pediatric surgeons on two separate occasions (18 wk apart). Inter-rater reliability was calculated using a Cohen's kappa (κ). Intra-rater reliability was calculated using an intraclass correlation coefficient.

Results: Inter-rater reliability was minimal to weak (κ _round1 = 0.395, κ _round2 = 0.424). Agreement ranged from 19.57% (κ = -0.0745) to 82.61% (κ = 0.7543). Inter-rater agreement was similar despite operative findings and outcomes: survival yes/no (κ = 0.3690, κ = 0.3518), need for ECMO yes/no (κ = 0.3323, κ = 0.3362), patch repair yes/no (κ = 0.2050, κ = 0.1916), and liver up/down (κ = 0.2941, κ = 0.4404). Intra-rater reliability was good to excellent (intraclass correlation coefficient = 0.88, 95% CI [0.83-0.92]). Agreement with oneself ranged from 71.74% to 93.48%.

Conclusions: The demonstrated weak inter-rater reliability of the current CDHSG grading system shows the need for improvement in how the grading system is applied by surgeons when reporting CDH defect size.