Assessment of a Large-Scale Peer Learning Program's Value by Manual Review of Case Submissions

Objective: Prior studies used submission numbers or report addendum rates to measure peer learning programs' (PLP) impact. We assessed the educational value of a PLP by manually reviewing cases submitted to identify factors correlating with meaningful learning opportunities (MLOs).

Methods: This institutional review board-exempted, retrospective study was performed in a large academic radiology department generating >800,000 reports annually. A PLP facilitating radiologist-to-radiologist feedback was implemented May 1, 2017, with subsequent pay-for-performance initiatives encouraging increasing submissions, >18,000 by 2019. Two radiologists blinded to submitter and receiver identity categorized 336 randomly selected submissions as a MLO, not meaningful, or equivocal, resolving disagreements in consensus review. Primary outcome was proportion of MLOs. Secondary outcomes included percent engagement by subspecialty clinical division and comparing MLO and report addendum rates via Fisher's exact tests. We assessed association between peer learning category, pay-for-performance interventions, and subspecialty division with MLOs using logistic regression.

Results: Of 336 PLP submissions, 65.2% (219 of 336) were categorized as meaningful, 27.4% (92 of 336) not meaningful, and 7.4% (25 of 336) equivocal, with substantial reviewer agreement (86.0% [289 of 336], κ = 0.71, 95% confidence interval 0.64-0.78). MLO rate (65.2% [219 of 336]) was five times higher than addendum rate (12.9% [43 of 333]) for the cohort. MLO proportion (adjusted odds ratios 0.05-1.09) and percent engagement (0.5%-3.6%) varied between subspecialty divisions, some submitting significantly fewer MLOs (P < .01). MLO proportion did not vary between peer learning categories.

Conclusion: Educational value of a large-scale PLP, estimated through manual review of case submissions, is likely a more accurate measure of program impact. Incentives to enhance PLP use did not diminish the program's educational value.