Automated Plan Checking Software Demonstrates Continuous and Sustained Improvements in Safety and Quality: A 3-year Longitudinal Analysis

Purpose: This study aimed to perform a longitudinal analysis of the performance of our automated plan checking software by retrospectively evaluating the number of errors identified in plans delivered to patients in 3, month-long, data collection periods between 2017 and 2020.

Methods and materials: Eleven automated checks were retrospectively run on 1169 external beam radiation therapy treatment plans identified as meeting the following criteria: planning target volume-based multifield photon plans receiving a status of treatment approved in March 2017, March 2018, or March 2020. The number of passes (true positives) and flags were recorded. Flags were subcategorized into false negatives, false negatives due to naming conventions, and true negatives. In addition, 2 × 2 contingency tables using a 2-tailed Fisher's exact test were used to determine whether there were nonrandom associations between the output of the automated plan checking software and whether the check was manual or automated at the original time of treatment approval.

Results: A statistically significant decrease in flags between the pre- and postautomation data sets was observed for 4 contour-based checks, namely adjacent structures overlap, empty structures and missing slices, overlap between body and couch, and laterality, as well as a check that determined whether the plan's global maximum dose was within the planning target volume. A review of the origins of false negatives was fed back into the design of the checks to improve the reliability of the system and help avoid warning fatigue.

Conclusions: Periodic and longitudinal review of the performance of automated software was essential for monitoring and understanding its impact on error rates, as well as for optimization of the tool to adapt to regular changes of clinical practice. The automated plan checking software has demonstrated continuous contributions to the safe and effective delivery of external beam radiation therapy to our patient population, an impact that extends beyond its initial implementation and deployment.