Development and Validation of Electronic Health Record Measures of Safety Planning Practices as Part of Zero Suicide Implementation

Objective: Safety planning for suicide prevention is an important quality metric for Zero Suicide implementation. We describe the development, validation, and application of electronic health record (EHR) programs to measure uptake of safety planning practices across six integrated healthcare systems as part of a Zero Suicide evaluation study.

Methods: Safety planning was documented in narrative notes and structured EHR templates using the Stanley Brown Safety Planning Intervention (SBSPI) in response to a high-risk cutoff score on the Columbia Suicide Severity Rating Scale (CSSRS). Natural Language Processing (NLP) metrics were developed and validated using chart review to characterize practices documented in narrative notes. We applied NLP to measure frequency of documentation in the narrative text and standard programming methods to examine structured SBSPI templates from 2010-2022.

Results: Chart reviews found three safety planning practices documented in narrative notes that were delivered to at least half of patients at risk: professional contacts, lethal means counseling for firearms, and lethal means counseling for medication access/storage. NLP methods were developed to identify these practices in clinical text with high levels of accuracy (Sensitivity, Specificity, & PPV ≥ 82%). Among visits with a high-risk CSSRS, 40% (Range 2-73% by health system) had an SBSPI template within 1 year of implementation.

Conclusions: This is one of the first reports describing development of measures that leverage electronic health records to track use of suicide prevention safety plans. There are opportunities to use the methods developed here in future evaluations of safety planning.