Does one size fit all? Developing an evaluation strategy to assess large language models for patient safety event report analysis

Objective: Collecting and analyzing patient safety event (PSE) reports is a key component to the improvement of patient safety yet report analysis has been challenging. Large language models (LLMs) may support analysis; however, PSE reports tend to be a hybrid of clinical and general language.

Materials and methods: We propose a data-driven evaluation strategy to assess LLM fit for report analysis. We identify target tokens and sentences from PSE reports and use perplexity to evaluate four LLMs comprehension of the target sentence.

Results: LLMs had statistically significantly different perplexity measures in six of seven event categories. Clinical models perform better with clinical narratives, often reported by nurses and physicians. General models perform better with colloquial language and communication themes.

Discussion and conclusion: For LLMs to support PSE report analysis there must be a good fit between the language model and the nature of the text in reports. A single LLM approach may not be the most useful strategy.