Precision Structuring of Free-Text Surgical Record for Enhanced Stroke Management: A Comparative Evaluation of Large Language Models

Introduction: Mechanical thrombectomy (MTB) is a critical procedure for acute ischemic stroke (AIS) patients. However, the free-text format of MTB surgical records limits the formulation of effective postoperative patient management and rehabilitation plans. This study compares the efficacy of large language models (LLMs) in structuring data from these free-text MTB surgical record.

Methods: This retrospective study collected a total of 382 MTB surgical records from a tertiary hospital. An initial analysis of 30 surgical record from these records provided a guiding prompt for LLMs, focusing on basic and advanced characteristics, such as occlusion locations, thrombectomy maneuvers, reperfusion status, and intraoperative complications. Six LLMs-ChatGPT, GPT-4, GeminiPro, ChatGLM4, Spark3, and QwenMax-were assessed against data extracted by neuroradiologists and a junior physician for comparison. The all 382 surgical records were used to test the performance of LLMs. The performance of the LLMs was quantified using Accuracy, Sensitivity, Specificity, AUC, and MSE as an additional metric for advanced characteristics.

Results: All LLMs showed high performance in characteristic extraction, achieving an average accuracy of 95.09 ± 4.98% across 48 items, and 78.05 ± 4.2% overall. GLM4 and GPT-4 were most accurate in advanced characteristics extraction, with accuracies of 84.03% and 82.20%, respectively. The processing time for LLMs averaged 73.10 ± 10.86 seconds of six models, significantly faster than the 427.88 seconds for manual extraction by physicians.

Conclusion: LLMs, particularly GLM4 and GPT-4, efficiently and accurately structured both general and advanced characteristics from MTB surgical record, outperforming manual extraction methods and demonstrating potential for enhancing clinical data management in AIS treatment.