Large language models can accurately populate Vascular Quality Initiative procedural databases using narrative operative reports

Objectives: Participation in the Vascular Quality Initiative (VQI) provides important resources to surgeons, but the ability to do so is often limited by time and data entry personnel. Large language models (LLMs) such as ChatGPT (OpenAI) are examples of generative artificial intelligence (AI) products that may help bridge this gap. Trained on large volumes of data, the models are used for natural language processing (NLP) and text generation. We evaluated the ability of LLMs to accurately populate VQI procedural databases using operative reports.

Methods: A single-center retrospective study was performed using institutional VQI data from 2021-2023. The most recent procedures for carotid endarterectomy (CEA), endovascular aneurysm repair (EVAR) and infrainguinal lower extremity bypass (LEB) were analyzed using Versa, a HIPAA-compliant institutional version of ChatGPT. We created an automated function to analyze operative reports and generate a shareable VQI file using two models: gpt-35-turbo and gpt-4. Application of the LLMs was accomplished with a cloud-based programming interface. The outputs of this model were compared to VQI data for accuracy. We defined a metric as "unavailable" to the LLM if it was discussed by surgeons in <20% of operative reports. unavailable.

Results: 150 operative notes were analyzed, including 50 CEA, 50 EVAR, and 50 LEB. These procedural VQI databases included 25, 179, and 51 metrics, respectively. For all fields, gpt-35-turbo had a median accuracy of 84.0% for CEA (IQR 80.0-88.0%), 92.2% for EVAR (IQR 87.2-94.0%), and 84.3% for LEB (IQR 80.2-88.1%). There were 3 of 25, 6 of 179, and 7 of 51 VQI variables were unavailable in the operative reports, respectively. Excluding metric information routinely unavailable in operative reports, the median accuracy rate was 95.5% for each CEA procedure (IQR 90.9-100.0%), 94.8% (IQR 92.2-98.5) for EVAR, and 93.2% for LEB (IQR 90.2-96.4%). Across procedures, gpt-4 did not meaningfully improve performance compared to gpt-35 (p=0.97, 0.85, 0.95 for CEA, EVAR, LEB overall performance). The cost for 150 operative reports analyzed with gpt-35-turbo and gpt-4 were $0.12 and $3.39, respectively.

Conclusion: LLMs can accurately populate VQI procedural databases with both structured and unstructured data, while incurring only minor processing costs. Increased workflow efficiency may improve center ability to successfully participate in the VQI. Further work examining other VQI databases and methods to increase accuracy are needed.