The evaluation of the performance of ChatGPT in the management of labor analgesia

Nada Ismaiel; Teresa Phuongtram Nguyen; Nan Guo; Brendan Carvalho; Pervez Sultan; study collaborators

doi:10.1016/j.jclinane.2024.111582

The evaluation of the performance of ChatGPT in the management of labor analgesia

J Clin Anesth. 2024 Nov:98:111582. doi: 10.1016/j.jclinane.2024.111582. Epub 2024 Aug 20.

Authors

Nada Ismaiel¹, Teresa Phuongtram Nguyen², Nan Guo³, Brendan Carvalho⁴, Pervez Sultan⁵; study collaborators

Collaborators

study collaborators:
Anthony Chau⁶, Ronald George⁷, Ashraf Habib⁸, Arvind Palanisamy⁹, Carolyn Weiniger¹⁰, Cynthia Wong¹¹

Affiliations

¹ Department of Anesthesiology, El Camino Health, 2500 Grant Road, Mountain View, California 94040, USA.
² Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, 300 Pasteur Drive, Room H3580, MC 5640, Stanford 94305, CA, USA. Electronic address: [email protected].
³ Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, 300 Pasteur Drive, Room H3580, MC 5640, Stanford 94305, CA, USA. Electronic address: [email protected].
⁴ Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, 300 Pasteur Drive, Room H3580, MC 5640, Stanford 94305, CA, USA. Electronic address: [email protected].
⁵ Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, 300 Pasteur Drive, Room H3580, MC 5640, Stanford 94305, CA, USA. Electronic address: [email protected].
⁶ Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, University of British Columbia, 11228-11(th) Floor, 2775 Laurel Street, Vancouver, British Columbia V5Z 1M9, Canada.
⁷ Department of Anesthesiology and Pain Medicine, Mount Sinai Hospital, University of Toronto, 600 University Avenue, Room 7-405, Toronto, Ontario M5G 1X5, Canada.
⁸ Department of Anesthesiology, Duke University Medical Center, 2301 Erwin Road, 5666 Hafs Building, Durham, NC 27710, USA.
⁹ Department of Anesthesiology, Washington University School of Medicine, 1 Barnes- Jewish Hospital Plaza, St. Louis, MO 63110, USA.
¹⁰ Anesthesiologist, Department of Anesthesiology, Tel Aviv Sourasky Medical Center, Weizmann St 6, Tel Aviv-Yafo, Israel.
¹¹ Department of Anesthesiology, Carver College of Medicine, University of Iowa, 200 Hawkins Drive, Iowa City, IA 52242, USA.

PMID: 39167880
DOI: 10.1016/j.jclinane.2024.111582

Abstract

ChatGPT4 is a leading large language model (LLM) chatbot released by OpenAI in 2023. ChatGPT4 can respond to free-text queries, answer questions and make suggestions regarding virtually any topic. ChatGPT4 has successfully answered anesthesia and even obstetric anesthesia knowledge-based questions with reasonable accuracy. However, ChatGPT4 has yet to be challenged in obstetric anesthesia clinical decision-making.

Study objective: In this study, we evaluated the performance of ChatGPT4 in the management of clinical labor analgesia scenarios compared to expert obstetric anesthesiologists.

Intervention: Eight clinical questions with progressively increasing medical complexity were posed to ChatGPT4.

Measurements: The ChatGPT4 responses were rated by seven expert obstetric anesthesiologists based on safety, accuracy and completeness of each response using a five-point Likert rating scale.

Main results: ChatGPT4 was deemed safe in 73% of responses to the presented obstetric anesthesia clinical scenarios (27% of responses were deemed unsafe). None of the ChatGPT4 responses were unanimously deemed to be safe by all seven expert obstetric anesthesiologists. Moreover, ChatGPT4 responses were overall partly accurate (score 4 out of 5) and somewhat incomplete (score 3.5 out of 5).

Conclusions: In summary, approximately one quarter of all responses by ChatGPT4 were deemed unsafe by expert obstetric anesthesiologists. These findings may suggest the need for more fine-tuning and training of LLMs such as ChatGPT4 specifically for clinical decision making in obstetric anesthesia or other specialized medical fields. These LLMs may come to play an important future role in assisting obstetric anesthesiologists in clinical decision making and enhancing overall patient care.

Keywords: Analgesia; Anesthesiology; ChatGPT4; Chatbot; Obstetric; Safety.

MeSH terms

Analgesia, Obstetrical* / methods
Female
Humans
Labor Pain* / therapy
Machine Learning*
Pain Management* / methods