Evaluating AI-Generated informed consent documents in oral surgery: A comparative study of ChatGPT-4, Bard gemini advanced, and human-written consents

Luigi Angelo Vaira; Jerome R Lechien; Antonino Maniaci; Giuseppe Tanda; Vincenzo Abbate; Fabiana Allevi; Antonio Arena; Giada Anna Beltramini; Michela Bergonzani; Alessandro Remigio Bolzoni; Salvatore Crimi; Andrea Frosolini; Guido Gabriele; Fabio Maglitto; Miguel Mayo-Yáñez; Ludovica Orrù; Marzia Petrocelli; Resi Pucci; Alberto Maria Saibene; Stefania Troise; Alessandro Tel; Valentino Vellone; Carlos Miguel Chiesa-Estomba; Paolo Boscolo-Rizzo; Giovanni Salzano; Giacomo De Riu

doi:10.1016/j.jcms.2024.10.002

Evaluating AI-Generated informed consent documents in oral surgery: A comparative study of ChatGPT-4, Bard gemini advanced, and human-written consents

J Craniomaxillofac Surg. 2024 Oct 25:S1010-5182(24)00283-X. doi: 10.1016/j.jcms.2024.10.002. Online ahead of print.

Authors

Luigi Angelo Vaira¹, Jerome R Lechien², Antonino Maniaci³, Giuseppe Tanda⁴, Vincenzo Abbate⁵, Fabiana Allevi⁶, Antonio Arena⁵, Giada Anna Beltramini⁷, Michela Bergonzani⁸, Alessandro Remigio Bolzoni⁹, Salvatore Crimi¹⁰, Andrea Frosolini¹¹, Guido Gabriele¹¹, Fabio Maglitto¹², Miguel Mayo-Yáñez¹³, Ludovica Orrù⁴, Marzia Petrocelli¹⁴, Resi Pucci¹⁵, Alberto Maria Saibene¹⁶, Stefania Troise⁵, Alessandro Tel¹⁷, Valentino Vellone¹⁸, Carlos Miguel Chiesa-Estomba¹⁹, Paolo Boscolo-Rizzo²⁰, Giovanni Salzano²¹, Giacomo De Riu²²

Affiliations

¹ Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Sassari, Italy; PhD School of Biomedical Science, Biomedical Sciences Department, University of Sassari, Sassari, Italy. Electronic address: [email protected].
² Department of Anatomy and Experimental Oncology, Mons School of Medicine, UMONS. Research Institute for Health Sciences and Technology, University of Mons (UMons), Mons, Belgium; Department of Otolaryngology-Head Neck Surgery, Elsan Polyclinic of Poitiers, Poitiers, France.
³ Department of Medicine and Surgery, University of Enna Kore, Enna, Italy.
⁴ Dental School, Department of Medicine, Surgery and Pharmacy, University of Sassari, Sassari, Italy.
⁵ Head and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy.
⁶ Maxillofacial Surgery Department, ASSt Santi Paolo e Carlo, University of Milan, Milan, Italy.
⁷ Department of Biomedical, Surgical and Dental Sciences, University of Milan, Milan, Italy; Maxillofacial and Dental Unit, Fondazione IRCCS Cà Granda Ospedale Maggiore Policlinico, Milan, Italy.
⁸ Maxillo-Facial Surgery Division, Head and Neck Department, University Hospital of Parma, Parma, Italy.
⁹ Maxillofacial and Dental Unit, Fondazione IRCCS Cà Granda Ospedale Maggiore Policlinico, Milan, Italy.
¹⁰ Department of CHIRMED. Maxillofacial Surgery Section, University of Catania, Catania, Italy.
¹¹ Maxillofacial Surgery Unit, Department of Medical Biotechnologies, University of Siena, Siena, Italy.
¹² Maxillo-Facial Surgery Unit, University of Bari "Aldo Moro", Bari, Italy.
¹³ Otorhinolaryngology, Head and Neck Surgery Department, Complexo Hospitalario Universitario A Coruña (CHUAC), A Coruña, Galicia, Spain.
¹⁴ Maxillofacial Surgery Operative Unit, Bellaria and Maggiore Hospital, Bologna, Italy.
¹⁵ Maxillofacial Surgery Unit, San Camillo-Forlanini Hospital, Rome, Italy.
¹⁶ Otolaryngology Unit, Santi Paolo e Carlo Hospital, Department of Health Sciences, University of Milan, Milan, Italy.
¹⁷ Clinic of Maxillofacial Surgery, Department of Head & Neck Surgery and Neuroscience, University Hospital of Udine, Italy.
¹⁸ Maxillofacial Surgery Unit, "S. Maria" Hospital, Terni, Italy.
¹⁹ Department of Otorhinolaryngology-Head & Neck Surgery, Hospital Universitario Donostia, San Sebastian, Spain.
²⁰ Department of Medical, Surgical and Health Sciences, Section of Otolaryngology, University of Trieste, Trieste, Italy.
²¹ Head and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy; Maxillo-Facial Surgery Unit, University of Bari "Aldo Moro", Bari, Italy.
²² Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Sassari, Italy.

PMID: 39490345
DOI: 10.1016/j.jcms.2024.10.002

Abstract

This study evaluates the quality and readability of informed consent documents generated by AI platforms ChatGPT-4 and Bard Gemini Advanced compared to those written by a first-year oral surgery resident for common oral surgery procedures. The evaluation, conducted by 18 experienced oral and maxillofacial surgeons, assessed consents for accuracy, completeness, readability, and overall quality. ChatGPT-4 consistently outperformed both Bard and human-written consents. ChatGPT-4 consents had a median accuracy score of 4 [IQR 4-4], compared to Bard's 3 [IQR 3-4] and human's 4 [IQR 3-4]. Completeness scores were higher for ChatGPT-4 (4 [IQR 4-5]) than Bard (3 [IQR 3-4]) and human (4 [IQR 3-4]). Readability was also superior for ChatGPT-4, with a median score of 4 [IQR 4-5] compared to Bard and human consents, both at 4 [IQR 4-4] and 4 [IQR 3-4], respectively. The Gunning Fog Index for ChatGPT-4 was 17.2 [IQR 16.5-18.2], better than Bard's 23.1 [IQR 20.5-24.7] and the human consents' 20 [IQR 19.2-20.9]. Overall, ChatGPT-4's consents received the highest quality ratings, underscoring AI's potential in enhancing patient communication and the informed consent process. The study suggests AI can reduce misinformation risks and improve patient understanding, but continuous evaluation, oversight, and patient feedback integration are crucial to ensure the effectiveness and appropriateness of AI-generated content in clinical practice.

Keywords: AI in healthcare; Artificial intelligence; Consent accuracy; Document quality; Informed consent; Large language models; Maxillofacial surgery; Oral surgery; Patient education.