Can Artificial Intelligence Improve the Readability of Patient Education Materials on Aortic Stenosis? A Pilot Study

Armaun D Rouhi; Yazid K Ghanem; Laman Yolchieva; Zena Saleh; Hansa Joshi; Matthew C Moccia; Alejandro Suarez-Pierre; Jason J Han

doi:10.1007/s40119-023-00347-0

Can Artificial Intelligence Improve the Readability of Patient Education Materials on Aortic Stenosis? A Pilot Study

Cardiol Ther. 2024 Mar;13(1):137-147. doi: 10.1007/s40119-023-00347-0. Epub 2024 Jan 9.

Authors

Armaun D Rouhi¹, Yazid K Ghanem², Laman Yolchieva³, Zena Saleh², Hansa Joshi², Matthew C Moccia², Alejandro Suarez-Pierre⁴, Jason J Han⁵

Affiliations

¹ Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
² Department of Surgery, Cooper University Hospital, Camden, NJ, USA.
³ College of Arts and Sciences, University of Pennsylvania, Philadelphia, PA, USA.
⁴ Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA.
⁵ Division of Cardiovascular Surgery, Department of Surgery, Perelman School of Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA, USA. [email protected].

Abstract

Introduction: The advent of generative artificial intelligence (AI) dialogue platforms and large language models (LLMs) may help facilitate ongoing efforts to improve health literacy. Additionally, recent studies have highlighted inadequate health literacy among patients with cardiac disease. The aim of the present study was to ascertain whether two freely available generative AI dialogue platforms could rewrite online aortic stenosis (AS) patient education materials (PEMs) to meet recommended reading skill levels for the public.

Methods: Online PEMs were gathered from a professional cardiothoracic surgical society and academic institutions in the USA. PEMs were then inputted into two AI-powered LLMs, ChatGPT-3.5 and Bard, with the prompt "translate to 5th-grade reading level". Readability of PEMs before and after AI conversion was measured using the validated Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook Index (SMOGI), and Gunning-Fog Index (GFI) scores.

Results: Overall, 21 PEMs on AS were gathered. Original readability measures indicated difficult readability at the 10th-12th grade reading level. ChatGPT-3.5 successfully improved readability across all four measures (p < 0.001) to the approximately 6th-7th grade reading level. Bard successfully improved readability across all measures (p < 0.001) except for SMOGI (p = 0.729) to the approximately 8th-9th grade level. Neither platform generated PEMs written below the recommended 6th-grade reading level. ChatGPT-3.5 demonstrated significantly more favorable post-conversion readability scores, percentage change in readability scores, and conversion time compared to Bard (all p < 0.001).

Conclusion: AI dialogue platforms can enhance the readability of PEMs for patients with AS but may not fully meet recommended reading skill levels, highlighting potential tools to help strengthen cardiac health literacy in the future.

Keywords: Aortic stenosis; Artificial intelligence; ChatGPT; Chatbots; Health literacy; Heart valve disease; Large language models; Patient education material; Readability.