Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification: Recommendations for the Future

Saman Andalib; Sean S Solomon; Bryce G Picton; Aidin C Spina; John A Scolaro; Ariana M Nelson

doi:10.2106/JBJS.OA.24.00007

Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification: Recommendations for the Future

JB JS Open Access. 2025 Jan 8;10(1):e24.00007. doi: 10.2106/JBJS.OA.24.00007. eCollection 2025 Jan-Mar.

Authors

Saman Andalib¹, Sean S Solomon¹, Bryce G Picton¹, Aidin C Spina¹, John A Scolaro², Ariana M Nelson³

Affiliations

¹ University of California, Irvine, School of Medicine, Irvine, California.
² Department of Orthopaedic Surgery, University of California, Irvine, Medical Center, Orange, California.
³ Department of Anesthesiology, University of California, Irvine, Medical Center, Orange, California.

Abstract

Background: This study assesses the effectiveness of large language models (LLMs) in simplifying complex language within orthopaedic patient education materials (PEMs) and identifies predictive factors for successful text transformation.

Methods: We transformed 48 orthopaedic PEMs using GPT-4, GPT-3.5, Claude 2, and Llama 2. The readability, quantified by the Flesch-Kincaid Reading Ease (FKRE) and Flesch-Kincaid Grade Level (FKGL) scores, was measured before and after transformation. Analysis included text characteristics such as syllable count, word length, and sentence length. Statistical and machine learning methods evaluated the correlations and predictive capacity of these features for transformation success.

Results: All LLMs improved FKRE and FKGL scores (p < 0.01). GPT-4 showed superior performance, transforming PEMs to a seventh-grade reading level (mean FKGL, 6.72 ± 0.99), with higher FKRE and lower FKGL than other models. GPT-3.5, Claude 2, and Llama 2 significantly shortened sentences and overall text length (p < 0.01). Importantly, correlation analysis revealed that transformation success varied substantially with the model used, depending on original text factors such as word length and sentence complexity.

Conclusions: LLMs successfully simplify orthopaedic PEMs, with GPT-4 leading in readability improvement. This study highlights the importance of initial text characteristics in determining the effectiveness of LLM transformations, offering insights for optimizing orthopaedic health literacy initiatives using artificial intelligence (AI).

Clinical relevance: This study provides critical insights into the ability of LLMs to simplify complex orthopaedic PEMs, enhancing their readability without compromising informational integrity. By identifying predictive factors for successful text transformation, this research supports the application of AI in improving health literacy, potentially leading to better patient comprehension and outcomes in orthopaedic care.