Diagnosis, treatment, and prevention of ankle sprains: Comparing free chatbot recommendations with clinical guidelines

Friederike Eva Roch; Franziska Melanie Hahn; Katharina Jäckle; Marc-Pascal Meier; Hartmut Stinus; Wolfgang Lehmann; Ronny Perthel; Paul Jonathan Roch

doi:10.1016/j.fas.2024.12.003

Diagnosis, treatment, and prevention of ankle sprains: Comparing free chatbot recommendations with clinical guidelines

Foot Ankle Surg. 2024 Dec 13:S1268-7731(24)00267-4. doi: 10.1016/j.fas.2024.12.003. Online ahead of print.

Authors

Friederike Eva Roch¹, Franziska Melanie Hahn², Katharina Jäckle³, Marc-Pascal Meier⁴, Hartmut Stinus⁵, Wolfgang Lehmann⁶, Ronny Perthel⁷, Paul Jonathan Roch⁸

Affiliations

¹ Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].
² Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].
³ Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].
⁴ Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].
⁵ Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].
⁶ Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].
⁷ Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].
⁸ Department of Trauma Surgery, Orthopaedics and Plastic Surgery, University of Göttingen, Robert-Koch-Str. 40, Göttingen 37075, Germany. Electronic address: [email protected].

PMID: 39730224
DOI: 10.1016/j.fas.2024.12.003

Abstract

Background: Free chatbots powered by large language models offer lateral ankle sprains (LAS) treatment recommendations but lack scientific validation.

Methods: The chatbots-Claude, Perplexity, and ChatGPT-were evaluated by comparing their responses to a questionnaire and their treatment algorithms against current clinical guidelines. Responses were graded on accuracy, conclusiveness, supplementary information, and incompleteness, and evaluated individually and collectively, with a 60 % pass threshold.

Results: The collective analysis of the questionnaire showed Perplexity scored significantly higher than Claude and ChatGPT (p < 0.001). In the individual analysis, Perplexity provided significantly more supplementary information than the other chatbots (p < 0.001). All chatbots met the pass threshold. In the algorithm evaluation, ChatGPT scored significantly higher than the others (p = 0.023), with Perplexity below the pass threshold.

Conclusions: Chatbots' recommendations generally aligned with current guidelines but sometimes missed crucial details. While they offer useful supplementary information, they cannot yet replace professional medical consultation or established guidelines.

Keywords: ChatGPT; Claude; Lateral ankle sprains; Perplexity; artificial intelligence (AI); chatbots; treatment recommendations.