Evaluation of online chat-based artificial intelligence responses about inflammatory bowel disease and diet

Introduction: The USA has the highest age-standardized prevalence of inflammatory bowel disease (IBD). Both genetic and environmental factors have been implicated in IBD flares and multiple strategies are centered around avoiding dietary triggers to maintain remission. Chat-based artificial intelligence (CB-AI) has shown great potential in enhancing patient education in medicine. We evaluate the role of CB-AI in patient education on dietary management of IBD.

Methods: Six questions evaluating important concepts about the dietary management of IBD which then were posed to three CB-AI models - ChatGPT, BingChat, and YouChat three different times. All responses were graded for appropriateness and reliability by two physicians using dietary information from the Crohn's and Colitis Foundation. The responses were graded as reliably appropriate, reliably inappropriate, and unreliable. The expert assessment of the reviewing physicians was validated by the joint probability of agreement for two raters.

Results: ChatGPT provided reliably appropriate responses to questions on dietary management of IBD more often than BingChat and YouChat. There were two questions that more than one CB-AI provided unreliable responses to. Each CB-AI provided examples within their responses, but the examples were not always appropriate. Whether the response was appropriate or not, CB-AIs mentioned consulting with an expert in the field. The inter-rater reliability was 88.9%.

Discussion: CB-AIs have the potential to improve patient education and outcomes but studies evaluating their appropriateness for various health conditions are sparse. Our study showed that CB-AIs have the ability to provide appropriate answers to most questions regarding the dietary management of IBD.