Evaluating ChatGPT's Multilingual Performance in Clinical Nutrition Advice Using Synthetic Medical Text: Insights from Central Asia

Background: While large language models like ChatGPT-4 have demonstrated competency in English, their performance for minority groups speaking underrepresented languages, as well as their ability to adapt to specific socio-cultural nuances and regional cuisines, such as those in Central Asia (e.g., Kazakhstan), still requires further investigation.

Objective: To evaluate and compare the effectiveness of the ChatGPT-4 system in providing personalized, evidence-based nutritional recommendations in English, Kazakh, and Russian in Central Asia.

Methods: This study was conducted from May 15 to August 31, 2023. Based on fifty mock patient profiles, ChatGPT-4 generated dietary advice, and responses were evaluated for personalization, consistency, and practicality using a 5-point Likert scale. To identify significant differences between the three languages, the Kruskal Wallis Test was conducted. Additional pairwise comparisons for each language were carried out using the post-hoc Dunn's Test.

Results: ChatGPT-4 showed a moderate level of performance in each category for English and Russian languages while in Kazakh language outputs were unsuitable for evaluation. The scores for English, Russian, and Kazakh were as follows: for personalization, 3.32±0.46, 3.18±0.38, and 1.01±0.06; for consistency, 3.48±0.43, 3.38±0.39, and 1.09±0.18; and for practicality, 3.25±0.41, 3.37±0.38, and 1.07±0.15, respectively. The Kruskal-Wallis test indicated statistically significant differences in ChatGPT-4's performance across the three languages (P<0.001). Subsequent post hoc analysis using Dunn's test showed that the performance in both English and Russian was significantly different from that in Kazakh.

Conclusions: Our findings show that, despite using identical prompts across three distinct languages, the ChatGPT-4 's capability to produce sensible outputs is limited by the lack of training data in non-English languages. Thus, a customized LLM should be developed to perform better in underrepresented languages and to take into account specific local diets and practices.