Analyzing the Effectiveness of AI-Generated Patient Education Materials: A Comparative Study of ChatGPT and Google Gemini

Objective: The study aims to compare ChatGPT and Google Gemini-generated patient education guides regarding claustrophobia during MRI, mammography screening, and MR safe and unsafe items and the importance of knowing what items can be carried into an MR room.

Methods: The study utilized ChatGPT 3.5 and Google Gemini to create patient education guides concerning claustrophobia during MRI, mammography screening, and MR safe and unsafe items. A Flesch-Kincaid calculator was used to evaluate readability and ease of understanding. QuillBot (QuillBot, Inc., Chicago, USA) was used to generate a similarity score to evaluate possible plagiarism. In order to assess the scientific reliability of the AI-generated responses, we utilized a modified DISCERN score. R Studio 4.3.2 (The R Foundation for Statistical Computing, Vienna, Austria) was used for statistical analyses, with unpaired t-tests used to determine statistical significance between variables.

Results: The average number of words in ChatGPT and Google Gemini were 468.7±132.07 and 328.7±163.65, respectively. The mean number of sentences was 35.67±18.15 for ChatGPT and 30.33±12.22 for Google Gemini. Ease of readability for ChatGPT responses was 36.30±7.01 and for Google Gemini 46.77±4.96. The similarity scores for the ChatGPT responses were 0.50±0.62 and for Google Gemini 9.43±6.20. The reliability score was evaluated at 2.67±0.25 for ChatGPT and 2.67± 0.58 for Google Gemini.

Conclusion: The AI generated by ChatGPT and Google Gemini had no statistically significant difference in regard to word count, average word per sentence, average syllables per word, grade level comprehension score, or scientific reliability. However, the ease score was significantly greater for the ChatGPT response compared to Google Gemini. In addition, the similarity score was much higher in Google Gemini than in ChatGPT responses.