Performance of ChatGPT-3.5 and ChatGPT-4o in the Japanese National Dental Examination

Osamu Uehara; Tetsuro Morikawa; Fumiya Harada; Nodoka Sugiyama; Yuko Matsuki; Daichi Hiraki; Hinako Sakurai; Takashi Kado; Koki Yoshida; Yukie Murata; Hirofumi Matsuoka; Toshiyuki Nagasawa; Yasushi Furuichi; Yoshihiro Abiko; Hiroko Miura

doi:10.1002/jdd.13766

Performance of ChatGPT-3.5 and ChatGPT-4o in the Japanese National Dental Examination

J Dent Educ. 2024 Nov 13. doi: 10.1002/jdd.13766. Online ahead of print.

Authors

Osamu Uehara¹, Tetsuro Morikawa², Fumiya Harada³, Nodoka Sugiyama⁴, Yuko Matsuki⁵, Daichi Hiraki⁶, Hinako Sakurai⁷, Takashi Kado^{4

8}, Koki Yoshida², Yukie Murata^{1

5}, Hirofumi Matsuoka¹, Toshiyuki Nagasawa^{4

5}, Yasushi Furuichi⁸, Yoshihiro Abiko², Hiroko Miura¹

Affiliations

¹ Division of Disease Control and Molecular Epidemiology, Department of Oral Growth and Development, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.
² Division of Oral Medicine and Pathology, Department of Human Biology and Pathophysiology, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.
³ Division of Oral and Maxillofacial Surgery, Department of Human Biology and Pathophysiology, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.
⁴ Division of Periodontology and Endodontology, Department of Oral Rehabilitation, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.
⁵ Division of Advanced Clinical Education, Department of Integrated Dental Education, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.
⁶ Division of Reconstructive Surgery for Oral and Maxillofacial Region, Department of Human Biology and Pathophysiology, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.
⁷ Division of General Dental Sciences, Department of Oral Rehabilitation, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.
⁸ Division of Dental Education Development, Department of Integrated Dental Education, School of Dentistry, Health Sciences University of Hokkaido, Ishikari-gun, Hokkaido, Japan.

PMID: 39538434
DOI: 10.1002/jdd.13766

Abstract

Objectives: In this study, we compared the performance of ChatGPT-3.5 to that of ChatGPT-4o in the context of the Japanese National Dental Examination, which assesses clinical reasoning skills and dental knowledge, to determine their potential usefulness in dental education.

Methods: ChatGPT's performance was assessed using 1399 (55% of the exam) of 2520 questions from the Japanese National Dental Examinations (111-117). The 1121 excluded questions (45% of the exam) contained figures or tables that ChatGPT could not recognize. The questions were categorized into 18 different subjects based on dental specialty. Statistical analysis was performed using SPSS software, with McNemar's test applied to assess differences in performance.

Results: A significant improvement was noted in the percentage of correct answers from ChatGPT-4o (84.63%) compared with those from ChatGPT-3.5 (45.46%), demonstrating enhanced reliability and subject knowledge. ChatGPT-4o consistently outperformed ChatGPT-3.5 across all dental subjects, with significant improvements in subjects such as oral surgery, pathology, pharmacology, and microbiology. Heatmap analysis revealed that ChatGPT-4o provided more stable and higher correct answer rates, especially for complex subjects.

Conclusions: This study found that advanced natural language processing models, such as ChatGPT-4o, potentially have sufficiently advanced clinical reasoning skills and dental knowledge to function as a supplementary tool in dental education and exam preparation.

Keywords: Chat Generative Pre‐trained Transformer (ChatGPT); Japanese National Dental Examination; artificial intelligence; dental education; natural language processing.