A novel resident physician examination using clinical simulation video to assess clinical competence in Japan: a cross-sectional study

Background: The general medicine in-training examination (GM-ITE) assesses physicians' clinical knowledge. This study expanded on findings from a previous pilot study to assess the relationship between general medicine in-training examination (GM-ITE) scores and the diagnostic skills of resident physicians in Japan by employing an innovative clinical simulation video (CSV-IE).

Methods: This multicenter cross-sectional study included 4,677 resident physicians who took the GMITE between January 17 and 30, 2023. Participants watched the CSV-IE, depicting an emergency room scenario, and provided a diagnosis. The CSV-IE depicts an emergency case and provides a diagnosis. Discrimination indices were used to assess the CSV-IE's effectiveness across clinical competence domains, and multilevel logistic regression was used to analyze physician- and hospital-level factors associated with correct diagnoses.

Results: Correct diagnoses were provided by 470 participants (10.0%). The CSV-IE demonstrated high discriminatory power across all assessed domains, including basic clinical knowledge (DI = 0.44), symptomatology and clinical reasoning (DI = 0.31), physical examination and clinical procedure (DI = 0.35), and knowledge about the disease (DI = 0.25), supporting its utility as an effective assessment tool. In the multivariable analysis, factors associated with a higher likelihood of providing a correct CSV-IE diagnosis included a higher annual number of emergency outpatients (adjusted odds ratio: 1.025; 95% confidence interval [CI]: 1.003-1.047; P = .0230) and being in a higher postgraduate year (adjusted odds ratio: 1.387; 95% CI: 1.104-1.742; P = .005). Conversely, resident physicians at university hospitals were less likely to provide a correct CSV-IE response (adjusted odds ratio: 0.624; 95% CI: 0.435-0.896; P = .0107).

Conclusions: CSV-IE modules may provide an integrative and realistic evaluation of clinical competence, addressing limitations of traditional MCQ-based assessments by offering contextualized, real-world scenarios that require dynamic decision-making and diagnostic reasoning.