A Pilot Study to Gauge Effectiveness of Standardized Patient Scenarios in Assessing General Surgery Milestones

Purpose: Some General Surgery Milestones can be difficult to assess in traditional clinical settings and especially difficult to assess in junior residents. The purpose of this pilot study was to METHODS: A total of 9 categorical interns participated in a comprehensive, 4-module, SP scenario designed to evaluate and manage right upper quadrant pain. SP checklist scores (SP%) were converted to Milestone-equivalent scores for direct comparison (SP-C). Milestone scores were analyzed from 3 different sources: SP, faculty (FAC), and CCC. Interns completed course evaluations at the end of each session. Spearman's rho was used to determine correlations. Wilcoxon signed rank tests were used to test for differences between scores from different sources.

Results: Individual intern Milestone scores from the 3 sources (SP-C, FAC, and CCC) did not correlate. All 7 mean Milestone scores from SPs were significantly higher than from FAC and CCC. FAC and CCC scores were statistically equivalent except for Systems-Based Practice 1 (SBP1) and Patient Care 3 (PC3) where CCC scores were significantly higher than FAC. Mean SP% scores for PC1 were significantly lower than for PROF1, MK1, MK2, and ICS1 (p < 0.05). Interns felt the modules were moderately to very useful.

Conclusions: Developing an SP scenario for Milestones evaluation is feasible. SPs, faculty observers, and CCC each use different data to provide a unique source of Milestone assessment. SP scenarios may be ideally suited to assess specific resident strengths and weaknesses and provide individualized feedback, thus augmenting traditional evaluations. Additional SP scenarios, assessing a broader range of skills and Milestones, are advisable for more reliable estimates of resident performance.