Unintended Consequence: Diversity as a Casualty of Eliminating United States Medical Licensing Examination Step 1 Scores

Purpose: The purpose of this study was to use a discrete-choice experiment to model the trade-offs evaluators make between academic attributes and demographics when the United States Medical Licensing Examination (USMLE) Step 1 switches to pass/fail.

Methods: A discrete-choice experiment was administered to faculty members from a geographically diverse mix of 14 academic and community radiology departments in the United States from August through November 2020. Reviewers reviewed 10 applicant pairs with numeric Step 1 scores (part 1) and 10 applicant pairs with a pass Step 1 result (part 2). Applicant attributes included medical school rank, gender, race/ethnicity, USMLE Step 1 score, USMLE Step 2 score, class rank, clerkship honors, and publications. Conditional logistic regression modeled the influence of attribute levels.

Results: Two hundred twelve evaluators completed the study (response rate 59%). The most influential attribute was Step 1 score in part 1 and medical school rank in part 2. The relative importance of race/ethnicity and gender decreased by 25% and 29%, respectively, when Step 1 switches to pass/fail. Evaluators weigh race/ethnicity the strongest when applicants have the same Step 1 score (preference weights of 0.85 for African American, 1.42 for Hispanic, and 0 for White and Asian applicants). Race/ethnicity is relatively more important when Step 1 scores are higher (preference weights of 1.58 for African American, 0.90 for Hispanic, and 0 for White and Asian applicants).

Conclusions: The loss of numeric Step 1 scores reduced the residency evaluator preference for diversity. Reviewers prioritize underrepresented-in-medicine applicants when Step 1 scores are higher and comparable with White and Asian applicants.