Can all roads lead to competency? School levels effects in Licensing examinations scores

At the foundation of research concerned with professional training is the idea of an assumed causal chain between the policies and practices of education and the eventual behaviours of those that graduate these programs. In medicine, given the social accountability to ensure that teaching and learning gives way to a health human resource that is willing and able to provide the healthcare that patients and communities need, it is of critical importance to generate evidence regarding this causal relationship. One question that medical education scholars ask regularly is the degree to which the unique features of training programs and learning environments impact trainee achievement of the intended learning outcomes. To date, this evidence has been difficult to generate because data pertaining to learners is only rarely systematically brought together across institutions or periods of training. We describe new research which leverages an inter-institutional data-driven approach to investigate the influence of school-level factors on the licensing outcomes of medical students. Specifically, we bring together sociodemographic, admissions, and in-training assessment variables pertaining to medical trainee graduates at each of the six medical schools in Ontario, Canada into multilevel stepwise regression models that determine the degree of association between these variables and graduate performances on the Medical Council of Canada Qualifying Examinations (Part 1, n = 1097 observations; Part 2, n = 616 observations), established predictors of downstream physician performance. As part of this analysis, we include an anonymized school-level (School 1, School 2) independent variable in each of these models. Our results demonstrate that the largest variable associated with performance on both the first and second parts of the licensing examinations is prior academic achievement, notably clerkship performance. Ratings of biomedical knowledge were also significantly associated with the first examination, while clerkship OSCE scores and enrollment in a family medicine residency were significantly associated with the Part 2. Small significant school effects were realized in both models accounting for 4% and 2% of the variance realized in the first and second examinations, respectively. These findings highlight that school enrollment plays a minor role relative to individual student performance in influencing examination outcomes.