Impact of virtual vs. in-person interviews among neurosurgery residency applicants

Background: The interview is considered a key factor in selecting residents in various medical and surgical specialties. However, the reliability of the interview process in selecting neurosurgery training program applicants remains largely under-investigated.

Objective: To investigate the reliability of the interview process for neurosurgery residency applicants and to evaluate the impact of virtual interviews on this process.

Methods: We analyzed the records of neurosurgery residency applicant interviews at our institution between 2016 and 2021. An average of 20 neurosurgery faculty members (clinical and research) interviewed each applicant and graded them 1 (best) to 4 (worst). Intraclass correlation coefficient (ICC) and Levene's test were used to assess the inter-rater and intra-rater reliability, respectively.

Results: 214 neurosurgery residency applicants were interviewed at a single institution between 2016 and 2021. The mean applicant rating each year ranged from 1.77 to 1.92. Inter-rater agreement was relatively poor in each year, (ICC < 0.5, P < 0.05). Among 60% of the raters, variability of scores significantly changed from year to year, (p < 0.05). When comparing the scores submitted during the virtual interview process (2021) with the scores submitted in the previous years (2016-2020), 2 interviewers (10%) had less variability using the virtual process.

Conclusion: Our analysis found that the current interview process for neurosurgery residency applicants' selection suffers from poor inter- and intra-rater reliability. Virtual interviews may be part of a cost-effective strategy to improve the reliability of the interview process. Further validation is needed, as well as identification of novel strategies to maximize the reliability of the selection process.