Objectives: The American Board of Emergency Medicine (ABEM) Emergency Medical Services Medicine (EMS) subspecialty was approved by the American Board of Medical Specialties on September 23, 2010. Subspecialty certification in EMS was contingent on two key elements-completing Accreditation Council for Graduate Medical Education (ACGME)-accredited EMS training and passing the subspecialty certification examination developed by ABEM. The first EMS certification examination was offered in October 2013. Meaningful certification requires rigorous assessment. In this instance, the EMS certification examination sought to embrace the tenets of validity, reliability, and fairness. For the purposes of this report, the sources of validity evidence were anchored on the EMS core content, the examination development process, and the association between fellowship training and passing the certification examination.
Methods: We chose to use validity evidence that included: 1) content validity (based on the EMS core content); 2) response processes (test items required intended cognitive processes); 3) internal structure supported by the internal relationships among items; 4) relations to other variables, specifically the association between examination performance and ACGME-accredited fellowship training; and 5) the consequences of testing.
Results: There is strong content validity evidence for the EMS examination based on the core content and its detailed development process. The core content and supporting job-task analysis was also used to define the examination blueprint. Internal structure support was evidenced by Cronbach's coefficient alpha, which ranged from 0.82 to 0.92. Physicians who completed ACGME-accredited EMS fellowship training were more likely to pass the EMS certification examination (chi square, p < 0.0001; Cramér's, V = 0.24). Finally, there were two sources of consequential validity evidence-use of test results to determine certification and use of the resulting certificate.
Conclusions: There is substantial and varied validity evidence to support the use of the EMS certifying examination in making summative decisions to award certification in EMS. Of note, there was a statistically significant association between ACGME-accredited fellowship training and passing the examination.