Doctors and numbers: an assessment of the critical risk interpretation test

Background: Risk interpretation affects decision making. Yet, there is no valid assessment of how clinicians interpret the risk data that they commonly encounter.

Objective: To establish the reliability and validity of a 20-item test of clinicians' risk interpretation.

Methods: The Critical Risk Interpretation Test (CRIT) measures clinicians' abilities to 1) modify the interpretation based on meaningful differences in the outcome (e.g., disease specific v. all-cause mortality) and time period (e.g., lifetime v. 10-year mortality), 2) maintain a stable interpretation for different risk framings (e.g., relative v. absolute risk), and 3) correctly interpret how diagnostic testing modifies risk. There were 658 clinicians and medical trainees who participated: 116 nurse practitioners (NPs) at a national conference, 273 medical students at 1 institution, 148 residents in internal medicine at 2 institutions, and 121 internists at 1 institution. Participants completed a self-administered paper test during educational conferences. Seventeen evidence-based medicine experts took the test online and formally assessed content validity. Eighteen second-year medical students were recruited to take the test and a retest 3 weeks later to explore test-retest correlation.

Results: Expert review supported test clarity and content validity. Factor analysis supported that the CRIT identifies at least 3 separable areas of clinician knowledge. Test-retest correlation was fair (intraclass correlation coefficient = 0.65; standard error = 0.15). Scores on our test correlated with other tests of related abilities. Mean test scores varied among groups, with differences in prior evidence-based medicine training and experience (93 for NPs, 101 for medical students, 101 for residents, 103 for academic internists, and 110 for physician experts; P < 0.001).

Conclusions: Our results provide supporting evidence for the reliability and validity of the CRIT as an index of critical risk interpretation abilities, which is acceptable and feasible to administer in an educational setting.