When impact trials are not feasible: alternatives to study the impact of prediction models on clinical practice

Patients with kidney disease have an uncertain future with prognosis varying greatly per patient. To get a better idea of what the future holds and tailor interventions to the individual patient, prediction models can be of great value. Before a prediction model can be applied in practice, its performance should be measured in target populations of interest (i.e., external validation) and whether it helps improve clinical practice (i.e., whether it impacts clinical practice) should be determined. The impact would ideally be determined using an impact trial, but such a trial is often not feasible, and the impact of prediction models is therefore rarely assessed. As a result, prediction models that may not be so impactful may end up in clinical practice and impactful models may not be implemented due to a lack of impact studies. Ultimately, many prediction models end up never being implemented, resulting in much research waste. To allow researchers to get an indication of a prediction model's impact on clinical practice, alternative methods to assess a prediction model's impact are important. In this paper, we discuss several alternatives, including interviews, case-based surveys, decision comparisons, outcome modelling, before-after analyses, and decision curve analyses. We discuss the general idea behind these approaches, including what information can be gathered from such studies and important pitfalls. Lastly, we provide examples of the different alternatives.