Improving event prediction using general practitioner clinical judgement in a digital risk stratification model: a pilot study

Background: Numerous tools based on electronic health record (EHR) data that predict risk of unscheduled care and mortality exist. These are often criticised due to lack of external validation, potential for low predictive ability and the use of thresholds that can lead to large numbers being escalated for assessment that would not have an adverse outcome leading to unsuccessful active case management. Evidence supports the importance of clinical judgement in risk prediction particularly when ruling out disease. The aim of this pilot study was to explore performance analysis of a digitally driven risk stratification model combined with GP clinical judgement to identify patients with escalating urgent care and mortality events.

Methods: Clinically risk stratified cohort study of 6 GP practices in a deprived, multi-ethnic UK city. Initial digital driven risk stratification into Escalated and Non-escalated groups used 7 risk factors. The Escalated group underwent stratification using GP global clinical judgement (GCJ) into Concern and No concern groupings.

Results: 3968 out of 31,392 patients were data stratified into the Escalated group and further categorised into No concern (n = 3450 (10.9%)) or Concern (n = 518 (1.7%)) by GPs. The 30-day combined event rate (unscheduled care or death) per 1,000 was 19.0 in the whole population, 67.8 in the Escalated group and 168.0 in the Concern group (p < 0.001). The de-escalation effect of GP assessment into No Concern versus Concern was strongly negatively predictive (OR 0.25 (95%CI 0.19-0.33; p < 0.001)). The whole population ROC for the global approach (Non-escalated, GP No Concern, GP Concern) was 0.614 (0.592-0.637), p < 0.001, and the increase in the ROC area under the curve for 30-day events was all focused here (+ 0.4% (0.3-0.6%, p < 0.001), translating into a specific ROC c-statistic for GP GCJ of 0.603 ((0.565-0.642), p < 0.001).

Conclusions: The digital only component of the model performed well but adding GP clinical judgement significantly improved risk prediction, particularly by adding negative predictive value.