Validation of an acute kidney injury prediction model as a clinical decision support system

Background: Acute kidney injury (AKI) is a critical clinical condition that requires immediate intervention. We developed an artificial intelligence (AI) model called PRIME Solution to predict AKI and evaluated its ability to enhance clinicians' predictions.

Methods: The PRIME Solution was developed using convolutional neural networks with residual blocks on 183,221 inpatient admissions from a tertiary hospital (2013-2017) and externally validated with 4,501 admissions at another tertiary hospital (2020-2021). To assess its application, we conducted a prospective evaluation using retrospectively collected data from 100 patients at the latter hospital, including 15 AKI cases. AKI prediction performance was compared among specialists, physicians, and medical students, both with and without AI assistance.

Results: Without assistance, specialists demonstrated the highest accuracy (0.797), followed by medical students (0.619) and the PRIME Solution (0.568). AI assistance improved overall recall (61.0% to 74.0%) and F1 scores (38.7% to 42.0%), while reducing average review time (73.8 to 65.4 seconds, p < 0.001). However, the impact varied across expertise levels. Specialists showed the greatest improvement (recall, 32.1% to 64.3%; F1, 36.4% to 48.6%), whereas medical students' performance improved but aligned more closely with the AI model. Additionally, the effect of AI assistance varied by prediction outcome, showing greater improvement in recall for cases predicted as AKI, and better precision, F1 score, and review time reduction (73.4 to 62.1 seconds, p < 0.001) for cases predicted as non-AKI.

Conclusion: AKI predictions were enhanced by AI assistance, but the improvements varied according to the expertise of the user.