Evaluating Prognostic Bias of Critical Illness Severity Scores Based on Age, Sex, and Primary Language in the United States: A Retrospective Multicenter Study

Xiaoli Liu; Max Shen; Margaret Lie; Zhongheng Zhang; Chao Liu; Deyu Li; Roger G Mark; Zhengbo Zhang; Leo Anthony Celi

doi:10.1097/CCE.0000000000001033

Evaluating Prognostic Bias of Critical Illness Severity Scores Based on Age, Sex, and Primary Language in the United States: A Retrospective Multicenter Study

Crit Care Explor. 2024 Jan 17;6(1):e1033. doi: 10.1097/CCE.0000000000001033. eCollection 2024 Jan.

Authors

Xiaoli Liu^{1

2

3}, Max Shen⁴, Margaret Lie⁴, Zhongheng Zhang⁵, Chao Liu⁶, Deyu Li², Roger G Mark³, Zhengbo Zhang^{1

2}, Leo Anthony Celi^{3

4

7}

Affiliations

¹ Center for Artificial Intelligence in Medicine, The General Hospital of PLA, Beijing, China.
² School of Biological Science and Medical Engineering, Beihang University, Beijing, China.
³ Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA.
⁴ Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA.
⁵ Department of Emergency Medicine, Key Laboratory of Precision Medicine in Diagnosis and Monitoring Research of Zhejiang Province, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China.
⁶ Department of Critical Care Medicine, The First Medical Center, The General Hospital of PLA, Beijing, China.
⁷ Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA.

Abstract

Objectives: Although illness severity scoring systems are widely used to support clinical decision-making and assess ICU performance, their potential bias across different age, sex, and primary language groups has not been well-studied.

Design setting and patients: We aimed to identify potential bias of Sequential Organ Failure Assessment (SOFA) and Acute Physiology and Chronic Health Evaluation (APACHE) IVa scores via large ICU databases.

Setting/patients: This multicenter, retrospective study was conducted using data from the Medical Information Mart for Intensive Care (MIMIC) and eICU Collaborative Research Database. SOFA and APACHE IVa scores were obtained from ICU admission. Hospital mortality was the primary outcome. Discrimination (area under receiver operating characteristic [AUROC] curve) and calibration (standardized mortality ratio [SMR]) were assessed for all subgroups.

Interventions: Not applicable.

Measurements and main results: A total of 196,310 patient encounters were studied. Discrimination for both scores was worse in older patients compared with younger patients and female patients rather than male patients. In MIMIC, discrimination of SOFA in non-English primary language speakers patients was worse than that of English speakers (AUROC 0.726 vs. 0.783, p < 0.0001). Evaluating calibration via SMR showed statistically significant underestimations of mortality when compared with overall cohort in the oldest patients for both SOFA and APACHE IVa, female patients (1.09) for SOFA, and non-English primary language patients (1.38) for SOFA in MIMIC.

Conclusions: Differences in discrimination and calibration of two scores across varying age, sex, and primary language groups suggest illness severity scores are prone to bias in mortality predictions. Caution must be taken when using them for quality benchmarking and decision-making among diverse real-world populations.

Keywords: bias evaluation; calibration; discrimination; hospital mortality; illness severity scores.

Grants and funding

R01 EB017205/EB/NIBIB NIH HHS/United States