Development of a reference standard to assign bacterial versus viral infection etiology using an all-inclusive methodology for comparison of novel diagnostic tool performance

Coburn Allen; J Kate Deanehan; Yaniv Dotan; Matthew A Eisenberg; Andrew M Fine; Jonathan Isenberg; Ann Kane; Dani Kirshner; Todd W Lyons; Yasmin Maor; Ami Neuberger; Daniel G Ostermayer; Sharona Paz; Oded Scheuerman; Shahaf Shiber; Victoria A Statler; Michal Stein; Renata Yakubov; Shirly Yanai; Roy Navon; Lior Kellerman; Tanya M Gottlieb; Eran Eden

doi:10.1093/cid/ciae656

Development of a reference standard to assign bacterial versus viral infection etiology using an all-inclusive methodology for comparison of novel diagnostic tool performance

Clin Infect Dis. 2025 Jan 3:ciae656. doi: 10.1093/cid/ciae656. Online ahead of print.

Authors

Coburn Allen¹, J Kate Deanehan², Yaniv Dotan³, Matthew A Eisenberg^{4

5}, Andrew M Fine^{4

5}, Jonathan Isenberg⁶, Ann Kane², Dani Kirshner^{7

8}, Todd W Lyons^{4

5}, Yasmin Maor^{9

10}, Ami Neuberger^{8

11

12}, Daniel G Ostermayer¹³, Sharona Paz¹⁴, Oded Scheuerman¹⁵, Shahaf Shiber^{10

16}, Victoria A Statler¹⁷, Michal Stein^{10

18

19}, Renata Yakubov^{8

20}, Shirly Yanai²¹, Roy Navon²², Lior Kellerman²², Tanya M Gottlieb²², Eran Eden^{22

23}

Affiliations

¹ University of Texas at Austin Dell Medical School, Austin, TX, USA.
² John's Hopkins Children's Center, Baltimore, MD, USA.
³ Rambam Health Care Campus, Haifa, Israel.
⁴ Division of Emergency Medicine, Boston Children's Hospital, Boston, MA, USA.
⁵ Harvard Medical School, Boston, MA, USA.
⁶ Department of Internal Medicine H, Rambam Health Care Campus, Haifa, Israel.
⁷ Geriatric unit, Clalit Health Services, Haifa, Israel.
⁸ The Ruth and Bruce Rappaport Faculty of Medicine, The Technion-Israel Institute of Technology, Haifa, Israel.
⁹ Infectious Disease Unit, E. Wolfson Medical Center Holon, Israel.
¹⁰ Faculty of Medical and Health Sciences, Tel Aviv university, Tel Aviv Israel.
¹¹ Division of Internal Medicine, Rambam Healthcare Campus, Haifa, Israel.
¹² Unit of Infectious Diseases, Rambam Healthcare Campus, Haifa, Israel.
¹³ Department of Emergency Medicine, McGovern Medical School, UTHealth, Houston, TX, USA.
¹⁴ Maccabi Health Care Services, Israel.
¹⁵ Department of Pediatrics B, Schneider Children's Medical Center of Israel, Petah Tikva, Israel.
¹⁶ Department of Emergency Medicine, Beilinson Hospital, Petah Tikva, Israel.
¹⁷ Department of Pediatric Infectious Diseases, Norton Children's and University of Louisville School of Medicine, Louisville, KY, USA.
¹⁸ Pediatric Infectious Disease Unit, The Edmond and Lily Safra Children's Hospital, Chaim Sheba Medical Center, Ramat Gan, Israel.
¹⁹ Division of Infectious Diseases, Children's National Hospital, Washington, DC, USA.
²⁰ The Pediatric Nephrology Institute, Rambam Health Care Campus, Haifa, Israel.
²¹ Rabin Medical Center, Petah Tikvah, Israel.
²² MeMed, Tirat Carmel, Israel.
²³ MeMed, Andover, MA, USA.

PMID: 39750735
DOI: 10.1093/cid/ciae656

Abstract

Background: Diagnostic test evaluation requires a reference standard. We describe an approach for creating a reference standard for acute infection using unrestricted adjudication and apply it to compare biomarker tools.

Methods: Adults and children with suspected acute infection enrolled in three prospective studies at emergency departments and urgent cares were included. Adjudicators, blinded to C-reactive protein, procalcitonin, and MeMed BV (MMBV), labeled each case (bacterial/viral/non-infectious/indeterminate). Initial adjudication involved 3 adjudicators. Reference standard cohorts were defined: Microbiologically confirmed (3/3 adjudicators concur with high confidence and a concordant microbiological finding), unanimous (3/3 adjudicators concur with high confidence), suspected (3/3 adjudicators concur with high/moderate confidence or 2/3 adjudicators concur with high confidence) and all-inclusive (remaining unlabeled cases were reviewed by up to 7 additional adjudicators until reaching a leading label).

Results: Among 1016 patients, 156 difficult-to-diagnose cases required over 3 adjudicators. The area under the receiver operating characteristic curve in the microbiologically confirmed (n=427), unanimous (n=565), suspected (n=860) and all-inclusive (n=1016) cohorts for MMBV were 0.98 (95% confidence interval 0.94-1.00), 0.98 (0.95-1.00), 0.95 (0.92-0.98) and 0.90 (0.87-0.93), respectively, and for procalcitonin were 0.69 (0.57-0.81), 0.77 (0.68-0.86), 0.74 (0.68-0.80) and 0.70 (0.65-0.75), respectively. A delta in performance between MMBV and procalcitonin was maintained across the different cohorts.

Conclusion: Creating a reference standard that includes difficult-to-diagnose cases demands an approach to addressing diagnostic uncertainty in acute infections. Tool performance depends on the reference standard applied and decreases as the difficulty to diagnose increases, highlighting the importance of using the same reference standard when comparing tools.

Keywords: Adjudication-based reference standard; diagnostic performance; diagnostic tool; difficult-to-diagnose cases; infection etiology.