The 2021 SIIM-FISABIO-RSNA Machine Learning COVID-19 Challenge: Annotation and Standard Exam Classification of COVID-19 Chest Radiographs

Paras Lakhani; J Mongan; C Singhal; Q Zhou; K P Andriole; W F Auffermann; P M Prasanna; T X Pham; Michael Peterson; P J Bergquist; T S Cook; S F Ferraciolli; G C A Corradi; M S Takahashi; C S Workman; M Parekh; S I Kamel; J Galant; A Mas-Sanchez; E C Benítez; M Sánchez-Valverde; L Jaques; M Panadero; M Vidal; M Culiañez-Casas; D Angulo-Gonzalez; S G Langer; María de la Iglesia-Vayá; G Shih

doi:10.1007/s10278-022-00706-8

The 2021 SIIM-FISABIO-RSNA Machine Learning COVID-19 Challenge: Annotation and Standard Exam Classification of COVID-19 Chest Radiographs

J Digit Imaging. 2023 Feb;36(1):365-372. doi: 10.1007/s10278-022-00706-8. Epub 2022 Sep 28.

Authors

Paras Lakhani¹, J Mongan², C Singhal³, Q Zhou³, K P Andriole⁴, W F Auffermann⁵, P M Prasanna⁵, T X Pham⁵, Michael Peterson⁵, P J Bergquist⁶, T S Cook⁷, S F Ferraciolli⁸, G C A Corradi⁸, M S Takahashi⁸, C S Workman⁹, M Parekh¹⁰, S I Kamel¹⁰, J Galant¹¹, A Mas-Sanchez¹¹, E C Benítez¹¹, M Sánchez-Valverde¹¹, L Jaques¹¹, M Panadero¹¹, M Vidal¹¹, M Culiañez-Casas¹¹, D Angulo-Gonzalez¹², S G Langer¹³, María de la Iglesia-Vayá¹⁴, G Shih¹⁵

Affiliations

¹ Department of Radiology, Thomas Jefferson University, Sidney Kimmel Jefferson Medical College, 111 S 11th St, Philadelphia, PA, 19107, USA. [email protected].
² University of California San Francisco, San Francisco, CA, USA.
³ MD.AI, New York, NY, USA.
⁴ Mass General Brigham and Harvard Medical School, Boston, MA, USA.
⁵ University of Utah Health, Salt Lake City, UT, USA.
⁶ Medstar Georgetown University Hospital, Washington DC, USA.
⁷ University of Pennsylvania, Philadelphia, PA, USA.
⁸ DASA, Alphaville, Barueri, SP, Brazil.
⁹ Vanderbilt University Medical Center, Nashville TN, USA.
¹⁰ Department of Radiology, Thomas Jefferson University, Sidney Kimmel Jefferson Medical College, 111 S 11th St, Philadelphia, PA, 19107, USA.
¹¹ Hospital Universitario San Juan de Alicante, San Juan de Alicante, Alicante, Spain.
¹² Virgen del Rocio University Hospital, Seville, Spain.
¹³ Mayo Clinic, Rochester, MN, USA.
¹⁴ The Foundation for the Promotion of Health and Biomedical Research of Valencia Region, Valencia, Spain.
¹⁵ Weill Cornell Medicine, New York, NY, USA.

Abstract

We describe the curation, annotation methodology, and characteristics of the dataset used in an artificial intelligence challenge for detection and localization of COVID-19 on chest radiographs. The chest radiographs were annotated by an international group of radiologists into four mutually exclusive categories, including "typical," "indeterminate," and "atypical appearance" for COVID-19, or "negative for pneumonia," adapted from previously published guidelines, and bounding boxes were placed on airspace opacities. This dataset and respective annotations are available to researchers for academic and noncommercial use.

Keywords: Artificial Intelligence; COVID-19; Machine Learning; Pneumonia; Radiography; Thorax.

Publication types

Research Support, Non-U.S. Gov't
Research Support, U.S. Gov't, Non-P.H.S.

MeSH terms

Artificial Intelligence
COVID-19*
Humans
Machine Learning
Radiography
Radiography, Thoracic / methods
Radiologists