Creating a Standardized Tool for the Evaluation and Comparison of Artificial Intelligence-Based Computer-Aided Detection Programs in Colonoscopy: a Modified Delphi Approach

Sanjay R V Gadi; Yuichi Mori; Masashi Misawa; James E East; Cesare Hassan; Alessandro Repici; Michael F Byrne; Daniel von Renteln; David G Hewett; Pu Wang; Yutaka Saito; Carolina Ogawa Matsubayashi; Omer F Ahmad; Prateek Sharma; Seth A Gross; Neil Sengupta; Nabil Mansour; Andrea Cherubini; Nhan Ngo Dinh; Xiao Xiao; Peter Mountney; Juana González-Bueno Puyal; Greg Little; Shawn LaRocco; Sailesh Conjeti; Hannes Seibt; Dror Zur; Hitoshi Shimada; Tyler M Berzin; Jeremy R Glissen Brown

doi:10.1016/j.gie.2024.11.042

Creating a Standardized Tool for the Evaluation and Comparison of Artificial Intelligence-Based Computer-Aided Detection Programs in Colonoscopy: a Modified Delphi Approach

Gastrointest Endosc. 2024 Nov 26:S0016-5107(24)03752-0. doi: 10.1016/j.gie.2024.11.042. Online ahead of print.

Authors

Sanjay R V Gadi¹, Yuichi Mori², Masashi Misawa³, James E East⁴, Cesare Hassan⁵, Alessandro Repici⁵, Michael F Byrne⁶, Daniel von Renteln⁷, David G Hewett⁸, Pu Wang⁹, Yutaka Saito¹⁰, Carolina Ogawa Matsubayashi¹¹, Omer F Ahmad¹², Prateek Sharma¹³, Seth A Gross¹⁴, Neil Sengupta¹⁵, Nabil Mansour¹⁶, Andrea Cherubini¹⁷, Nhan Ngo Dinh¹⁷, Xiao Xiao¹⁸, Peter Mountney¹⁹, Juana González-Bueno Puyal¹⁹, Greg Little²⁰, Shawn LaRocco²⁰, Sailesh Conjeti²⁰, Hannes Seibt²¹, Dror Zur²², Hitoshi Shimada²³, Tyler M Berzin²⁴, Jeremy R Glissen Brown²⁵

Affiliations

¹ Department of Medicine, Duke University School of Medicine, Durham, North Carolina, USA. Electronic address: [email protected].
² Digestive Disease Center, Showa University Northern Yokohama Hospital, Yokohama, Japan; Clinical Effectiveness Research Group, Institute of Health and Society, University of Oslo, Oslo, Norway; Department of Transplantation Medicine, Oslo University Hospital, Oslo, Norway.
³ Digestive Disease Center, Showa University Northern Yokohama Hospital, Yokohama, Japan.
⁴ Translational Gastroenterology Unit, John Radcliffe Hospital, Oxford, UK; Oxford NIHR Biomedical Research Centre, University of Oxford, Oxford, UK.
⁵ Department of Gastroenterology, Humanitas Clinical and Research Center, IRCCS, Milan, Italy; Humanitas University, Department of Biomedical Sciences, Pieve Emanuele, Milan, Italy.
⁶ Division of Gastroenterology, Vancouver General Hospital, University of British Columbia, Vancouver, British Columbia, Canada; Satisfai Health, Vancouver, British Columbia, Canada.
⁷ Division of Gastroenterology, Montréal University Hospital and Research Center, Montreal, Canada.
⁸ School of Medicine, The University of Queensland, Brisbane, Australia.
⁹ Sichuan Academy of Medical Sciences and Sichuan Provincial People's Hospital, Chengdu, China.
¹⁰ Endoscopy Division, National Cancer Center Hospital, Tokyo, Japan.
¹¹ Gastrointestinal Endoscopy Unit, Gastroenterology Department, University of São Paulo Medical School, São Paulo, Brazil.
¹² Wellcome/EPSRC Centre for Interventional & Surgical Sciences, University College London, London, UK.
¹³ Division of Gastroenterology and Hepatology, University of Kansas School of Medicine and VA Medical Center, Kansas City, USA.
¹⁴ Division of Gastroenterology and Hepatology, New York University Langone Health System, New York, New York, USA.
¹⁵ Section of Gastroenterology, University of Chicago Medicine, Chicago, Illinois, USA.
¹⁶ Section of Gastroenterology and Hepatology, Baylor College of Medicine, Houston, Texas, USA.
¹⁷ Cosmo Intelligent Medical Devices, Dublin, Ireland.
¹⁸ Wision AI, Palo Alto, California, USA.
¹⁹ Odin Vision, London, UK; Olympus Corporation, Tokyo, Japan.
²⁰ Olympus Corporation, Tokyo, Japan.
²¹ Pentax Medical Europe, Hamburg, Germany.
²² Magentiq Eye, Haifa, Israel.
²³ FUJIFILM Healthcare Americas Corporation, Lexington, Massachusetts, USA.
²⁴ Center for Advanced Endoscopy, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, Massachusetts, USA. Electronic address: [email protected].
²⁵ Division of Gastroenterology, Duke University Medical Center, Durham, North Carolina, USA. Electronic address: [email protected].

PMID: 39608592
DOI: 10.1016/j.gie.2024.11.042

Abstract

Background and aim: Multiple computer-aided detection (CADe) software have now achieved regulatory approval in the US, Europe, and Asia and are being used in routine clinical practice to support colorectal cancer screening. There is uncertainty regarding how different CADe algorithms may perform. No objective methodology exists for comparing different algorithms. We aimed to identify priority scoring metrics for CADe evaluation and comparison.

Methods: A modified Delphi approach was used. Twenty-five global leaders in CADe in colonoscopy, including endoscopists, researchers, and industry representatives, participated in an online survey over the course of 8 months. Participants generated 121 scoring criteria, 54 of which were deemed within the study scope and distributed for review and asynchronous email-based open comment. Participants then scored criteria in order of priority on a 5-point Likert scale during ranking round one. The top eleven highest-priority criteria were re-distributed, with another opportunity for open-comment, followed by a final round of priority scoring to identify the final 6 criteria.

Results: Mean priority scores for the 54 criteria ranged from 2.25 to 4.38 following the first ranking round. The top eleven criteria following ranking round one yielded mean priority scores ranging from 3.04 to 4.16. The final six highest priority criteria were 1) sensitivity (average = 4.16) and separate & independent validation of the CADe algorithm (4.16), 3) adenoma detection rate (4.08), 4) false positive rate (4.00), 5) latency (3.84), and 6) adenoma miss rate (3.68).

Conclusions: This is the first reported international consensus statement of priority scoring metrics for CADe in colonoscopy. These scoring criteria should inform CADe software development and refinement. Future research should validate these metrics on a benchmark video data set to develop a validated scoring instrument.

Keywords: Colonoscopy; adenoma detection rate; artificial intelligence; computer-aided detection; sensitivity.