Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol

Viknesh Sounderajah; Hutan Ashrafian; Robert M Golub; Shravya Shetty; Jeffrey De Fauw; Lotty Hooft; Karel Moons; Gary Collins; David Moher; Patrick M Bossuyt; Ara Darzi; Alan Karthikesalingam; Alastair K Denniston; Bilal Akhter Mateen; Daniel Ting; Darren Treanor; Dominic King; Felix Greaves; Jonathan Godwin; Jonathan Pearson-Stuttard; Leanne Harling; Matthew McInnes; Nader Rifai; Nenad Tomasev; Pasha Normahani; Penny Whiting; Ravi Aggarwal; Sebastian Vollmer; Sheraz R Markar; Trishan Panch; Xiaoxuan Liu; STARD-AI Steering Committee

doi:10.1136/bmjopen-2020-047709

Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol

BMJ Open. 2021 Jun 28;11(6):e047709. doi: 10.1136/bmjopen-2020-047709.

Authors

Viknesh Sounderajah^{1

2}, Hutan Ashrafian^{3

2}, Robert M Golub⁴, Shravya Shetty⁵, Jeffrey De Fauw⁶, Lotty Hooft^{7

8}, Karel Moons^{7

8}, Gary Collins⁹, David Moher¹⁰, Patrick M Bossuyt¹¹, Ara Darzi^{1

2}, Alan Karthikesalingam¹², Alastair K Denniston^{13

14

15

16}, Bilal Akhter Mateen¹⁷, Daniel Ting¹⁸, Darren Treanor¹⁹, Dominic King²⁰, Felix Greaves²¹, Jonathan Godwin⁶, Jonathan Pearson-Stuttard²², Leanne Harling¹, Matthew McInnes²³, Nader Rifai²⁴, Nenad Tomasev⁶, Pasha Normahani¹, Penny Whiting²⁵, Ravi Aggarwal^{1

2}, Sebastian Vollmer¹⁷, Sheraz R Markar¹, Trishan Panch²⁶, Xiaoxuan Liu^{13

14

15

16}; STARD-AI Steering Committee

Affiliations

¹ Department of Surgery and Cancer, Imperial College London, Paddington, UK.
² Institute of Global Health Innovation, Imperial College London, London, UK.
³ Department of Surgery and Cancer, Imperial College London, Paddington, UK [email protected].
⁴ Journal of the American Medical Association, Chicago, Illinois, USA.
⁵ Google Health, Pal Alto, California, USA.
⁶ DeepMind Technologies Ltd, London, UK.
⁷ Cochrane Netherlands, University Medical Center Utrecht, University of Utrecht, Utrecht, The Netherlands.
⁸ Department of Epidemiology, Julius Center for Health Sciences and Primary Care, Utrecht, The Netherlands.
⁹ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
¹⁰ Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada.
¹¹ Department of Epidemiology and Data Science, Amsterdam University Medical Centres, Duivendrecht, The Netherlands.
¹² Google Health, London, UK.
¹³ Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁴ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
¹⁵ Health Data Research UK, London, UK.
¹⁶ Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK.
¹⁷ The Alan Turing Institute, London, UK.
¹⁸ Singapore Eye Research Institute, Singapore National Eye Center, Singapore.
¹⁹ University of Leeds, Leeds, UK.
²⁰ Optum, Paddington, London, UK.
²¹ Department of Primary Care and Public Health, Imperial College London, London, UK.
²² School of Public Health, Imperial College London, London, UK.
²³ Department of Radiology, University of Ottawa, Ottawa, Ontario, Canada.
²⁴ Harvard Medical School, Boston, Massachusetts, USA.
²⁵ School of Social and Community Medicine, University of Bristol, Bristol, UK.
²⁶ Division of Health Policy and Management, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA.

Abstract

Introduction: Standards for Reporting of Diagnostic Accuracy Study (STARD) was developed to improve the completeness and transparency of reporting in studies investigating diagnostic test accuracy. However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions. As such, we propose an AI-specific version of the STARD checklist (STARD-AI), which focuses on the reporting of AI diagnostic test accuracy studies. This paper describes the methods that will be used to develop STARD-AI.

Methods and analysis: The development of the STARD-AI checklist can be distilled into six stages. (1) A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established; (2) An item generation process has been completed following a literature review, a patient and public involvement and engagement exercise and an online scoping survey of international experts; (3) A three-round modified Delphi consensus methodology is underway, which will culminate in a teleconference consensus meeting of experts; (4) Thereafter, the Project Team will draft the initial STARD-AI checklist and the accompanying documents; (5) A piloting phase among expert users will be undertaken to identify items which are either unclear or missing. This process, consisting of surveys and semistructured interviews, will contribute towards the explanation and elaboration document and (6) On finalisation of the manuscripts, the group's efforts turn towards an organised dissemination and implementation strategy to maximise end-user adoption.

Ethics and dissemination: Ethical approval has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 19IC5679). A dissemination strategy will be aimed towards five groups of stakeholders: (1) academia, (2) policy, (3) guidelines and regulation, (4) industry and (5) public and non-specific stakeholders. We anticipate that dissemination will take place in Q3 of 2021.

Keywords: health informatics; protocols & guidelines; quality in health care.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Artificial Intelligence*
Diagnostic Tests, Routine*
Humans
London
Research Design
Research Report

Abstract

Publication types

MeSH terms

Grants and funding