Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: A systematic review

To evaluate the performance accuracy and workload savings of artificial intelligence (AI)-based automation tools in comparison with human reviewers in medical literature screening for systematic reviews (SR) of primary studies in cancer research in order to gain insights on improving the efficiency of producing SRs. Medline, Embase, the Cochrane Library, and PROSPERO databases were searched from inception to November 30, 2022. Then, forward and backward literature searches were completed, and the experts in this field including the authors of the articles included were contacted for a thorough grey literature search. This SR was registered on PROSPERO (CRD 42023384772). Among the 3947 studies obtained from search, five studies met the preplanned study selection criteria. These five studies evaluated four AI tools: Abstrackr (four studies), RobotAnalyst (one), EPPI-Reviewer (one), and DistillerSR (one). Without missing final included citations, Abstrackr eliminated 20%-88% of titles and abstracts (time saving of 7-86 hours) and 59% of the full-texts (62 h) from human review across four different cancer-related SRs. In comparison, RobotAnalyst (1% of titles and abstracts, 1 h), EPPI Review (38% of titles and abstracts, 58 h; 59% of full-texts, 62 h), DistillerSR (42% of titles and abstracts, 22 h) also provided similar or lower work savings for single cancer-related SRs. AI-based automation tools exhibited promising but varying levels of accuracy and efficiency during the screening process of medical literature for conducting SRs in the cancer field. Until further progress is made and thorough evaluations are conducted, AI tools should be utilized as supplementary aids rather than complete substitutes for human reviewers.