A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

Tianjing Li; Ian J Saldanha; Jens Jap; Bryant T Smith; Joseph Canner; Susan M Hutfless; Vernal Branch; Simona Carini; Wiley Chan; Berry de Bruijn; Byron C Wallace; Sandra A Walsh; Elizabeth J Whamond; M Hassan Murad; Ida Sim; Jesse A Berlin; Joseph Lau; Kay Dickersin; Christopher H Schmid

doi:10.1016/j.jclinepi.2019.07.005

A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

J Clin Epidemiol. 2019 Nov:115:77-89. doi: 10.1016/j.jclinepi.2019.07.005. Epub 2019 Jul 11.

Authors

Tianjing Li¹, Ian J Saldanha², Jens Jap³, Bryant T Smith³, Joseph Canner⁴, Susan M Hutfless⁵, Vernal Branch⁶, Simona Carini⁷, Wiley Chan⁸, Berry de Bruijn⁹, Byron C Wallace¹⁰, Sandra A Walsh¹¹, Elizabeth J Whamond¹², M Hassan Murad¹³, Ida Sim¹⁴, Jesse A Berlin¹⁵, Joseph Lau³, Kay Dickersin¹⁶, Christopher H Schmid¹⁷

Affiliations

¹ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, E6140, Baltimore, MD 21205, USA. Electronic address: [email protected].
² Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA; Department of Epidemiology (Secondary), Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA.
³ Department of Health Services, Policy, and Practice, Center for Evidence Synthesis in Health, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA.
⁴ Department of Surgery, Center for Outcomes Research, Johns Hopkins School of Medicine, 720 Rutland Avenue, Turner 74, Baltimore, MD 21205, USA.
⁵ Department of Surgery, Johns Hopkins School of Medicine, 600 N Wolfe St, Blalock 406, Baltimore, MD 21205, USA.
⁶ 119 Landings Drive, Suite 403, Mooresville, NC 28117, USA.
⁷ Department of Medicine, School of Medicine, University of California San Francisco, 3333 California Street, San Francisco, CA 94143, USA.
⁸ Internal Medicine, Kaiser Permanente Northwest, 500 NE Multnomah Street, Suite 100, KPB-2, Portland, OR 97232, USA.
⁹ Internal Medicine, National Research Council Canada, Digital Technologies Research Centre, 1200 Montreal Road, Ottawa, Ontario K1A 0R6, Canada.
¹⁰ Internal Medicine, College of Computer and Information Science, Northeastern University, 440 Huntington Avenue, 476 West Village H, Boston, MA 02115, USA.
¹¹ Internal Medicine, California Breast Cancer Organizations, 900 Kent Drive, Davis, CA 95616, USA.
¹² Internal Medicine, Cochrane Consumer Network, 1421 Woodstock Road, Fredericton, New Brunswick E3C 1P2, Canada.
¹³ Internal Medicine, Evidence-based Practice Center, Mayo Clinic, 200 1st Street, Rochester, MN 55905, USA.
¹⁴ Department of Medicine, University of California San Francisco School of Medicine, San Francisco, CA, USA.
¹⁵ Internal Medicine, Epidemiology, Johnson & Johnson, 1125 Trenton-Harbourton Road, K3-Zone 1, Titusville, NJ 08560, USA.
¹⁶ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, E6152, Baltimore, MD 21205, USA.
¹⁷ Department of Biostatistics, Brown University School of Public Health, 121 South Main Street, Box G-S121-7, Providence, RI 02912, USA.

PMID: 31302205
DOI: 10.1016/j.jclinepi.2019.07.005

Abstract

Objectives: Data Abstraction Assistant (DAA) is a software for linking items abstracted into a data collection form for a systematic review to their locations in a study report. We conducted a randomized cross-over trial that compared DAA-facilitated single-data abstraction plus verification ("DAA verification"), single data abstraction plus verification ("regular verification"), and independent dual data abstraction plus adjudication ("independent abstraction").

Study design and setting: This study is an online randomized cross-over trial with 26 pairs of data abstractors. Each pair abstracted data from six articles, two per approach. Outcomes were the proportion of errors and time taken.

Results: Overall proportion of errors was 17% for DAA verification, 16% for regular verification, and 15% for independent abstraction. DAA verification was associated with higher odds of errors when compared with regular verification (adjusted odds ratio [OR] = 1.08; 95% confidence interval [CI]: 0.99-1.17) or independent abstraction (adjusted OR = 1.12; 95% CI: 1.03-1.22). For each article, DAA verification took 20 minutes (95% CI: 1-40) longer than regular verification, but 46 minutes (95% CI: 26 to 66) shorter than independent abstraction.

Conclusion: Independent abstraction may only be necessary for complex data items. DAA provides an audit trail that is crucial for reproducible research.

Keywords: Accuracy; Data abstraction; Efficiency; Randomized cross-over trial; Software application; Systematic review.

Publication types

Comparative Study
Research Support, N.I.H., Extramural

MeSH terms

Abstracting and Indexing / methods*
Cross-Over Studies
Data Collection
Humans
Odds Ratio
Random Allocation
Software
Systematic Reviews as Topic*
Young Adult