Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

Elie A Akl; Alonso Carrasco-Labra; Romina Brignardello-Petersen; Ignacio Neumann; Bradley C Johnston; Xin Sun; Matthias Briel; Jason W Busse; Shanil Ebrahim; Carlos E Granados; Alfonso Iorio; Affan Irfan; Laura Martínez García; Reem A Mustafa; Anggie Ramírez-Morera; Anna Selva; Ivan Solà; Andrea Juliana Sanabria; Kari A O Tikkinen; Per O Vandvik; Robin W M Vernooij; Oscar E Zazueta; Qi Zhou; Gordon H Guyatt; Pablo Alonso-Coello

doi:10.1136/bmjopen-2015-009368

Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

BMJ Open. 2015 Sep 30;5(9):e009368. doi: 10.1136/bmjopen-2015-009368.

Authors

Elie A Akl¹, Alonso Carrasco-Labra², Romina Brignardello-Petersen³, Ignacio Neumann⁴, Bradley C Johnston⁵, Xin Sun⁶, Matthias Briel⁷, Jason W Busse⁸, Shanil Ebrahim⁹, Carlos E Granados¹⁰, Alfonso Iorio¹¹, Affan Irfan¹², Laura Martínez García¹³, Reem A Mustafa¹⁴, Anggie Ramírez-Morera¹⁵, Anna Selva¹³, Ivan Solà¹³, Andrea Juliana Sanabria¹³, Kari A O Tikkinen¹⁶, Per O Vandvik¹⁷, Robin W M Vernooij¹³, Oscar E Zazueta¹³, Qi Zhou¹⁸, Gordon H Guyatt¹¹, Pablo Alonso-Coello¹³

Affiliations

¹ Department of Internal Medicine, American University of Beirut, Beirut, Lebanon Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada.
² Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Santiago, Chile.
³ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Santiago, Chile Institute of Health Policy, Management & Evaluation, University of Toronto, Toronto, Ontario, Canada.
⁴ Department of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile.
⁵ Institute of Health Policy, Management & Evaluation, University of Toronto, Toronto, Ontario, Canada Department of Anesthesia & Pain Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada.
⁶ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Clinical Research and Evaluation Unit, Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, Chengdu, China.
⁷ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland.
⁸ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada The Michael G. DeGroote Institute for Pain Research and Care, McMaster University, Hamilton, Ontario, Canada.
⁹ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada Department of Medicine, Stanford Prevention Research Center, Stanford University, Palo Alto, California, USA.
¹⁰ Área de investigaciones, Facultad de Medicina, Universidad de La Sabana, Chía, Colombia.
¹¹ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Department of Medicine, McMaster University, Hamilton, Ontario, Canada.
¹² Department of Cardiology, University of Louisville, Louisville, Kentucky, USA.
¹³ Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau-CIBER of Epidemiology and Public Health (CIBERESP- IIB Sant Pau), Barcelona, Spain.
¹⁴ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Departments of Medicine, Nephrology and Biomedical and Health Informatics, University of Missouri-Kansas City, Kansas City, Missouri, USA.
¹⁵ Health Care Development Division, IHCAI Foundation & Central America Cochrane, San José, Costa Rica.
¹⁶ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada Departments of Urology and Public Health, Helsinki University Central Hospital and University of Helsinki, Helsinki, Finland.
¹⁷ Norwegian Knowledge Centre for the Health Services, Oslo, Norway Department of Medicine, Innlandet Hospital Trust, Gjøvik, Norway.
¹⁸ Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada.

Abstract

Objectives: To describe how systematic reviewers are reporting missing data for dichotomous outcomes, handling them in the analysis and assessing the risk of associated bias.

Methods: We searched MEDLINE and the Cochrane Database of Systematic Reviews for systematic reviews of randomised trials published in 2010, and reporting a meta-analysis of a dichotomous outcome. We randomly selected 98 Cochrane and 104 non-Cochrane systematic reviews. Teams of 2 reviewers selected eligible studies and abstracted data independently and in duplicate using standardised, piloted forms with accompanying instructions. We conducted regression analyses to explore factors associated with using complete case analysis and with judging the risk of bias associated with missing participant data.

Results: Of Cochrane and non-Cochrane reviews, 47% and 7% (p<0.0001), respectively, reported on the number of participants with missing data, and 41% and 9% reported a plan for handling missing categorical data. The 2 most reported approaches for handling missing data were complete case analysis (8.5%, out of the 202 reviews) and assuming no participants with missing data had the event (4%). The use of complete case analysis was associated only with Cochrane reviews (relative to non-Cochrane: OR=7.25; 95% CI 1.58 to 33.3, p=0.01). 65% of reviews assessed risk of bias associated with missing data; this was associated with Cochrane reviews (relative to non-Cochrane: OR=6.63; 95% CI 2.50 to 17.57, p=0.0001), and the use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology (OR=5.02; 95% CI 1.02 to 24.75, p=0.047).

Conclusions: Though Cochrane reviews are somewhat less problematic, most Cochrane and non-Cochrane systematic reviews fail to adequately report and handle missing data, potentially resulting in misleading judgements regarding risk of bias.

Keywords: EPIDEMIOLOGY; STATISTICS & RESEARCH METHODS.

Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Bias*
Data Accuracy*
Humans
Meta-Analysis as Topic*
Research Design / standards
Review Literature as Topic*
Surveys and Questionnaires