Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses - the METSA systematic review

Christian Gunge Riberholt; Markus Harboe Olsen; Joachim Birch Milan; Sigurlaug Hanna Hafliðadóttir; Jeppe Houmann Svanholm; Elisabeth Buck Pedersen; Charles Chin Han Lew; Mark Aninakwah Asante; Johanne Pereira Ribeiro; Vibeke Wagner; Buddheera W M B Kumburegama; Zheng-Yii Lee; Julie Perrine Schaug; Christina Madsen; Christian Gluud

doi:10.1186/s12874-024-02318-y

Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses - the METSA systematic review

BMC Med Res Methodol. 2024 Sep 9;24(1):196. doi: 10.1186/s12874-024-02318-y.

Authors

Christian Gunge Riberholt^{1

2

3}, Markus Harboe Olsen^{4

5}, Joachim Birch Milan⁴, Sigurlaug Hanna Hafliðadóttir⁶, Jeppe Houmann Svanholm⁷, Elisabeth Buck Pedersen⁸, Charles Chin Han Lew^{9

10}, Mark Aninakwah Asante⁴, Johanne Pereira Ribeiro^{11

12}, Vibeke Wagner^{8

13}, Buddheera W M B Kumburegama⁴, Zheng-Yii Lee^{14

15}, Julie Perrine Schaug¹¹, Christina Madsen¹⁶, Christian Gluud^{4

17}

Affiliations

¹ Copenhagen Trial Unit, Centre for Clinical Intervention Research, The Capital Region, Copenhagen University Hospital - Rigshospitalet, Blegdamsvej 9, Copenhagen, 2100, Denmark. [email protected].
² Department of Brain and Spinal Cord Injury, Neuroscience Centre, Copenhagen University Hospital - Rigshospitalet, Valdemar Hansens Vej 23, Glostrup, 2600, Denmark. [email protected].
³ Department of Neuroanaesthesiology, Neuroscience Centre, Copenhagen University Hospital - Rigshospitalet, Blegdamsvej 9, Copenhagen, 2100, Denmark. [email protected].
⁴ Copenhagen Trial Unit, Centre for Clinical Intervention Research, The Capital Region, Copenhagen University Hospital - Rigshospitalet, Blegdamsvej 9, Copenhagen, 2100, Denmark.
⁵ Department of Neuroanaesthesiology, Neuroscience Centre, Copenhagen University Hospital - Rigshospitalet, Blegdamsvej 9, Copenhagen, 2100, Denmark.
⁶ Bjarg Rehabilitation Center, Bugðusíðu 1, 603, Akureyri, Iceland.
⁷ Department of Gastrointestinal Surgery, Aalborg University Hospital South, Hobrovej 18-22, Aalborg, 9000, Denmark.
⁸ Department of Brain and Spinal Cord Injury, Neuroscience Centre, Copenhagen University Hospital - Rigshospitalet, Valdemar Hansens Vej 23, Glostrup, 2600, Denmark.
⁹ Department of Dietetics and Nutrition, Ng Teng Fong General Hospital, Singapore, Singapore.
¹⁰ Faculty of Health and Social Sciences, Singapore Institute of Technology, Singapore, Singapore.
¹¹ Center for Evidence-Based Psychiatry, Psychiatric Research Unit, Psychiatry Region Zealand, Faelledvej 6, Slagelse, 4200, Denmark.
¹² Department of Psychology, Faculty of Health Sciences, University of Southern Denmark, Campusvej 55, 5230, Odense, Denmark.
¹³ Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark.
¹⁴ Department of Anaesthesiology, Faculty of Medicine, University of Malaya, Kuala Lumpur, Malaysia.
¹⁵ Department of Cardiac Anesthesiology & Intensive Care Medicine, Charité, Berlin, Germany.
¹⁶ Psychiatric Research Unit, Psychiatry Region Zealand, Region Zealand, Fælledvej 6, Slagelse, 4200, Denmark.
¹⁷ Department of Regional Health Research, The Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark.

Abstract

Background: Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis.

Methods: We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision.

Results: We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%).

Conclusions: The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity.

Study registration: PROSPERO CRD42021273811.

Keywords: Meta-analysis; Methodology; Research-on-research; Systematic review; Trial sequential analysis.

Publication types

Systematic Review

MeSH terms

Humans
Meta-Analysis as Topic*
Randomized Controlled Trials as Topic* / methods
Randomized Controlled Trials as Topic* / standards
Randomized Controlled Trials as Topic* / statistics & numerical data
Research Design / standards
Systematic Reviews as Topic* / methods