Assessing the Effectiveness of metaHelper: A Randomized Controlled Trial of a Web Application for Statistical Transformations in Meta-Analysis

Objective: To test whether the use of a web application (www.metaHelper.eu) can reduce errors and time needed for statistical transformations commonly performed for meta-analysis.

Study design and setting: We developed an R package and a corresponding web application to streamline statistical transformations in meta-analysis. A parallel randomized controlled trial following CONSORT guidelines was conducted. The trial was preregistered, and ethical approval was obtained from the University of Krems' ethics committee. We conducted a parallel randomized controlled trial with twenty-six participants, including researchers and students with varying levels of statistical expertise. Participants performed six statistical calculations commonly used in meta-analysis: computing pooled standard deviations, converting standard errors, and calculating standardized mean differences for both two-group and three-arm studies. Performance was evaluated based on accuracy and time taken to complete each task. A Bayesian mixed-effects model was used to analyze the results. Participants were randomly assigned to either the experimental group (metaHelper), or the control group (allowed to use internet sources, the Cochrane Handbook, or any other sources chosen by the participant).

Results: Main outcomes were correctness of calculations and time taken to complete tasks. The metaHelper group had a higher probability of providing correct answers (85 percent) compared to the control group (31 percent). Additionally, the metaHelper group completed an average of 133 seconds per task faster (95% CrI: 83 to 180).

Conclusion: MetaHelper effectively enhances the accuracy and efficiency of statistical transformations required for meta-analysis.