Do pooled estimates from orthodontic meta-analyses change depending on the meta-analysis approach? A meta-epidemiological study

Background: In meta-analyses involving a few trials, appropriate measures should be employed to assess between-study heterogeneity. When the number of studies is less than five and heterogeneity is evident, the Hartung and Knapp (HK) correction should be used. The aim of this study was to compare the reported estimates of published orthodontic meta-analyses with the pooled effect size estimates and prediction intervals (PI) calculated using eight heterogeneity estimators and corrected using the HK correction.

Material and methods: Systematic reviews (SRs) published between 2017 and 2022 in four orthodontic journals and the Cochrane Database of Systematic Reviews with a meta-analysis of at least three studies were sourced. Study characteristics were extracted at the SR and the outcome/meta-analysis levels. All selected meta-analyses were re-analysed by fitting a random-effects model using eight different heterogeneity estimators, both with and without the HK correction. For each meta-analysis, the overall estimate, along with its standard error, the P-value, and the corresponding 95% confidence interval (CI), the between-study variance (tau2), the I2 statistic, and the PI were calculated.

Results: One-hundred-six SRs were analysed. The most prevalent type of SR was non-Cochrane (95.3%), and the most used meta-analyses synthesis model was the random effect (83.0%). The median number of primary studies was 6 (interquartile range: 5, range: 3-45). The between-study variance was reported in most of the eligible meta-analyses (91.5%), but the type of heterogeneity estimator was reported in only one of them (0.9%). In 5 of 106 meta-analyses (4.7%), the HK correction was applied to adjust the CI of the pooled estimate. The percentage of statistically significant results, which became statistically non-significant, ranged from 16.7% to 25%, depending on the heterogeneity estimator. As the number of studies in a meta-analysis increased, the difference between corrected and uncorrected CIs reduced. Based on the PIs, more than half of the meta-analyses having statistically significant results are likely to change in the future, suggesting the result of the meta-analysis is not conclusive.

Conclusions: The statistical significance of pooled estimates from meta-analyses with at least three studies is sensitive to the HK correction, the heterogeneity variance estimator, and PIs. Clinicians should be aware of the clinical implications of not appropriately assessing the effect of the small number of studies and the between-study heterogeneity when interpreting results from meta-analyses.