Most meta-analyses in oral health do not have conclusive and robust results

Objectives: In meta-analyses with few studies, between-study heterogeneity is poorly estimated. The Hartung and Knapp (HK) correction and the prediction intervals can account for the uncertainty in estimating heterogeneity and the range of effect sizes we may encounter in future trials, respectively. The aim of this study was to assess the reported use of the HK correction in oral health meta-analyses and to compare the published reported results and interpretation i) to those calculated using eight heterogeneity estimators and the HK adjustment ii) and to the prediction intervals (PIs).

Methods: We sourced systematic reviews (SRs) published between 2021 and 2023 in eighteen leading specialty and general dental journals. We extracted study characteristics at the SR and meta-analysis level and re-analyzed the selected meta-analyses via the random-effects model and eight heterogeneity estimators, with and without the HK correction. For each meta-analysis, we re-calculated the overall estimate, the P-value, the 95 % confidence interval (CI) and the PI.

Results: We analysed 292 meta-analyses. The median number of primary studies included in meta-analysis was 8 (interquartile range [IQR] = [5.75-15] range: 3-121). Only 3/292 meta-analyses used the HK adjustment and 12/292 reported PIs. The percentage of statistically significant results that became non-significant varied across the heterogeneity estimators (7.45 %- 16.59 %). Based on the PIs, >60 % of meta-analyses with statistically significant results are likely to change in the future and >40 % of the PIs included the opposite pooled effect.

Conclusions: The precision and statistical significance of the pooled estimates from meta-analyses with at least three studies is sensitive to the HK correction, the heterogeneity variance estimator, and the PIs.

Clinical significance: Uncertainty in meta-analyses estimates should be considered especially when a small number of trials is available or vary notably in their precision. Misinterpretation of the summary results can lead to ineffective interventions being applied in clinical practice.