Objectives: The Global Health Evidence Evaluation Framework project is aimed at developing an evidence framework for the purposes of informing efficacious, effective, sustainable global health programs at the community and scale level.
Data sources: Literature search of published and gray literature, input from a multidisciplinary Technical Expert Panel (TEP).
Methods: With input from the TEP we identified six existing evidence frameworks for public health/global health interventions or programs and applied these frameworks to the evidence bases for three exemplar interventions chosen to represent a diverse set of global health programs or interventions: household water chlorination, prevention of mother-to-child transmission of HIV, and lay community health workers to reduce child mortality. Review of the findings with the TEP identified an important gap as the reporting of information about the implementation of the intervention or program. We identified three existing criteria sets for implementation reporting, and selected from them 10 criteria that covered areas identified by the TEP. We pilot tested these 10 criteria on three published articles of effectiveness for each of the three exemplar interventions.
Results: Assessing the same body of evidence yields different conclusions regarding strength of evidence depending on which framework is used, some of which were extreme (with the same evidence base assessed as “strong” in one framework and “low quality” in another framework). All frameworks focus on efficacy and/or effectiveness with most attention going to the allocation method of study participants to the intervention (randomization versus other method) in assessing study quality. Many fail to consider implementation issues, and none explicitly assessed costs or sustainability at the community or scale level. Incorporating insights from the frameworks for implementation criteria helped to address some of these gaps, yet our pilot test of 10 implementation reporting criteria on nine published studies showed great variability both within and across studies. Some criteria were classified as having “good” reporting in almost all studies (criteria about the setting of the study, the characteristics of the recipients and the mode of delivery of the intervention or program) while others were reported on rarely or never (the rationale for the intervention, the costs of the intervention, an assessment of the population needs, and two criteria about barriers and facilitators of the implementation). Similarly, two articles had “good” or “fair” documentation for greater than 75 percent of criteria, while three articles had “poor or none” documentation for more than 50 percent of criteria.
Conclusions: Existing frameworks for the assessment of public health evidence do not deliver key pieces of information to inform best practices for community and large-scale global health programs, with the lack of information about implementation and sustainability being an important identified gap. In a pilot study, our application of existing criteria for the reporting of implementation information shows great variability within and across published global health intervention studies. The reporting of implementation information using criteria that have broad support across diverse stakeholders and that can be reliably assessed would provide policymakers a stronger evidence base upon which to make decisions.