Consensus recommendations for improvement of unmet clinical needs--the example of chronic graft-versus-host disease: a systematic review and meta-analysis

Background: Consensus recommendations are used to improve the methodology of research about rare disorders, but their uptake is unknown. We studied the uptake of consensus recommendations in steroid-refractory chronic graft-versus-host disease (SR-cGVHD). Although in 2006 the National Institutes of Health (NIH) cGVHD consensus project produced recommendations for clinical trials, guidelines have emphasised the scarcity of valuable evidence for all tested interventions.

Methods: We searched Medline (PubMed) between Jan 1, 1998, and Oct 1, 2013, for non-randomised studies of systemic treatment for SR-cGVHD. To measure adherence to NIH recommendations, we applied a 61 item checklist derived from the NIH consensus document. We did a meta-analysis to measure pooled effect size for overall response rate (ORR) and meta-regression analyses to measure the effect of deviations from NIH recommendations on pooled effect size.

Findings: We included 82 studies related to nine interventions. Conformity to NIH recommendations was evenly low across the analysed timeframe (1998-2013), and did not change significantly after publication of NIH recommendations. The pooled effect size for ORR for systemic treatment of SR-cGVHD was 0.66 (95% CI 0.62-0.70). Increased adherence to NIH recommendations in a score of items defining correct response assessment was associated with a significant reduction in ORR (-4.2%, 95% CI -6.6 to -1.9; p=0.001). We recorded no significant association between ORR and sets of items related to correct diagnostic definition of SR-cGVHD (change in ORR -3.1%, 95% CI -7.7 to 1.5), specification of primary intervention (0, -3.8 to 3.6), or concomitant treatments (-1.6%, -5.4 to 2.3). The score of items defining correct response assessment increased after publication of NIH recommendations.

Interpretation: Our findings show evidence of bias in the reported efficacy of treatment of SR-cGVHD. The overall effect of NIH recommendations in scientific literature is scarce; however, NIH recommendations improved assessment of response, possibly reducing the overestimation bias. Better implementation of NIH recommendations might reduce false expectations about new interventions, and thus prevent clinical studies with ineffective treatments.

Funding: None.