Challenges of estimating treatment effects after a positive interim analysis

Background: This research investigates why a beneficial treatment effect reported at the first interim analysis (IA) may diminish at a subsequent analysis (SA). We examined three challenges in interpreting treatment effects from randomized clinical trials (RCTs) after the first positive IA: overestimation bias; non-proportional hazards; and heterogeneity in recruitment. We investigate how a penalized estimation method can address overestimation bias, and discuss additional factors to consider when interpreting positive IA results.

Methods: We identified oncology RCTs reporting positive results at the initial IA and a SA for event-free (EFS) and overall survival (OS). We modeled: (1) the hazard ratio at IA (HR_IA) versus its timing as measured by the information fraction (IF; i.e., events at IA versus total events sought); and (2), the ratio of HR_IA to HR_SA (rHR) versus the IF. This was repeated for HR_IA adjusted for overestimation bias. Examples of the other two challenges were sought.

Results: Amongst 71 RCTs, HR_IA were positively associated with the IF (slope: EFS 0.83, 95 % CI 0.44-1.22; OS 0.25, 95 % CI 0.10-0.41). HR_IA tended to exaggerate HR_SA, and more so the lower the IF (slope rHR versus IF: EFS 0.10, 95 % CI - 0.22 to 0.42; OS 0.26, 95 % CI 0.07-0.46). Adjusted HR_IA did not exaggerate HR_SA (slope rHR versus IF: EFS - 0.14, 95 % CI - 0.67 to 0.39; OS 0.02, 95 % CI - 0.26 to 0.30). Examples of two other challenges are shown.

Conclusion: Overestimation bias, non-proportional hazards, and heterogeneity in recruitment and other important treatments should be considered when communicating estimates of treatment effects from positive IAs.