Design analysis indicates Potential overestimation of treatment effects in randomized controlled trials supporting Food and Drug Administration cancer drug approvals

J Clin Epidemiol. 2018 Nov:103:1-9. doi: 10.1016/j.jclinepi.2018.06.012. Epub 2018 Jul 2.

Abstract

Objective: Statistical significance drives interpretation of randomized controlled trials (RCTs). We examined the type S error risk-claiming a new drug is falsely beneficial-and exaggeration ratio-how estimated effects differ from true effects-to re-emphasize direction and magnitude of treatment effects.

Study design and setting: We systematically reviewed RCTs supporting Food and Drug Administration (FDA) approval of cancer drugs between 2007 and 2016. We extracted data for overall survival (OS), progression-free survival (PFS), and response outcomes from FDA reviews. We estimated type S error risks and exaggeration ratios by considering replicated RCTs of equal size and a range of true effects.

Results: We analyzed 42 RCTs for 39 approved drugs. Across 38 RCTs reporting OS, the median type S error risk was 0.00% (Q1-Q3, 0.00-0.01%) and 3.56% (0.40-6.74%), for true hazard ratios of 0.7 and 0.9, respectively, indicating confidence in effect direction. The corresponding exaggeration ratios were 1.09 (1.01-1.11) and 1.30 (1.13-1.42), indicating median overestimations of 9% and 30%. Similar results held for PFS and response outcomes.

Conclusions: The type S error risk and exaggeration ratio provide additional insights into the replicability of RCTs. Our analyses also quantify the winner's curse, in which pivotal RCTs tend toward overoptimism.

Keywords: Bias; Disease-free survival; Drug approval; Randomized controlled trials; Reproducibility of results; Statistical data interpretation.

Publication types

  • Research Support, N.I.H., Extramural

MeSH terms

  • Antineoplastic Agents / pharmacology*
  • Bias
  • Data Interpretation, Statistical
  • Drug Approval / statistics & numerical data
  • Humans
  • Neoplasms / drug therapy
  • Outcome Assessment, Health Care* / methods
  • Outcome Assessment, Health Care* / standards
  • Outcome Assessment, Health Care* / statistics & numerical data
  • Proportional Hazards Models
  • Randomized Controlled Trials as Topic* / methods
  • Randomized Controlled Trials as Topic* / statistics & numerical data
  • Reproducibility of Results
  • Vereinigte Staaten
  • United States Food and Drug Administration / standards

Substances

  • Antineoplastic Agents