Purpose: The primary results of phase III oncology trials may be challenging to interpret, given that results are generally based on P value thresholds. The probability of whether a treatment is beneficial, although more intuitive, is not usually provided. Here, we developed and released a user-friendly tool that calculates the probability of treatment benefit using trial summary statistics.
Methods: We curated 415 phase III randomized trials enrolling 338,600 patients published between 2004 and 2020. A phase III prior probability distribution for the treatment effect was developed on the basis of a three-component zero-mean mixture distribution of the observed z-scores. Using this prior, we computed the probability of clinically meaningful benefit (hazard ratio [HR] <0.8). The distribution of signal-to-noise ratios and power of phase III oncology trials were compared with that of 23,551 randomized trials from the Cochrane Database.
Results: The signal-to-noise ratios of phase III oncology trials tended to be much larger than randomized trials from the Cochrane Database. Still, the median power of phase III oncology trials was only 49% (IQR, 14%-95%), and the power was <80% in 65% of trials. Using the phase III oncology-specific prior, only 53% of trials claiming superiority (114 of 216) had a ≥90% probability of clinically meaningful benefits. Conversely, the probability that the experimental arm was superior to the control arm (HR <1) exceeded 90% in 17% of trials interpreted as having no benefit (34 of 199).
Conclusion: By enabling computation of contextual probabilities for the treatment effect from summary statistics, our robust, highly practical tool, now posted on a user-friendly webpage, can aid the wider oncology community in the interpretation of phase III trials.