The effect of estimating prevalences on the population-wise error rate

The population-wise error rate is a type I error rate for clinical trials with multiple target populations. In such trials, a treatment is tested for its efficacy in each population. The population-wise error rate is defined as the probability that a randomly selected, future patient will be exposed to an inefficient treatment based on the study results. It can be understood and computed as an average of strata-specific family wise error rates and involves the prevalences of these strata. A major issue of this concept is that the prevalences are usually unknown in practice, so that the population-wise error rate cannot be directly controlled. Instead, one could use an estimator based on the given sample, like their maximum-likelihood estimator under a multinomial distribution. In this article, we demonstrate through simulations that this does not substantially inflate the true population-wise error rate. We differentiate between the expected population-wise error rate, which is almost perfectly controlled, and study-specific values of the population-wise error rate which are conditioned on all subgroup sample sizes and vary within a narrow range. Thereby, we consider up to eight different overlapping populations and moderate to large sample sizes. In these settings, we also consider the maximum strata-wise family wise error rate, which is found to be, on average, at least bounded by twice the significance level used for population-wise error rate control.