p-Curve and p-Hacking in Observational Research

PLoS One. 2016 Feb 17;11(2):e0149144. doi: 10.1371/journal.pone.0149144. eCollection 2016.

Abstract

The p-curve, the distribution of statistically significant p-values of published studies, has been used to make inferences on the proportion of true effects and on the presence of p-hacking in the published literature. We analyze the p-curve for observational research in the presence of p-hacking. We show by means of simulations that even with minimal omitted-variable bias (e.g., unaccounted confounding) p-curves based on true effects and p-curves based on null-effects with p-hacking cannot be reliably distinguished. We also demonstrate this problem using as practical example the evaluation of the effect of malaria prevalence on economic growth between 1960 and 1996. These findings call recent studies into question that use the p-curve to infer that most published research findings are based on true effects in the medical literature and in a wide range of disciplines. p-values in observational research may need to be empirically calibrated to be interpretable with respect to the commonly used significance threshold of 0.05. Violations of randomization in experimental studies may also result in situations where the use of p-curves is similarly unreliable.

MeSH terms

  • Humans
  • Malaria / economics
  • Malaria / epidemiology
  • Observational Studies as Topic*
  • Prevalence
  • Research*
  • Sample Size
  • Statistics as Topic*
  • Vibration

Grants and funding

The authors received no specific funding for this work.