High replicability of newly discovered social-behavioural findings is achievable

John Protzko; Jon Krosnick; Leif Nelson; Brian A Nosek; Jordan Axt; Matt Berent; Nicholas Buttrick; Matthew DeBell; Charles R Ebersole; Sebastian Lundmark; Bo MacInnis; Michael O'Donnell; Hannah Perfecto; James E Pustejovsky; Scott S Roeder; Jan Walleczek; Jonathan W Schooler

doi:10.1038/s41562-023-01749-9

High replicability of newly discovered social-behavioural findings is achievable

Nat Hum Behav. 2024 Feb;8(2):311-319. doi: 10.1038/s41562-023-01749-9. Epub 2023 Nov 9.

Authors

John Protzko^{1

2}, Jon Krosnick³, Leif Nelson⁴, Brian A Nosek^{5

6}, Jordan Axt⁷, Matt Berent⁸, Nicholas Buttrick⁹, Matthew DeBell³, Charles R Ebersole⁶, Sebastian Lundmark¹⁰, Bo MacInnis³, Michael O'Donnell¹¹, Hannah Perfecto¹², James E Pustejovsky¹³, Scott S Roeder¹⁴, Jan Walleczek¹⁵, Jonathan W Schooler¹⁶

Affiliations

¹ Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA. [email protected].
² Department of Psychological Science, Central Connecticut State University, New Britain, CT, USA. [email protected].
³ Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA.
⁴ Haas School of Business, University of California, Berkeley, Berkeley, CA, USA.
⁵ Center for Open Science, Charlottesville, VA, USA.
⁶ Department of Psychology, University of Virginia, Charlottesville, VA, USA.
⁷ Department of Psychology, McGill University, Montreal, Quebec, Canada.
⁸ Matt Berent Consulting, Sharon, PA, USA.
⁹ Department of Psychology, University of Wisconsin-Madison, Madison, WI, USA.
¹⁰ SOM Institute, University of Gothenburg, Gothenburg, Sweden.
¹¹ McDonough School of Business, Georgetown University, Washington, DC, USA.
¹² Olin School of Business, Washington University in St. Louis, St. Louis, MO, USA.
¹³ Educational Psychology Department, University of Wisconsin-Madison, Madison, WI, USA.
¹⁴ Darla Moore School of Business, University of South Carolina, Columbia, SC, USA.
¹⁵ Phenoscience Laboratories, Berlin, Germany.
¹⁶ Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.

Abstract

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

Publication types

Retracted Publication

MeSH terms

Humans
Prospective Studies
Reproducibility of Results
Research Design*
Sample Size
Social Behavior*