The transparency of quantitative empirical legal research published in highly ranked law journals (2018-2020): an observational study

F1000Res. 2024 Mar 7:12:144. doi: 10.12688/f1000research.127563.1. eCollection 2023.

Abstract

Background: Scientists are increasingly concerned with making their work easy to verify and build upon. Associated practices include sharing data, materials, and analytic scripts, and preregistering protocols. This shift towards increased transparency and rigor has been referred to as a "credibility revolution." The credibility of empirical legal research has been questioned in the past due to its distinctive peer review system and because the legal background of its researchers means that many often are not trained in study design or statistics. Still, there has been no systematic study of transparency and credibility-related characteristics of published empirical legal research.

Methods: To fill this gap and provide an estimate of current practices that can be tracked as the field evolves, we assessed 300 empirical articles from highly ranked law journals including both faculty-edited journals and student-edited journals.

Results: We found high levels of article accessibility (86%, 95% CI = [82%, 90%]), especially among student-edited journals (100%). Few articles stated that a study's data are available (19%, 95% CI = [15%, 23%]). Statements of preregistration (3%, 95% CI = [1%, 5%]) and availability of analytic scripts (6%, 95% CI = [4%, 9%]) were very uncommon. (i.e., they collected new data using the study's reported methods, but found results inconsistent or not as strong as the original).

Conclusion: We suggest that empirical legal researchers and the journals that publish their work cultivate norms and practices to encourage research credibility. Our estimates may be revisited to track the field's progress in the coming years.

Keywords: credibility; empirical legal research; metaresearch; open science; transparency.

Publication types

  • Observational Study

MeSH terms

  • Empirical Research
  • Humans
  • Peer Review
  • Periodicals as Topic*
  • Publications
  • Research Design

Grants and funding

Boston University School of Law and the University of Sydney School of Law provided funding for student research assistance. We received no other funding.