How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal

Aidan S Weitzner; Matthew Davis; Andrew H Han; Olivia O Liu; Anuj B Patel; Brian D Sites; Steven P Cohen

doi:10.1136/rapm-2024-105490

How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal

Reg Anesth Pain Med. 2024 Jun 27:rapm-2024-105490. doi: 10.1136/rapm-2024-105490. Online ahead of print.

Authors

Aidan S Weitzner¹, Matthew Davis², Andrew H Han³, Olivia O Liu¹, Anuj B Patel⁴, Brian D Sites⁵, Steven P Cohen^{6

7

8}

Affiliations

¹ Johns Hopkins Medicine, Baltimore, Maryland, USA.
² Departments of Learning Health Sciences and Nursing, University of Michigan, Ann Arbor, Michigan, USA.
³ Department of Anesthesiology & Critical Care Medicine, Johns Hopkins Medicine, Baltimore, MD, USA.
⁴ Department of Anesthesiology, Dartmouth Health, Lebanon, New Hampshire, USA.
⁵ Departments of Anesthesiology and Orthopedics, Dartmouth College Geisel School of Medicine, Hanover, New Hampshire, USA.
⁶ Departments of Anesthesiology, Neurology, Physical Medicine & Rehabilitation, Psychiatry and Neurological Surgery, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA [email protected].
⁷ Departments of Anesthesiology & Critical Care Medicine, Neurology, Physical Medicine & Rehabilitation and Psychiatry & Behavioral Sciences, Johns Hopkins School of Medicine, Baltimore, MD, USA.
⁸ Departments of Physical Medicine & Rehabilitation and Anesthesiology, Walter Reed National Military Medical Center, Uniformed Services University of the Health Sciences, Bethesda, MD, USA.

PMID: 38942427
DOI: 10.1136/rapm-2024-105490

Abstract

Background: Peer review represents a cornerstone of the scientific process, yet few studies have evaluated its association with scientific impact. The objective of this study is to assess the association of peer review scores with measures of impact for manuscripts submitted and ultimately published.

Methods: 3173 manuscripts submitted to Regional Anesthesia & Pain Medicine (RAPM) between August 2018 and October 2021 were analyzed, with those containing an abstract included. Articles were categorized by topic, type, acceptance status, author demographics and open-access status. Articles were scored based on means for the initial peer review where each reviewer's recommendation was assigned a number: 5 for 'accept', 3 for 'minor revision', 2 for 'major revision' and 0 for 'reject'. Articles were further classified by whether any reviewers recommended 'reject'. Rejected articles were analyzed to determine whether they were subsequently published in an indexed journal, and their citations were compared with those of accepted articles when the impact factor was <1.4 points lower than RAPM's 5.1 impact factor. The main outcome measure was the number of Clarivate citations within 2 years from publication. Secondary outcome measures were Google Scholar citations within 2 years and Altmetric score.

Results: 422 articles met inclusion criteria for analysis. There was no significant correlation between the number of Clarivate 2-year review citations and reviewer rating score (r=0.038, p=0.47), Google Scholar citations (r=0.053, p=0.31) or Altmetric score (p=0.38). There was no significant difference in 2-year Clarivate citations between accepted (median (IQR) 5 (2-10)) and rejected manuscripts published in journals with impact factors >3.7 (median 5 (2-7); p=0.39). Altmetric score was significantly higher for RAPM-published papers compared with RAPM-rejected ones (median 10 (5-17) vs 1 (0-2); p<0.001).

Conclusions: Peer review rating scores were not associated with citations, though the impact of peer review on quality and association with other metrics remains unclear.

Keywords: CHRONIC PAIN; EDUCATION; OUTCOMES.