Critical appraisal of methodological quality and completeness of reporting in Chinese social science systematic reviews with meta-analysis: A systematic review

Background: A systematic review is a type of literature review that uses rigorous methods to synthesize evidence from multiple studies on a specific topic. It is widely used in academia, including medical and social science research. Social science is an academic discipline that focuses on human behaviour and society. However, consensus regarding the standards and criteria for conducting and reporting systematic reviews in social science is lacking. Previous studies have found that the quality of systematic reviews in social science varies depending on the topic, database, and country.

Objectives: This study evaluates the completeness of reporting and methodological quality of intervention and non-intervention systematic reviews in social science in China. Additionally, we explore factors that may influence quality.

Search methods: We searched three major Chinese electronic databases-CNKI, VIP, and Wangfang-for intervention and non-intervention reviews in social science published in Chinese journals from 1 January 2009 to 2 December 2022.

Selection criteria: We included intervention and non-intervention reviews; however, we excluded overviews, qualitative syntheses, integrative reviews, rapid reviews, and evidence syntheses/summaries. We also excluded meta-analyses that used advanced methods (e.g., cross-sectional, cumulative, Bayesian, structural equation, or network meta-analyses) or that focused on instrument validation.

Data collection and analysis: We extracted data using a coding form with publication information and study content characteristics. This study conducted pilot extraction and quality assessment with four authors and formal extraction and assessment with two groups of four authors each. PRISMA2020 and MOOSE were used to evaluate the reporting completeness of intervention and non-intervention reviews. AMSTAR-2 and DART tools were adopted to assess their methodological quality. We described the characteristics of the included reviews with frequencies and percentages. We used SPSS (version 26.0) to conduct a linear regression analysis and ANOVA to explore the factors that may influence both completeness of reporting and methodological quality.

Main results: We included 1176 systematic reviews with meta-analyses published in Chinese journals between 2009 and 2022. The top three fields of publication were psychology (417, 35.5%), education (388, 33.0%), and management science (264, 22.4%). Four hundred and thirty-two intervention reviews were included. The overall completeness of reporting in PRISMA and compliance rate of the methodological process in AMSTAT-2 were 49.9% and 45.5%, respectively. Intervention reviews published in Chinese Social Science Citation Index (CSSCI) journals had lower reporting completeness than those published in non-CSSCI journals (46.7% vs. 51.1%), similar to methodological quality (39.6% vs. 47.9%). A few reviews reported the details on registration (0.2%), rationality of study selection criteria (1.6%), sources of funding for primary studies (0.2%), reporting bias assessment (2.8%), certainty of evidence assessment (1.2%), and sensitivity analysis (107, 24.8%). Seven hundred and forty-four non-intervention reviews were included. The overall completeness of reporting in MOOSE and compliance rate of the methodological process in DART were 51.8% and 50.5%, respectively. Non-intervention reviews published in CSSCI journals had higher reporting completeness than those published in non-CSSCI journals (53.3% vs. 50.3%); however, there was no difference in methodological quality (51.0% vs. 50.0%). Most reviews did not report the process and results of selection (80.8%), and 58.9% of reviews did not describe the process of data extraction; only 9.5% assessed the quality of included studies; while none of the reviews examined bias by confounding, outcome reporting bias, and loss to follow-up. An improving trend over time was observed for both intervention and non-intervention reviews in completeness of reporting and methodological quality (PRISMA: β = 0.24, p < 0.01; AMSTAR-2: β = 0.17, p < 0.01; MOOSE: β = 0.34, p < 0.01; DART: β = 0.30, p < 0.01). The number of authors and financial support also have a positive effect on quality.

Authors' conclusions: Completeness of reporting and methodological quality were low in both intervention and non-intervention reviews in Chinese social sciences, especially regarding registration, protocol, risk of bias assessment, and data and code sharing. The sources of literature, number of authors, publication year, and funding source declarations were identified as factors that may influence the quality of reviews. More rigorous standards and guidelines for conducting and reporting reviews are required in social science research as well as more support and incentives for reviewers to adhere to them.