The Methodological Quality of Systematic Reviews and Meta-Analyses on the Effectiveness of Non-pharmacological Cancer Pain Management

Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR

Background

Systematic reviews (SRs) in the field of neuropathic pain (NeuP) are increasingly important for decision-making. However, methodological flaws in SRs can reduce the validity of conclusions. Hence, it is important to assess the methodological quality of NeuP SRs critically. Additionally, it remains unclear which assessment tool should be used. We studied the methodological quality of SRs published in the field of NeuP and compared two assessment tools.

Methods

We systematically searched 5 electronic databases to identify SRs of randomized controlled trials of interventions for NeuP available up to March 2015. Two independent reviewers assessed the methodological quality of the studies using the Assessment of Multiple Systematic Reviews (AMSTAR) and the revised AMSTAR (R-AMSTAR) tools. The scores were converted to percentiles and ranked into 4 grades to allow comparison between the two checklists. Gwet’s AC1 coefficient was used for interrater reliability assessment.

Results

The 97 included SRs had a wide range of methodological quality scores (AMSTAR median (IQR): 6 (5–8) vs. R-AMSTAR median (IQR): 30 (26–35)). The overall agreement score between the 2 raters was 0.62 (95% CI 0.39–0.86) for AMSTAR and 0.62 (95% CI 0.53–0.70) for R-AMSTAR. The 31 Cochrane systematic reviews (CSRs) were consistently ranked higher than the 66 non-Cochrane systematic reviews (NCSRs). The analysis of individual domains showed the best compliance in a comprehensive literature search (item 3) on both checklists. The results for the domain that was the least compliant differed: conflict of interest (item 11) was the item most poorly reported on AMSTAR vs. publication bias assessment (item 10) on R-AMSTAR. A high positive correlation between the total AMSTAR and R-AMSTAR scores for all SRs, as well as for CSRs and NCSRs, was observed.

Conclusions

The methodological quality of analyzed SRs in the field of NeuP was not optimal, and CSRs had a higher quality than NCSRs. Both AMSTAR and R-AMSTAR tools produced comparable quality ratings. Our results point out to weaknesses in the methodology of existing SRs on interventions for the management NeuP and call for future improvement by better adherence to analyzed quality checklists, either AMSTAR or R-AMSTAR.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0493-y) contains supplementary material, which is available to authorized users.

Methodological and Reporting Quality of Systematic Reviews Published in the Highest Ranking Journals in the Field of Pain

BACKGROUND

Systematic reviews (SRs) are important for making clinical recommendations and guidelines. We analyzed methodological and reporting quality of pain-related SRs published in the top-ranking anesthesiology journals.

METHODS

This was a cross-sectional meta-epidemiological study. SRs published from 2005 to 2015 in the first quartile journals within the Journal Citation Reports category Anesthesiology were analyzed based on the Journal Citation Reports impact factor for year 2014. Each SR was assessed by 2 independent authors using Assessment of Multiple Systematic Reviews (AMSTAR) and Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) tools. Total score (median and interquartile range, IQR) on checklists, temporal trends in total score, correlation in total scores between the 2 checklists, and variability of those results between journals were analyzed.

RESULTS

A total of 446 SRs were included. Median total score of AMSTAR was 6/11 (IQR: 4-7) and of PRISMA 18.5/27 (IQR: 15-22). High compliance (reported in over 90% SRs) was found in only 1 of 11 AMSTAR and 5 of 27 PRISMA items. Low compliance was found for the majority of AMSTAR and PRISMA individual items. Linear regression indicated that there was no improvement in the methodological and reporting quality of SRs before and after the publication of the 2 checklists (AMSTAR: F(1,8) = 0.22; P = .65, PRISMA: F(1,7) = 0.22; P = .47). Total scores of AMSTAR and PRISMA had positive association (R = 0.71; P < .0001).

CONCLUSIONS

Endorsement of PRISMA in instructions for authors was not a guarantee of compliance. Methodological and reporting quality of pain-related SRs should be improved using relevant checklists. This can be remedied by a joint effort of authors, editors, and peer reviewers.

Recent Research Trends in Meta-analysis

The use of meta-analysis (MA), which is placed on top of the evidence hierarchy, in studies has been increasing exponentially. MA has three effect size families. Using the category of effect size families, this paper introduces the important points in the MA process and highlights the recent research trends in this field, such as network MA, meta-analytic structural equation modeling, and diagnostic test accuracy MA. Several reporting standards were established for primary studies and MA. The critical assessment reviews demonstrated that the current quality of nursing MA reporting was low. The problematic areas of the current nursing MA include study search, study selection, risk of bias, publication bias, and additional analysis based on quality assessment. Directions for future research are also presented in this paper.

Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study

OBJECTIVE

Systematic reviews (SRs) often poorly report key information, thereby diminishing their usefulness. Previous studies evaluated published SRs and determined that they failed to meet explicit criteria or characteristics. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was recommended as a reporting guideline for SR and meta-analysis (MA), but previous studies showed that adherence to the statement was not high for SRs published in different medical fields. Thus, the aims of this study are twofold: (1) to investigate the number of nursing journals that have required or recommended the use of the PRISMA statement for reporting SR, and (2) to examine the adherence of SRs and/or meta-analyses to the PRISMA statement published in nursing journals.

DESIGN

A cross-sectional study.

METHODS

Nursing journals listed in the ISI journal citation report were divided into 2 groups based on the recommendation of PRISMA statement in their 'Instruction for Authors'. SRs and meta-analyses published in 2014 were searched in 3 databases. 37 SRs and meta-analyses were randomly selected in each group. The adherence of each item to the PRISMA was examined and summarised using descriptive statistics. The quality of the SRs was assessed by Assessing the Methodological Quality of Systematic Reviews. The differences between the 2 groups were compared using the Mann-Whitney U test.

RESULTS

Out of 107 nursing journals, 30 (28.0%) recommended or required authors to follow the PRISMA statement when they submit SRs or meta-analyses. The median rates of adherence to the PRISMA statement for reviews published in journals with and without PRISMA endorsement were 64.9% (IQR: 17.6-92.3%) and 73.0% (IQR: 59.5-94.6%), respectively. No significant difference was observed in any of the items between the 2 groups.

CONCLUSIONS

The median adherence of SRs and meta-analyses in nursing journals to PRISMA is low at 64.9% and 73.0%, respectively. Nonetheless, the adherence level of nursing journals to the PRISMA statement does not significantly vary whether they endorse or recommend such a guideline.