Methodological quality, completeness of reporting and use of systematic reviews as evidence in clinical practice guidelines for paediatric overweight and obesity

Reliability and reproducibility of systematic reviews informing the 2020-2025 Dietary Guidelines for Americans: a pilot study

BACKGROUND

Although high-quality nutrition systematic reviews (SRs) are important for clinical decision making, there remains debate on their methodological quality and reporting transparency.

OBJECTIVES

The objective of this study was to assess the reliability and reproducibility of a sample of SRs produced by the Nutrition Evidence Systematic Review (NESR) team to inform the 2020-2025 Dietary Guidelines for Americans (DGAs).

METHODS

We evaluated a sample of 8 SRs from the DGA dietary patterns subcommittee for methodological quality using the Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) tool and for reporting transparency using the PRISMA 2020 and PRISMA literature search extension (PRISMA-S) checklists. We assessed the quality and reproducibility of the original search strategy of one selected SR using the Peer Review of Electronic Search Strategies checklist. The reporting transparency of the SR's narrative data synthesis was assessed using the Synthesis Without Meta-Analysis (SWiM) checklist. Interpretation bias was evaluated using existing spin bias classifications in systematic reviews.

RESULTS

The AMSTAR 2 assessment identified critical methodological weaknesses, and all included SRs were judged to be of critically low quality. Overall, 74% of the PRISMA 2020 checklist items and 63% of the PRISMA-S checklist items were satisfactorily fulfilled. We identified several errors and inconsistencies in the search strategy and could not reproduce searches within a 10% margin of the original results. The SWiM assessment identified concerns regarding the reporting transparency of the narrative data synthesis, but the spin bias assessment revealed no evidence of interpretation bias.

CONCLUSIONS

Several methodological quality and reporting concerns were identified, which could lead to reliability and reproducibility issues should a full reproduction attempt be made. However, additional research is needed to confirm the impact of these findings on conclusions statements and their generalizability across the NESR team SRs. This study was registered in the Open Science Framework (https://osf.io/ns6a9/).

Heterogeneous Outcome Selection and Incomplete Prespecification of Outcomes in Systematic Reviews: A Case Study on Pressure Injury

OBJECTIVE

To understand how reviewers select and prespecify outcomes for systematic reviews (SRs), the authors report on the outcomes used in SRs of pressure injury (PI) intervention and treatment and evaluate their completeness of prespecification.

DATA SOURCES

The authors searched four electronic databases for SRs involving PI prevention and/or treatments.

STUDY SELECTION

Inclusion criteria were SRs and meta-analyses evaluating interventions for preventing or treating PI. Studies without systematic search or risk-of-bias assessment, conference proceedings, and articles not in Chinese or English were excluded.

DATA EXTRACTION

Two reviewers extracted and categorized the outcomes in domains, assessing outcome prespecification using a five-element framework. Data items included study characteristics, target population, type of interventions, and outcome variables.

DATA SYNTHESIS

This review included 95 SRs that reported a total of 432 instances of 24 different outcome domains. An average of four outcome domains were reported per SR. The most frequently reported domains were PI healing, PI occurrence, and PI status. Of the 62 SRs that prespecified primary outcomes, 40 (64.52%) reported more than one primary outcome. Only 24 of the 432 instances (5.56%) were completely specified. Among the 24 outcome domains, 12 (50.00%) were listed as primary outcomes at least once. Primary outcomes were more completely specified than nonprimary outcomes.

CONCLUSIONS

Systematic reviews of PI prevention and/or treatment report diverse, incompletely prespecified outcomes, highlighting the need for a core outcome set to standardize key clinical outcomes.

The Reporting and Methodological Quality of Systematic Reviews Underpinning Clinical Practice Guidelines Focused on the Management of Cutaneous Melanoma: Cross-Sectional Analysis

BACKGROUND

Clinical practice guidelines (CPGs) inform evidence-based decision-making in the clinical setting; however, systematic reviews (SRs) that inform these CPGs may vary in terms of reporting and methodological quality, which affects confidence in summary effect estimates.

OBJECTIVE

Our objective was to appraise the methodological and reporting quality of the SRs used in CPGs for cutaneous melanoma and evaluate differences in these outcomes between Cochrane and non-Cochrane reviews.

METHODS

We conducted a cross-sectional analysis by searching PubMed for cutaneous melanoma guidelines published between January 1, 2015, and May 21, 2021. Next, we extracted SRs composing these guidelines and appraised their reporting and methodological rigor using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and AMSTAR (A Measurement Tool to Assess Systematic Reviews) checklists. Lastly, we compared these outcomes between Cochrane and non-Cochrane SRs. All screening and data extraction occurred in a masked, duplicate fashion.

RESULTS

Of the SRs appraised, the mean completion rate was 66.5% (SD 12.29%) for the PRISMA checklist and 44.5% (SD 21.05%) for AMSTAR. The majority of SRs (19/50, 53%) were of critically low methodological quality, with no SRs being appraised as high quality. There was a statistically significant association (P<.001) between AMSTAR and PRISMA checklists. Cochrane SRs had higher PRISMA mean completion rates and higher methodological quality than non-Cochrane SRs.

CONCLUSIONS

SRs supporting CPGs focused on the management of cutaneous melanoma vary in reporting and methodological quality, with the majority of SRs being of low quality. Increasing adherence to PRISMA and AMSTAR checklists will likely increase the quality of SRs, thereby increasing the level of evidence supporting cutaneous melanoma CPGs.

What Are the Determinants of the Quality of Systematic Reviews in the International Journals of Occupational Medicine? A Methodological Study Review of Published Literature

Objective: The aim of this study was to evaluate the methodological quality of systematic reviews published in occupational medicine journals from 2014 to 2021. Methods: Papers edited between 2014 and 2021 in the 14 open access journals with the highest impact were assessed for their quality. Studies were included if they were systematic reviews and meta-analyses, and if they were published in English. Results: The study included 335 studies. Among these, 149 were meta-analyses and 186 were systematic reviews. The values of the AMSTAR-2 score range between three and fourteen with a mean value of 9.85 (SD = 2.37). The factors that significantly and directly associate to a higher AMSTAR-2 score were impact factor (p = 0.003), number of consulted research databases (p = 0.011), declaration of PRISMA statement (p = 0.003), year of publication (p < 0.001) and performing a meta-analysis (p < 0.001).The R² values from the multivariate analysis showed that the AMSTAR-2 score could be predicted by the inclusion of these parameters by up to 23%. Conclusions: This study suggests a quality assessment methodology that could help readers in a fast identification of good systematic reviews or meta-analyses. Future studies should analyze more journals without applying language restrictions and consider a wider range of years of publication in order to give a more robust evidence for results.

Quality of reporting among systematic reviews underpinning the ESC/ACC guidelines on ventricular arrhythmias and sudden cardiac death

OBJECTIVES

The main objective of this study was to assess the methodological and reporting quality of the systematic reviews (SRs) supporting the European Society of Cardiology (ESC) and the American College of Cardiology (ACC) clinical practice guidelines (CPGs) recommendations for the management of patients with ventricular arrhythmias and sudden cardiac death (SCD). As a secondary objective, we sought to determine: (1) the proportion of Cochrane SRs were cited; and (2) whether Cochrane SRs scored higher on Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR-2) appraisals.

DESIGN

Cross-sectional analysis.

MAIN OUTCOME MEASURES

We searched for CPGs published by the ESC and the ACC from 2010 to 2020. We selected the CPGs for ventricular arrhythmias and the prevention of SCD. The reference sections were searched for SRs. Two independent investigators evaluated eligible SR using the PRISMA checklist and the AMSTAR-2 assessment tool.

RESULTS

Two CPGs for ventricular arrhythmia and SCD were included in this study. Fifty-five SRs were included in our analysis. Across all SRs, the mean PRISMA score was 0.70. The lowest scoring PRISMA item related to the presence of a pre-published protocol (item 5, score 0.17). Overall, 40% of included SRs were found to have 'critically low' AMSTAR-2 ratings. One of the lowest scoring items for AMSTAR-2 was reporting of sources of funding (item 10). The 4 Cochrane SRs that were included scored higher on both assessment tools than non-Cochrane studies, specifically in PRISMA overall completion (88.7% vs 69.7%).

CONCLUSION

Our study suggests the methodological and reporting quality of SRs used within ESC and ACC CPGs is insufficient, as demonstrated by the lack of adherence to both AMSTAR-2 and PRISMA checklists. Given the importance of CPGs on clinical decision making, and ultimately patient care, the methodological rigour and quality reporting within SRs used in CPGs should be held to the highest standard within the field of cardiology.

Clinical practice guidelines for the prevention of childhood obesity: A systematic review of quality and content

Obesity in childhood is a significant global issue, and prevention is key to reducing prevalence. Healthcare providers can play an important role in the prevention of obesity. The aim of this systematic review was to identify and evaluate clinical practice guidelines (CPGs) for preventing childhood obesity with a focus on the role of medical doctors. Peer-reviewed literature and gray literature sources were searched for CPGs published from 2010 to 2021. Eleven CPGs were identified. Quality was evaluated using the Appraisal of Guidelines for Research and Evaluation Collaboration (AGREE II) instrument; seven CPGs were higher quality and four lower quality. Recommendations within the CPGs covered three main areas: growth monitoring, maintaining a healthy weight, and managing overweight. The importance of involving the whole family and healthy lifestyle behaviors was emphasized. The majority of the CPGs rated poorly in guideline applicability highlighting the need for practical implementation tools. Although our review identified a number of CPGs relevant to the prevention of obesity for doctors working with children and their families, more research is needed to produce high-quality meaningful and applicable CPGs to maximize uptake, implementation, and ultimately, benefit to children and their families.

Analysis of the Evidence Underpinning the American Academy of Orthopedic Surgeons Knee Osteoarthritis Clinical Practice Guidelines

CONTEXT

Clinical practice guidelines (CPGs) are vital to establishing a standardized and evidence-based approach in medicine. These guidelines rely on the use of methodologically sound clinical trials, and the subsequent reporting of their methodology.

OBJECTIVE

To evaluate the completeness of randomized controlled trials (RCTs) underpinning CPGs published by the American Academy of Orthopedic Surgeons (AAOS) for management of osteoarthritis of the knee.

DATA SOURCES

We searched the most recent AAOS CPGs for surgical and nonsurgical management of osteoarthritis of the knee for RCTs. To estimate the necessary sample size, we performed a power analysis using OpenEpi 3.0 (openepi.com).

STUDY SELECTION

Two authors independently screened the reference sections of the included CPGs. Included studies met the definition of an RCT, were retrievable in the English language, and were cited in at least one of the included CPGs.

STUDY DESIGN

Meta-Analysis.

LEVEL OF EVIDENCE

Level 1a.

DATA EXTRACTION

We performed double-blind screening and extraction of RCTs included in the AAOS CPGs. We evaluated each RCT for adherence to the Consolidated Standards of Reporting Trials (CONSORT) 2010 checklist. A multiple regression analysis was conducted to assess CONSORT adherence against characteristics of included studies (ie, type of intervention, funding source, etc).

RESULTS

Our study included 179 RCTs. The overall adherence was 68.5% with significant differences between those published before and since the development of the 2010 CONSORT guidelines (P = 0.02). We found that RCTs receiving funding from industry/private sources as well as studies that included a conflict of interest statement showed more completeness than RCTs that reported receiving no funding (P < 0.01).

CONCLUSION

We found suboptimal CONSORT adherence for RCTs cited in AAOS CGPs for management of osteoarthritis of the knee. Therefore, the CPGs are likely supported by outdated evidence and lack of high-quality reporting. It is important that evidence used to guide clinical decision making be of the highest quality in order to optimize patient outcomes. In order for clinicians to confer the greatest benefits to their patients, CPGs should provide the totality of evidence and emphasize emerging high-quality RCTs to ensure up-to-date, evidence-based clinical decision-making.

Alcohol use disorder: An analysis of the evidence underpinning clinical practice guidelines

BACKGROUND

As Clinical Practice Guidelines (CPGs) provide effective guidance for providing medical care for individuals with alcohol use disorder (AUD), the evidence behind them should be robust.

OBJECTIVE

Our primary objective was to critically appraise the methodological and reporting quality of systematic reviews cited within CPGs regarding the treatment of AUD. Our secondary objective was to determine how frequently Cochrane Reviews were cited as justification and to evaluate appraisals between Cochrane and non-Cochrane reviews.

METHODS

We searched PubMed to identify CPGs for the treatment of AUD published between 2015 and 2021. Systematic reviews included in each CPG were evaluated using the Preferred Reporting Instrument for Systematic Reviews and Meta-Analyses (PRISMA) and a validated quality assessment tool (AMSTAR-2). Additional study characteristics were recorded.

RESULTS

From the screening process, 98 systematic reviews from 6 CPGs met inclusion criteria. PRISMA adherence ranged from 72% to 85% (mean of 79%). AMSTAR-2 adherence ranged from 52% to 73% (mean of 68%). AMSTAR appraisal ratings found 32 (35.6%) critically low, 10 (11.1%) low, 35 (38.9%) moderate, and only 13 (14.4%) high systematic reviews. Cochrane systematic reviews displayed greater PRISMA (0.92 vs. 0.75: p < 0.001) and AMSTAR-2 (0.90 vs. 0.61.; p < 0.001) scores compared to the non-Cochrane studies.

CONCLUSION

Systematic reviews included in CPGs for AUD treatment showed variable adherence to PRISMA and AMSTAR-2 guidelines, with almost half of the systematic reviews being critically low to low methodological quality. Given the prevalence of alcohol use disorder, methodological and reporting quality recommendations are important to strengthening evidence informing CPGs.

An Evaluation of Evidence Underpinning Management Recommendations in Tobacco Use Disorder Clinical Practice Guidelines

INTRODUCTION

Clinical practice guidelines(CPGs) are important tools for medical decision-making. Given the high prevalence and financial burden associated with tobacco use disorder(TUD), it is critical that recommendations within CPGs are based on robust evidence. Systematic reviews(SRs) are considered the highest level of evidence, thus, we evaluated the quality of SRs underpinning CPG recommendations for TUD.

METHODS

We used PubMed to search for CPGs relating to TUD published between January 1, 2010 and May 21, 2021. SRs were extracted from CPG references and evaluated using Preferred Reporting Items for Systematic Reviews and Meta-Analyses(PRISMA) and A MeaSurement Tool to Assess Systematic Reviews(AMSTAR-2) tools. We then compared SRs conducted by the Cochrane Collaboration with non-Cochrane SRs using a Mann-Whitney U test and determined associations between PRISMA and AMSTAR-2 extracted characteristics using multiple regression.

RESULTS

Our search generated 10 CPGs with 98 SRs extracted. Mean PRISMA completion was 74.7%(SD = 16.7) and mean AMSTAR-2 completion was 53.8%(SD = 22.0) across all guidelines. Cochrane SRs were more complete than non-Cochrane studies in the PRISMA and AMSTAR-2 assessments. The regression model showed a statistically significant association between PRISMA completion and AMSTAR-2 rating, with those classified as "low" or "moderate" quality having higher PRISMA completion than those with "critically low" ratings.

CONCLUSION

We found substandard adherence to PRISMA and AMSTAR-2 checklists across SRs cited in TUD CPGs. A lack of recent SRs in CPGs could lead to outdated recommendations. Therefore, frequent guideline updates with recently published evidence may ensure more accurate clinical recommendations and improve patient care.

IMPLICATIONS

Systematic reviews used to underpin clinical practice guideline recommendations influence treatment decisions and, ultimately, patient outcomes. We found that many systematic reviews underpinning tobacco use disorder guideline recommendations were out of date and unsatisfactory in reporting and quality. Thus, including newer systematic reviews containing more recently conducted trials and better reporting could alter recommendations and improve the rate of successful tobacco cessation attempts.

Tonsillar-related pathologies: An analysis of the evidence underpinning management recommendations

OBJECTIVE

Evidence-based decision making is crucial in reducing the health and economic burdens imposed by tonsillar-related pathologies. Clinical practice guidelines are used to guide these decisions; however, uptake of recommendations in these guidelines is low. Systematic reviews are the highest level of evidence used to influence guideline recommendations; therefore, improving the reporting and methodological quality of systematic reviews related to tonsillar-related pathologies may improve guideline uptake and patient care.

METHODS

We used PubMed to search for all clinical practice guidelines related to tonsillar-related pathologies from 2010 to 2020. Included guidelines were then searched for all systematic reviews and meta-analyses. Study characteristics were extracted from each cited systematic review/meta-analysis before being evaluated using the PRISMA (Preferred Reporting Instrument for Systematic Reviews and Meta-Analyses) and AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews 2) instruments. We then compared systematic reviews conducted by a Cochrane systematic review group with non-Cochrane systematic reviews.

RESULTS

Seven clinical practice guidelines were included in our study and within these guidelines 98 SRs/MAs were cited, 80 of which were unique and included. Systematic reviews composed 9.1% (98/1082) of all guideline citations. Guideline PRISMA scores ranged from 0.47 to 0.83 with a mean score of 0.71 (n = 80) and guideline AMSTAR-2 scores ranged from 0.52 to 0.83 with a mean of 0.56 (7.29/13) and 0.75 (11.94/16) (n = 80). Cochrane systematic reviews displayed greater PRISMA (0.88 vs. 0.64: p < 0.001) and AMSTAR-2 (0.90 vs. 0.57; p < 0.001) scores compared to the non-Cochrane studies. We found PRISMA and AMSTAR-2 scores were positively correlated across guidelines (r = 0.93).

CONCLUSION

Wide variation exists in adherence to PRISMA and AMSTAR-2 guidelines among systematic reviews cited in clinical practice guidelines for tonsillar-related pathologies. Prior registration and adequate risk of bias assessment are two areas where improvements may be needed. Given the importance of guideline uptake, careful considerations to improve the methodological and reporting quality of evidence supporting tonsillar-related pathology recommendations are necessary.

Methodological quality and reporting standards in systematic reviews with meta-analysis of physical activity studies: a report from the Strengthening the Evidence in Exercise Sciences Initiative (SEES Initiative)

BACKGROUND

Several resources have been developed (e.g., reporting guidelines) to promote high-standard practices in health research. However, there was no continuous and systematic assessment of recommended practices in published systematic reviews with meta-analysis (SRMAs), which increases the usability of the available resources. Therefore, we aimed to assess the methodological and reporting standards in SRMAs of physical activity studies. This report presents the main results of the SEES Initiative in 2019.

METHODS

Our approach is based on a prospective systematic review methodology to implement post-publication surveillance of research practices in exercise sciences. Briefly, during the year 2019, pre-specified searches were conducted monthly (PubMed/MEDLINE) in journals from the exercise sciences (n = 9) and medicine (n = 5). The assessments were independently conducted by two authors, based on 36 items/practices derived from established statements/tools (PRISMA, AMSTAR 2, ROBIS). To be eligible, SRMAs should summarize studies that had, at least, one arm consisting of physical activity interventions/exposures and one health or behavioral outcome.

RESULTS

Out of 1028 studies assessed for eligibility, 103 SRMAs were included. The minimum adherence was 13/36 items, whereas only one SRMA adhered to all items. Some highly contemplated items included identification of title as SRMA (97.1%) and descriptions of the main outcome in the abstract (95.1%) and risk of bias (RoB) assessment (95.1%). Some poorly contemplated items included publicly available protocol (4.9%), discussion of the results in light of RoB in studies included (32.0%), and data sharing statements (35.9%).

CONCLUSION

In summary, there is a suboptimal adherence to recommended practices on methodological quality and reporting standards in the SRMAs of physical activity intervention/exposure evaluated from the selected journals in 2019, which likely reduce the reproducibility and usefulness of these studies. This incipient evidence from our first 12 months of post-publication surveillance should serve as a call for attention and action for multiple stakeholders (e.g., authors, reviewers, editors, funders, academic institutions) in this important health research field.

An analysis of the evidence underpinning the national comprehensive cancer network practice guidelines

OBJECTIVE

This study assesses the quality and completeness of systematic reviews (SRs) included by the National Comprehensive Cancer Network (NCCN) cancer screening clinical practice guidelines (CPGs).

METHODS

We evaluated SRs according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and AMSTAR-2 (A Measurement Tool to Assess systematic Reviews).

RESULTS

Seven NCCN CPGs were included with 109 SRs. The mean PRISMA percent completeness of included SRs was 71 % (range 0.1-1.0). The mean AMSTAR-2 percent completeness was 56 % (range 0.05-0.99). Of the 70 SRs assessed via AMSTAR-2, 42 (60 %) received a "critically low" rating, 11 (15.7 %) received "low" ratings, and 17 (24.3 %) received "moderate". None of the SRs received a "high" rating.

CONCLUSION

Lack of adherence to AMSTAR-2 and PRISMA reporting standards among the SRs included is prevalent. We suggest improved reporting of SR inclusion criteria and evaluation to bolster the reporting quality of SRs underpinning CPG recommendations.

Methodological quality of systematic reviews comprising clinical practice guidelines for cardiovascular risk assessment and management for noncardiac surgery

BACKGROUND

Cardiac assessment in noncardiac surgery clinical practice guidelines should be supported by the highest-quality evidence such as that offered by systematic reviews. Currently, the methodological and reporting quality of these studies remains unknown.

METHODS

We used PubMed to search for all clinical practice guidelines related to perioperative cardiovascular patients undergoing noncardiac surgery from 2010 to 2021. The included clinical practice guidelines were analysed for all systematic reviews and meta-analyses. The primary objective of this study was to determine reporting and methodological quality using the PRISMA (Preferred Reporting Instrument for Systematic Reviews and Meta-Analyses) and AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews-2) instruments. Our secondary objective was to compare systematic reviews conducted by the Cochrane Collaboration with non-Cochrane studies.

RESULTS

Three clinical practice guidelines were included in our study. Within these, 78 systematic reviews were included. PRISMA completion ranged from 34.8% to 100.0% with a mean of 76.9%. AMSTAR-2 completion ranged from 15.6% to 96.9% with a mean of 58.0%. Fifty-four systematic reviews underpinned a clinical practice guidelines recommendation, of which 25 were rated 'critically low' by AMSTAR-2 appraisal. Cochrane systematic reviews typically performed better than non-Cochrane studies, but were a minority of the included studies (10/78).

CONCLUSION

We found deficiencies in several key areas regarding the methodological and reporting qualities of systematic reviews included in cardiac assessment in noncardiac surgery clinical practice guidelines. As these clinical practice guidelines are instrumental to clinical decision-making and patient care in cardiac assessment in noncardiac surgery, we advocate for improved reporting quality among systematic reviews cited as supportive evidence for these recommendations.

Clinical practice guidelines for the nutritional risk screening and assessment of cancer patients: a systematic quality appraisal using the AGREE II instrument

PURPOSE

To evaluate the quality of published clinical practice guidelines (CPGs) regarding the nutritional risk screening and assessment of cancer patients and to identify high-quality CPGs for clinical healthcare professionals.

METHODS

Guidelines for the nutritional risk screening and assessment of cancer patients were comprehensively searched in eight electronic databases, including The Lancet, PubMed, Cochrane Library, Excerpta Medica dataBASE (EMBASE), Web of Science, China National Knowledge Infrastructure (CNKI), China Biology Medicine disc (CBMdisc), and Wan Fang Data, through August 2020. Six relevant guideline databases, including the National Comprehensive Cancer Network (NCCN), the National Guideline Clearinghouse (NGC), the Guideline International Network (GIN), the New Zealand Guidelines Group (NZGG), the China Guideline Clearinghouse (CGC), and Medlive, and relevant nutrition society websites, were also searched through August 2020. The methodological quality of the included CPGs was appraised independently by three assessors using the Appraisal of Guidelines for Research and Evaluation, 2nd edition (AGREE II) tool.

RESULTS

Seven CPGs were located, and the domain with the highest percentage was "clarity of presentation" (85.44%), while the domain with the lowest percentage was "applicability" (40.26%). From the AGREE II results, two guidelines were rated as "strongly recommended," three were assessed as "recommended with modifications," and two were deemed as "not recommended."

CONCLUSION

Considering that the two "strongly recommended" guidelines were developed within the American and European contexts, translation, validation, and cultural adaptation are recommended prior to implementing these guidelines in other countries or healthcare contexts to improve their effectiveness and sensitivity for local cancer patients.

TRIAL REGISTRATION

PROSPERO registration of the study protocol: CRD42020177390 (July 5, 2020).

Risk of Bias and Quality of Reporting in Colon and Rectal Cancer Systematic Reviews Cited by National Comprehensive Cancer Network Guidelines

INTRODUCTION

Given the changing landscape of colorectal cancer, systematic reviews are likely to play a key role in advancing the understanding of prevention, diagnosis, and treatment.

METHODS

We conducted a cross-sectional investigation of the risk of bias and reporting quality of systematic reviews referenced by colon and rectal cancer National Comprehensive Cancer Network (NCCN) guidelines. We used two widely accepted tools: Risk of Bias in Systematic reviews (ROBIS) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).

RESULTS

Using ROBIS, only 3 (4.8%) systematic reviews were judged with low risk of bias, 35 (55.6%) systematic reviews were judged with unclear risk of bias, and 25 (39.7%) systematic reviews were judged with high risk of bias. Across all systematic reviews, the individual bias domains at the highest risk of bias were domains 1 (protocol and eligibility criteria) and 2 (methods to identify and select studies). Across all studies, the median adherence to PRISMA was 74.1% (IQR 69.2-80.0%), corresponding to approximately 20 of 27 items.

CONCLUSIONS

Systematic reviews cited in NCCN guidelines for colon and rectal cancer are frequently at unclear or high risk of bias and do not report key systematic review items that are important for the critical appraisal of results.

Evaluation of the Level of Evidence Supporting the Recommendations Constituting the American Academy of Dermatology Clinical Practice Guidelines: Cross-Sectional Analysis

Background

Clinical practice guidelines are evidence-based recommendations used by physicians to improve patient care. These guidelines provide the physician with an assessment of the benefits and harms of a treatment and its alternatives. Therefore, it is essential that the clinical practice guidelines be based on the strongest available evidence. Numerous studies in a variety of different fields of medicine have demonstrated that recommendations supported by weak evidence are a common theme in clinical practice guidelines. A clinical guideline based solely on weak evidence has the capability to reduce the quality of care provided by physicians.

Objective

Our primary objective is to evaluate the levels of evidence supporting the recommendations constituting the American Academy of Dermatology clinical practice guidelines.

Methods

Using a cross-sectional study design, authors SM and RO located all current clinical practice guidelines on the American Academy of Dermatology website on June 10, 2017, and December 11, 2019. Each recommendation and its corresponding evidence rating were extracted in a duplicate and blinded fashion. A consensus meeting was planned a priori to resolve disagreements in extractions or stratifications.

Results

In total, 6 clinical guidelines and their subsections were screened and 899 recommendations were identified. Our final data set included 841 recommendations, as 58 recommendations contained no level of evidence and were excluded from calculations. Many recommendations were supported by a moderate level of evidence and therefore received a B rating (346/841, 41.1%). Roughly one-third of the recommendations were supported by a strong level of evidence and were given an A rating (n=307, 36.5%). The clinical practice guideline with the highest overall strength of evidence was regarding the treatment of acne, which had 17 of 35 (48.6%) recommendations supported by strong evidence and only 2 (5.7%) supported by weak evidence. The clinical practice guideline with the fewest recommendations supported by strong evidence was melanoma (13/63, 20.6%).

Conclusions

Clinical practice guidelines that lack strong supporting evidence could negatively affect patient care, and dermatologists should be mindful that not all recommendations are supported by the strongest level of evidence. An increased quantity of quality research needs to be performed in the field of dermatology to improve the evidence supporting the American Academy of Dermatology clinical practice guidelines.

Methodological and Reporting Quality of Systematic Reviews and Meta-analyses in Endodontics

INTRODUCTION

The aim of this systematic review (SR) was to evaluate the quality of SRs and meta-analyses (MAs) in endodontics.

METHODS

A comprehensive literature search was conducted to identify relevant articles in the electronic databases from January 2000 to June 2017. Two reviewers independently assessed the articles for eligibility and data extraction. SRs and MAs on interventional studies with a minimum of 2 therapeutic strategies in endodontics were included in this SR. Methodologic and reporting quality were assessed using A Measurement Tool to Assess Systematic Reviews (AMSTAR) and Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA), respectively. The interobserver reliability was calculated using the Cohen kappa statistic. Statistical analysis with the level of significance at P < .05 was performed using Kruskal-Wallis tests and simple linear regression analysis.

RESULTS

A total of 30 articles were selected for the current SR. Using AMSTAR, the item related to the scientific quality of studies used in conclusion was adhered by less than 40% of studies. Using PRISMA, 3 items were reported by less than 40% of studies, which were on objectives, protocol registration, and funding. No association was evident comparing the number of authors and country with quality. Statistical significance was observed when quality was compared among journals, with studies published as Cochrane reviews superior to those published in other journals. AMSTAR and PRISMA scores were significantly related.

CONCLUSIONS

SRs in endodontics showed variability in both methodologic and reporting quality.

Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review

BACKGROUND

The PRISMA Statement is a reporting guideline designed to improve transparency of systematic reviews (SRs) and meta-analyses. Seven extensions to the PRISMA Statement have been published to address the reporting of different types or aspects of SRs, and another eight are in development. We performed a scoping review to map the research that has been conducted to evaluate the uptake and impact of the PRISMA Statement and extensions. We also synthesised studies evaluating how well SRs published after the PRISMA Statement was disseminated adhere to its recommendations.

METHODS

We searched for meta-research studies indexed in MEDLINE® from inception to 31 July 2017, which investigated some component of the PRISMA Statement or extensions (e.g. SR adherence to PRISMA, journal endorsement of PRISMA). One author screened all records and classified the types of evidence available in the studies. We pooled data on SR adherence to individual PRISMA items across all SRs in the included studies and across SRs published after 2009 (the year PRISMA was disseminated).

RESULTS

We included 100 meta-research studies. The most common type of evidence available was data on SR adherence to the PRISMA Statement, which has been evaluated in 57 studies that have assessed 6487 SRs. The pooled results of these studies suggest that reporting of many items in the PRISMA Statement is suboptimal, even in the 2382 SRs published after 2009 (where nine items were adhered to by fewer than 67% of SRs). Few meta-research studies have evaluated the adherence of SRs to the PRISMA extensions or strategies to increase adherence to the PRISMA Statement and extensions.

CONCLUSIONS

Many studies have evaluated how well SRs adhere to the PRISMA Statement, and the pooled result of these suggest that reporting of many items is suboptimal. An update of the PRISMA Statement, along with a toolkit of strategies to help journals endorse and implement the updated guideline, may improve the transparency of SRs.

Inconsistent selection of outcomes and measurement devices found in shoulder arthroplasty research: An analysis of studies on ClinicalTrials.gov

Introduction

Recent evidence suggests a lack of standardization of shoulder arthroplasty outcomes. This issue is a limiting factor in systematic reviews. Core outcome set (COS) methodology could address this problem by delineating a minimum set of outcomes for measurement in all shoulder arthroplasty trials.

Methods

A ClinicalTrials.gov search yielded 114 results. Eligible trials were coded on the following characteristics: study status, study type, arthroplasty type, sample size, measured outcomes, outcome measurement device, specific metric of measurement, method of aggregation, outcome classification, and adverse events.

Results

Sixty-six trials underwent data abstraction and data synthesis. Following abstraction, 383 shoulder arthroplasty outcomes were organized into 11 outcome domains. The most commonly reported outcomes were shoulder outcome score (n = 58), pain (n = 33), and quality of life (n = 15). The most common measurement devices were the Constant-Murley Shoulder Outcome Score (n = 38) and American Shoulder and Elbow Surgeons Shoulder Score (n = 33). Temporal patterns of outcome use was also found.

Conclusion

Our study suggests the need for greater standardization of outcomes and instruments. The lack of consistency across trials indicates that developing a core outcome set for shoulder arthroplasty trials would be worthwhile. Such standardization would allow for more effective comparison across studies in systematic reviews, while at the same time consider important outcomes that may be underrepresented otherwise. This review of outcomes provides an evidence-based foundation for the development of a COS for shoulder arthroplasty.