Compound Kushen injection combined with transarterial chemoembolization for hepatocellular carcinoma: An evidence map and overview of systematic reviews

ETHNOPHARMACOLOGICAL RELEVANCE

For the treatment of hepatocellular carcinoma (HCC), compound Kushen injection (CKi) is commonly used in combination with transarterial chemoembolization (TACE).

AIMS OF THE STUDY

Our objective was to evaluate the reporting quality, methodological quality, risk of bias, and certainty of evidence for CKi combined with TACE for the treatment of patients with HCC by conducting systematic reviews (SRs). The purpose of this study was to improve the clinical application of CKis, strengthen clinical decision-making regarding CKis, and inform future research.

MATERIALS AND METHODS

We used eight databases to systematically search SRs of CKi combined with TACE for HCC through February 21, 2023. The quality of reporting of SRs was evaluated using the 2009 Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, methodological quality using A MeaSurement Tool to Assess systematic Reviews 2, risk of bias using the Risk of Bias in Systematic Review, and certainty of evidence using the Grading of Recommendations Assessment. Finally, the assessment results were visualized by the evidence mapping method. This overview has been registered on PROSPERO with the registration title "Compound Kushen injection for hepatocellular carcinoma: An overview of systematic reviews" and registration number CRD42022369120.

RESULTS

A total of 12 SRs meeting the inclusion criteria were included. In terms of reporting quality, 42% of SRs reported relatively complete reports and 58% had certain deficiencies. The methodological quality of all SRs was " critically low". The risk of bias was evaluated as low in 33% of SRs and high in 67% of SRs. The results of the evidence synthesis showed that, in the "moderate" level of evidence, CKi combined with TACE resulted in a 12.7%-21.5% benefit for one-year survival rate, 11.7%-17.2% benefit for objective response rate (ORR), 20.5%-27.1% benefit for quality of life, 22.2% benefit for nausea and vomiting, and 24.7%-27.4% benefit for leukopenia in HCC patients.

CONCLUSION

In conclusion, CKi combined with TACE improved survival, ORR and quality of life in patients with HCC, and reduced adverse events. The results should be interpreted with caution due to the low methodological quality of the included SRs. The clinical efficacy of CKis must be confirmed in a large number of randomized controlled trials.

Uncovering gaps in GRADE reporting of Eastern Association for the Surgery of Trauma (EAST) guidelines

OBJECTIVES

To formally evaluate the uptake and reporting of the Grading of Recommendation Assessments, Development and Evaluation (GRADE) approach in clinical practice guidelines (CPGs) developed by the Eastern Association for the Surgery of Trauma (EAST).

STUDY DESIGN AND SETTING

Based on an a priori, written protocol, we developed a dedicated data abstraction form that included the six suggested criteria for using and applying GRADE. By searching the EAST website, we identified all EAST guidelines that referenced the use of GRADE. All steps of the data abstraction process were completed independently and in duplicate by two members of the research team.

RESULTS

We identified a total of 48 CPGs that used GRADE. Trauma and violence prevention (n = 11; 23.9%) was the most common topic. The median number of patient/population, intervention, comparison, and outcomes (PICO) questions addressed was 3 (interquartile range: 2; 4) with a median of 2.5 (interquartile range: 1; 4) critical outcomes. A conditional/weak recommendation was provided for n = 79 (51.4%) PICOs, whereas a strong recommendation was provided for 33 PICOs (23.9%). For 22 PICOs (15.9%), no recommendation was made. Nearly all guideline documents provided search dates (n = 44; 95.7%) and a Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram (n = 44; 95.7%). Most described categories for rating down (n = 35; 76.1%). GRADE decision-making domains related to the ratio of benefits to harms, values and preferences, and resource utilization were referenced by 43.5% (n = 20), 43.5% (n = 20), and 30.4% (n = 14) of CPGs, respectively. For nearly half of PICO questions (n = 59; 44.2%) authors did not provide an evidence profile or summary of findings table. Comparing time periods from 2014-2018 to 2019-2022, the proportion of recommendations with an overall certainty of evidence increased (52.4% vs 83.9%; P < 0.001).

CONCLUSION

EAST has successfully adopted GRADE to develop many trauma-related guidelines, each addressing a finite number of focused clinical questions based on systematic reviews conducted in-house. Overall reporting improved over time. There is for improvement when it comes to consistent provision of an overall certainty of evidence, the reporting of the evidence to decision-making process, and the justification of strong recommendations based on low/very low certainty evidence.

GRADE concept 6: a novel application of external indirect evidence into GRADE ratings of evidence certainty in network meta-analysis

OBJECTIVES

We describe how consideration of external evidence may play an important role in judging certainty in the process of establishing the certainty of the evidence. Our example is a network meta-analysis (NMA) addressing treatment for Ebola virus disease, which informed a World Health Organization guideline.

STUDY DESIGN AND SETTING

Through Grading of Recommendations Assessment, Development, and Evaluations (GRADE) project group iterative online, in-person and email discussions, we developed this GRADE concept and obtained approval from the GRADE working group. Using the null as a threshold, we rated our certainty for network estimates in mortality, including consideration of evidence external to the NMA (i.e., did not meet eligibility criteria) and formal logical construction.

RESULTS

Based on the existing GRADE guidance, we rated the network estimate for one indirect comparison as low certainty. The formal logical construction that lead us reevaluate the certainty of the evidence is as follows: if A is superior to B, and B is not inferior to C, then A must be superior to C. After considering the logic and the external indirect evidence, we concluded at least moderate certainty for the comparison.

CONCLUSION

Systematic review authors and guideline developers should apply the fundamental logical construction for indirect comparisons and consider compelling external evidence in NMA certainty ratings.

Clinical systematic reviews - a brief overview

OBJECTIVE

Systematic reviews answer research questions through a defined methodology. It is a complex task and multiple articles need to be referred to acquire wide range of required knowledge to conduct a systematic review. The aim of this article is to bring the process into a single paper.

METHOD

The statistical concepts and sequence of steps to conduct a systematic review or a meta-analysis are examined by authors.

RESULTS

The process of conducting a clinical systematic review is described in seven manageable steps in this article. Each step is explained with examples to understand the method evidently.

CONCLUSION

A complex process of conducting a systematic review is presented simply in a single article.

Grading of recommendations assessment, development, and evaluation concept article 5: addressing intransitivity in a network meta-analysis

OBJECTIVES

This article describes considerations for addressing intransitivity when assessing the certainty of the evidence from network meta-analysis (NMA) using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach. Intransitivity is induced by effect modification, that is, when the magnitude of the effect between an intervention and outcome differs depending on the level of another factor.

STUDY DESIGN AND SETTING

To develop this GRADE concept paper, the lead authors conducted iterative discussions, computer simulations, and presentations to the GRADE project group and at GRADE working group meetings. The GRADE Working Group formally approved the article in July 2022.

RESULTS

NMA authors can have a higher or a lower threshold to rate down the certainty of the evidence due to intransitivity, which depends on the extent of their concerns regarding the trustworthiness of indirect comparisons, and their view of the relative problems with rating down excessively or insufficiently. NMA authors should consider three main factors when addressing intransitivity: the credibility of effect modification, the strength of the effect modification, and the distribution of effect modifiers across the direct comparisons. To avoid double counting limitations of the evidence, authors should consider the relationship between intransitivity and other GRADE domains.

CONCLUSION

NMA authors face theoretic and pragmatic challenges and in most situations need to assess intransitivity without the availability of empirical data. Thus, explicitness regarding perspective is crucial.

GRADE concept 4: rating the certainty of evidence when study interventions or comparators differ from PICO targets

OBJECTIVES

This Grading of Recommendations Assessment, Development and Evaluation (GRADE) concept article offers systematic reviewers, guideline authors, and other users of evidence assistance in addressing randomized trial situations in which interventions or comparators differ from those in the target people, interventions, comparators, and outcomes. To clarify what GRADE considers under indirectness of interventions and comparators, we focus on a particular example: when comparator arm participants receive some or all aspects of the intervention management strategy (treatment switching).

STUDY DESIGN AND SETTING

An interdisciplinary panel of the GRADE working group members developed this concept article through an iterative review of examples in multiple teleconferences, small group sessions, and e-mail correspondence. After presentation at a GRADE working group meeting in November 2022, attendees approved the final concept paper, which we support with examples from systematic reviews and individual trials.

RESULTS

In the presence of safeguards against risk of bias, trials provide unbiased estimates of the effect of an intervention on the people as enrolled, the interventions as implemented, the comparators as implemented, and the outcomes as measured. Within the GRADE framework, differences in the people, interventions, comparators, and outcomes elements between the review or guideline recommendation targets and the trials as implemented constitute issues of indirectness. The intervention or comparator group management strategy as implemented, when it differs from the target comparator, constitutes one potential source of indirectness: Indirectness of interventions and comparators-comparator group receipt of the intervention constitutes a specific subcategory of said indirectness. The proportion of comparator arm participants that received the intervention and the apparent magnitude of effect bear on whether one should rate down, and if one does, to what extent.

CONCLUSION

Treatment switching and other differences between review or guideline recommendation target interventions and comparators vs. interventions and comparators as implemented in otherwise relevant trials are best considered issues of indirectness.

Use of GRADE in systematic reviews of health effects on pollutants and extreme temperatures: A cross-sectional survey

OBJECTIVES

(i) To analyze trends and gaps in evidence of health effects on pollutants and extreme temperatures by evidence mapping; (ii) to conduct a cross-sectional survey on the use of the Grades of Recommendations Assessment Development and Evaluation (GRADE) in systematic reviews or meta-analyses (SR/MAs) of health effects on pollutants and extreme temperatures.

STUDY DESIGN AND SETTING

PubMed, Embase, the Cochrane Library, Web of Science, and Cumulative Index to Nursing and Allied Health Literature (CINAHL) were searched until July 7, 2022. SR/MAs investigated health effects of pollutants and extreme temperatures were included.

RESULTS

Out of 22,658 studies, 312 SR/MAs were included in evidence mapping, and the effects of pollutants on cancer and congenital malformations were new research hotspots. Among 16 SR/MAs involving 108 outcomes that were rated using GRADE, the certainty of evidence was mostly downgraded for inconsistency (50, 42.7%), imprecision (33, 28.2%), and risk of bias (24, 20.5%). In contrast, concentration-response gradient (26, 65.0%) was the main upgrade factor.

CONCLUSION

GRADE is not widely used in SR/MAs of health effects on pollutants and extreme temperatures. The certainty of evidence is generally low, mainly because of the serious inconsistency or imprecision. Use of the GRADE in SR/MAs of health effects on pollutants and extreme temperatures should strengthen.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

One-Third of Systematic Reviews in Rehabilitation Applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) System to Evaluate Certainty of Evidence: A Meta-Research Study

OBJECTIVE

To determine how many systematic reviews (SRs) of the literature in rehabilitation assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to do this.

DATA SOURCES

For this meta-research study, we searched PubMed and Cochrane Database of Systematic Reviews databases for SRs on rehabilitation published in 2020.

STUDY SELECTION AND DATA EXTRACTION

Two reviewers independently selected the SRs and extracted the data. Reporting characteristics and appropriate use of the GRADE system were assessed.

DATA SYNTHESIS

The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of which applied the GRADE system. GRADE was used by SRs of randomized controlled trials (RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according to the study design was not provided in 75% (12/16). The reasons for GRADE judgment were reported in 82% (134/163) of SRs.

CONCLUSIONS

One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment was presented transparently by most SRs. Journal editors and funders should encourage the uptake of the GRADE system when considering SRs in rehabilitation for publication. The authors should pre-define GRADE assessment in a registered and/or published protocol.

Editor's Choice - European Society for Vascular Surgery Clinical Practice Guideline Development Scheme: An Overview of Evidence Quality Assessment Methods, Evidence to Decision Frameworks, and Reporting Standards in Guideline Development

OBJECTIVE

A structured and transparent approach is instrumental in translating research evidence to health recommendations and evidence informed clinical decisions. The aim was to conduct an overview and analysis of principles and methodologies for health guideline development.

METHODS

A literature review on methodologies, strategies, and fundamental steps in the process of guideline development was performed. The clinical practice guideline development process and methodology adopted by the European Society for Vascular Surgery are also presented.

RESULTS

Sophisticated methodologies for health guideline development are being applied increasingly by national and international organisations. Their overarching principle is a systematic, structured, transparent, and iterative process that is aimed at making well informed healthcare choices. Critical steps in guideline development include the assessment of the certainty of the body of evidence; evidence to decision frameworks; and guideline reporting. The goal of strength of evidence assessments is to provide well reasoned judgements about the guideline developers' confidence in study findings, and several evidence hierarchy schemes and evidence rating systems have been described for this purpose. Evidence to decision frameworks help guideline developers and users conceptualise and interpret the construct of the quality of the body of evidence. The most widely used evidence to decision frameworks are those developed by the GRADE Working Group and the WHO-INTEGRATE, and are structured into three distinct components: background; assessment; and conclusions. Health guideline reporting tools are employed to ensure methodological rigour and transparency in guideline development. Such reporting instruments include the AGREE II and RIGHT, with the former being used for guideline development and appraisal, as well as reporting.

CONCLUSION

This guide will help guideline developers/expert panels enhance their methodology, and patients/clinicians/policymakers interpret guideline recommendations and put them in context. This document may be a useful methodological summary for health guideline development by other societies and organisations.

GRADE guidance 24 optimizing the integration of randomized and non-randomized studies of interventions in evidence syntheses and health guidelines

BACKGROUND AND OBJECTIVE

This is the 24th in the ongoing series of articles describing the GRADE approach for assessing the certainty of a body of evidence in systematic reviews and health technology assessments and how to move from evidence to recommendations in guidelines.

METHODS

Guideline developers and authors of systematic reviews and other evidence syntheses use randomized controlled studies (RCTs) and non-randomized studies of interventions (NRSI) as sources of evidence for questions about health interventions. RCTs with low risk of bias are the most trustworthy source of evidence for estimating relative effects of interventions because of protection against confounding and other biases. However, in several instances, NRSI can still provide valuable information as complementary, sequential, or replacement evidence for RCTs.

RESULTS

In this article we offer guidance on the decision regarding when to search for and include either or both types of studies in systematic reviews to inform health recommendations.

CONCLUSION

This work aims to help methodologists in review teams, technology assessors, guideline panelists, and anyone conducting evidence syntheses using GRADE.

An approach to quantifying the potential importance of residual confounding in systematic reviews of observational studies: A GRADE concept paper

Small relative effect sizes are common in observational studies of exposure in environmental and public health. However, such effects can still have considerable policy importance when the baseline rate of the health outcome is high, and many persons are exposed. Assessing the certainty of the evidence based on these effect sizes is challenging because they can be prone to residual confounding due to the non-randomized nature of the evidence. When applying GRADE, a precise relative risk >2.0 increases the certainty in an existing effect because residual confounding is unlikely to explain the association. GRADE also suggests rating up when opposing plausible residual confounding exists for other effect sizes. In this concept paper, we propose using the E-value, defined as the smallest effect size of a confounder that still can reduce an observed RR to the null value, and a reference confounder to assess the likelihood of residual confounding. We propose a 4-step approach. 1. Assess the association of interest for relevant exposure levels. 2. Calculate the E-value for this observed association. 3. Choose a reference confounder with sufficient strength and information and assess its effect on the observed association using the E-value. 4. Assess how likely it is that residual confounding will still bias the observed RR. We present three case studies and discuss the feasibility of the approach.

Visual transformation for guidelines presentation of the strength of recommendations and the certainty of evidence

OBJECTIVE

The objective of this paper is to propose an approach to visual unification of adapted guidelines and transformation of classifications of certainty of evidence (CoE) and strength of recommendations (SoR) into the approach suggested by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group.

STUDY DESIGN AND SETTING

We carried out a literature search in MEDLINE and Epistemonikos, an analysis of selected guidelines, and an iterative discussion to decide on a consistent visual presentation and CoE and SoR depictions.

RESULTS

The results of the literature search suggested this issue had not been addressed yet. The analysis of the chosen eight guidelines showed significant heterogeneity in the visual presentation of recommendations. Recommendations were often worded similarly to whether or not they were strong or conditional. Many guidelines contained "statements," almost all of which did not fulfill the good practice statement (GPS) criteria. We proposed an approach for transforming recommendations that are being adapted and which use various classification systems for CoE and SoR into GRADE and a consistent visual style.

CONCLUSION

Guideline developers should aim for unification in the formulation of recommendations to improve transferability.

The use of GRADE approach in Cochrane reviews of TCM was insufficient: a cross-sectional survey

OBJECTIVE

To conduct a cross-sectional survey on the application status of the Grades of Recommendations Assessment Development and Evaluation (GRADE) in Cochrane systematic reviews (CSRs) of traditional Chinese medicine (TCM).

STUDY DESIGN AND SETTING

We searched CSRs of TCM from the inception to December 2020 in the Cochrane Library database. General characteristics and details of GRADE were extracted.

RESULTS

Among 226 CSRs of TCM, 86 (38.05%) involving 711 outcomes used GRADE to rate the certainty of evidence. Topics mainly focused on genitourinary diseases (17.44%), diseases of the musculoskeletal system or connective tissue (11.63%), and diseases of the nervous system (10.47%). Only 15.89% of the outcomes reported high or moderate certainty of evidence. Acupuncture was the most common intervention. There were no significant differences in evidence certainty between acupuncture and non-acupuncture, between TCM alone and integrated Chinese and western medicine, or between Chinese patent medicines and non-Chinese patent medicines (P > 0.05). Among 1 273 instances of downgrading, 44.62% were due to the risk of bias and 40.14% due to imprecision.

CONCLUSION

Overall, GRADE approach is not widely used in CSRs of TCM. The certainty of evidence is generally low to very low, mainly because of the serious risk of bias and imprecision.

Risk Factors for Periventricular Leukomalacia in Preterm Infants: A Systematic Review, Meta-analysis, and GRADE-Based Assessment of Certainty of Evidence

BACKGROUND

We analyzed the certainty of evidence (CoE) for risk factors of periventricular leukomalacia (PVL) in preterm neonates, a common morbidity of prematurity.

METHODS

Medline, CENTRAL, Embase, and CINAHL were searched. Cohort and case-control studies and randomised randomized controlled trials were included. Data extraction was performed in duplicate. A random random-effects meta-analysis was utilizedused. CoE was evaluated as per Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidelines.

RESULTS

One hundred eighty-six studies evaluating 95 risk factors for PVL were included. Of the 2,509,507 neonates assessed, 16,569 were diagnosed with PVL. Intraventricular hemorrhage [adjusted odds ratio: 3.22 (2.52-4.12)] had moderate CoE for its association with PVL. Other factors such as hypocarbia, chorioamnionitis, PPROM >48 hour, multifetal pregnancy reduction, antenatal indomethacin, lack of antenatal steroids, perinatal asphyxia, ventilation, shock/hypotension, patent ductus arteriosus requiring surgical ligation, late-onset circulatory collapse, sepsis, necrotizing enterocolitis, and neonatal surgery showed significant association with PVL after adjustment for confounders (CoE: very low to low). Amongst the risk factors associated with mother placental fetal (MPF) triad, there was paucity of literature related to genetic predisposition and defective placentation. Sensitivity analysis revealed that the strength of association between invasive ventilation and PVL decreased over time (P < 0.01), suggesting progress in ventilation strategies. Limited studies had evaluated diffuse PVL.

CONCLUSION

Despite decades of research, our findings indicate that the CoE is low to very low for most of the commonly attributed risk factors of PVL. Future studies should evaluate genetic predisposition and defective placentation in the MPF triad contributing to PVL. Studies evaluating exclusively diffuse PVL are warranted.

GRADE Notes 2: Criteria for searching non-randomized or indirect evidence should be defined early in the guideline production process

OBJECTIVE

To discuss two alternative approaches for complementing the body of direct evidence from Randomized Controlled Trials (RCTs) when it is judged insufficient from a guideline panel making recommendations. The approaches included expanding the evidence's body to non-randomises studies on the population of interest or to RCTs on indirect populations.

STUDY DESIGN AND SETTING

In this report, we adopt the perspective of an evidence review team developing guidelines following the GRADE approach. Our experience is based on the development of two evidence-based guidelines promoted by The Italian National Institute of Health (ISS) and focusing on diagnosis and treatment of Autism Spectrum Disorders (ASD) in children/adolescents and adults.

RESULTS

We left panel members deciding case by case whether the direct evidence from RCTs was sufficient or not and indicating which alternative to implement. This strategy presented unanticipated challenges both from an organizational and methodological standpoint.

CONCLUSION

We suggest an early-stage production of a research protocol to define the criteria for expanding the body of evidence. These criteria should be informed by considerations around the certainty in the evidence, the clinical applicability of the results, feasibility and conflict of interest.

GRADE guidelines 33: Addressing imprecision in a network meta-analysis

OBJECTIVE

This article describes GRADE guidance for assessing imprecision when rating the certainty of the evidence from network meta-analysis.

STUDY DESIGN AND SETTING

A project group within the GRADE working group conducted iterative discussions, computer simulations, and presentations at GRADE working group meetings to produce and obtain approval for this guidance.

RESULTS

When addressing imprecision of a network estimate, reviewers should consider the 95% confidence or credible interval, and the optimal information size. If the 95% confidence or credible interval crosses a pre-specified threshold, reviewers should rate down the certainty of the evidence. If the 95% confidence interval does not cross any pre-specfied threshold, reviewers should consider the optimal information size. Because addressing the optimal information size may be challenging, reviewers can use the effect size to decide if any calculations are necessary. When the size of the effect is modest or the optimal information size is met, reviewers should not rate down for imprecision.

CONCLUSION

Reviewers should use this guidance when to appropriately address imprecision in the context of the assessment of certainty of evidence from network meta-analysis.

[GRADE guidelines 20: Assessing the certainty of evidence in the importance of outcomes or values and preferences-inconsistency, imprecision, and other domains]

OBJECTIVE

To provide Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) guidance for assessing inconsistency, imprecision, and other domains for the certainty of evidence about the relative importance of outcomes.

STUDY DESIGN AND SETTING

We applied the GRADE domains to rate the certainty of evidence in the importance of outcomes to several systematic reviews, iteratively reviewed draft guidance, and consulted GRADE members and other stakeholders for feedback.

RESULTS

We describe the rationale for considering the remaining GRADE domains when rating the certainty in a body of evidence for the relative importance of outcomes. As meta-analyses are not common in this context, inconsistency and imprecision assessments are challenging. Furthermore, confusion exists about inconsistency, imprecision, and true variability in the relative importance of outcomes. To clarify this issue, we suggest that the true variability is neither equivalent to inconsistency nor imprecision. Specifically, inconsistency arises from population, intervention, comparison and outcome and methodological elements that should be explored and, if possible, explained. The width of the confidence interval and sample size inform judgments about imprecision. We also provide suggestions on how to detect publication bias and discuss the domains to rate up the certainty.

CONCLUSION

We provide guidance and examples for rating inconsistency, imprecision, and other domains for a body of evidence describing the relative importance of outcomes.

GRADE Guidance: 31. Assessing the certainty across a body of evidence for comparative test accuracy

OBJECTIVES

This article provides GRADE guidance on how authors of evidence syntheses and health decision makers, including guideline developers, can rate the certainty across a body of evidence for comparative test accuracy questions.

STUDY DESIGN AND SETTING

This guidance extends the previously published GRADE guidance for assessing certainty of evidence for test accuracy to scenarios in which two or more index tests are compared. Through an iterative brainstorm-discussion-feedback process within the GRADE working group, we developed a guidance accompanied by practical examples.

RESULTS

Rating the certainty of evidence for comparative test accuracy shares many concepts and ideas with the existing GRADE guidance for test accuracy. The rating in comparisons of test accuracy requires additional considerations, such as the selection of appropriate comparative study designs, additional criteria for judging risk of bias, and the consequences of using comparative measures of test accuracy. Distinct approaches to rating certainty are required for comparative test accuracy studies and between-study (indirect) comparisons.

CONCLUSION

This GRADE guidance will support transparent assessment of the certainty for a body of comparative test accuracy evidence.

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making

OBJECTIVES

The objective of the study is to present the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty of evidence from modeling studies (i.e., certainty associated with model outputs).

STUDY DESIGN AND SETTING

Expert consultations and an international multidisciplinary workshop informed development of a conceptual approach to assessing the certainty of evidence from models within the context of systematic reviews, health technology assessments, and health care decisions. The discussions also clarified selected concepts and terminology used in the GRADE approach and by the modeling community. Feedback from experts in a broad range of modeling and health care disciplines addressed the content validity of the approach.

RESULTS

Workshop participants agreed that the domains determining the certainty of evidence previously identified in the GRADE approach (risk of bias, indirectness, inconsistency, imprecision, reporting bias, magnitude of an effect, dose-response relation, and the direction of residual confounding) also apply when assessing the certainty of evidence from models. The assessment depends on the nature of model inputs and the model itself and on whether one is evaluating evidence from a single model or multiple models. We propose a framework for selecting the best available evidence from models: 1) developing de novo, a model specific to the situation of interest, 2) identifying an existing model, the outputs of which provide the highest certainty evidence for the situation of interest, either "off-the-shelf" or after adaptation, and 3) using outputs from multiple models. We also present a summary of preferred terminology to facilitate communication among modeling and health care disciplines.

CONCLUSION

This conceptual GRADE approach provides a framework for using evidence from models in health decision-making and the assessment of certainty of evidence from a model or models. The GRADE Working Group and the modeling community are currently developing the detailed methods and related guidance for assessing specific domains determining the certainty of evidence from models across health care-related disciplines (e.g., therapeutic decision-making, toxicology, environmental health, and health economics).

An updated systematic review and meta-analysis on adherence to mediterranean diet and risk of cancer

Purpose

The aim of current systematic review was to update the body of evidence on associations between adherence to the Mediterranean diet (MedDiet) and risk of cancer mortality, site-specific cancer in the general population; all-cause, and cancer mortality as well as cancer reoccurrence among cancer survivors.

Methods

A literature search for randomized controlled trials (RCTs), case–control and cohort studies published up to April 2020 was performed using PubMed and Scopus. Study-specific risk estimates for the highest versus lowest adherence to the MedDiet category were pooled using random-effects meta-analyses. Certainty of evidence from cohort studies and RCTs was evaluated using the NutriGrade scoring system.

Results

The updated search revealed 44 studies not identified in the previous review. Altogether, 117 studies including 3,202,496 participants were enclosed for meta-analysis. The highest adherence to MedDiet was inversely associated with cancer mortality (RR_cohort: 0.87, 95% CI 0.82, 0.92; N = 18 studies), all-cause mortality among cancer survivors (RR_cohort: 0.75, 95% CI 0.66, 0.86; N = 8), breast (RR_{observational}: 0.94, 95% CI 0.90, 0.97; N = 23), colorectal (RR_{observational}: 0.83, 95% CI 0.76, 0.90; N = 17), head and neck (RR_{observational}: 0.56, 95% CI 0.44, 0.72; N = 9), respiratory (RR_cohort: 0.84, 95% CI 0.76, 0.94; N = 5), gastric (RR_{observational}: 0.70, 95% CI 0.61, 0.80; N = 7), bladder (RR_{observational}: 0.87, 95% CI 0.76, 0.98; N = 4), and liver cancer (RR_{observational}: 0.64, 95% CI 0.54, 0.75; N = 4). Adhering to MedDiet did not modify risk of blood, esophageal, pancreatic and prostate cancer risk.

Conclusion

In conclusion, our results suggest that highest adherence to the MedDiet was related to lower risk of cancer mortality in the general population, and all-cause mortality among cancer survivors as well as colorectal, head and neck, respiratory, gastric, liver and bladder cancer risks. Moderate certainty of evidence from cohort studies suggest an inverse association for cancer mortality and colorectal cancer, but most of the comparisons were rated as low or very low certainty of evidence.

Electronic supplementary material

The online version of this article (10.1007/s00394-020-02346-6) contains supplementary material, which is available to authorized users.

GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy

OBJECTIVES

This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can assess the results and the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA).

STUDY DESIGN AND SETTING

We present an overview of the GRADE approach and guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests. Part 1 of the two parts in this 21st guidance article about how to apply GRADE focuses on understanding study design issues in test accuracy, provide an overview of the domains, and describe risk of bias and indirectness specifically.

RESULTS

Supplemented by practical examples, we describe how raters of the evidence using GRADE can evaluate study designs focusing on tests and how they apply the GRADE domains risk of bias and indirectness to a body of evidence of TA studies.

CONCLUSION

Rating the certainty of a body of evidence using GRADE in Cochrane and other reviews and World Health Organization and other guidelines dealing with in TA studies helped refining our approach. The resulting guidance will help applying GRADE successfully for questions and recommendations focusing on tests.

Test accuracy: inconsistency, imprecision, publication bias, and other domains for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables

OBJECTIVES

This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can rate the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA) on the domains imprecision, inconsistency, publication bias, and other domains. It also provides guidance for how to present synthesized information in evidence profiles and summary of findings tables.

STUDY DESIGN AND SETTING

We present guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests.

RESULTS

Supplemented by practical examples, we describe how raters of the evidence can apply the GRADE domains inconsistency, imprecision, and publication bias to a body of evidence of TA studies.

CONCLUSION

Using GRADE in Cochrane and other reviews as well as World Health Organization and other guidelines helped refining the GRADE approach for rating the certainty of a body of evidence from TA studies. Although several of the GRADE domains (e.g., imprecision and magnitude of the association) require further methodological research to help operationalize them, judgments need to be made on the basis of what is known so far.

Development of the summary of findings table for network meta-analysis

OBJECTIVES

The aim of the study was to develop a Grading of Recommendations, Assessment, Development and Evaluation (GRADE) summary of findings (SoF) table format that displays the critical information from a network meta-analysis (NMA).

STUDY DESIGN AND SETTING

We applied a user experience model for data analysis based on four rounds of semistructured interviews.

RESULTS

We interviewed 32 stakeholders who conduct or use MA. Four rounds of interviews produced six candidate NMA-SoF tables. Users found a final NMA-SoF table that included the following components highly acceptable: (1) details of the clinical question (PICO), (2) a plot depicting network geometry, (3) relative and absolute effect estimates, (4) certainty of evidence, (5) ranking of treatments, and (6) interpretation of findings.

CONCLUSION

Using stakeholder feedback, we developed a new GRADE NMA-SoF table that includes the relevant components that facilitate understanding NMA findings and health decision-making.

Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis

This article describes conceptual advances of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group guidance to evaluate the certainty of evidence (confidence in evidence, quality of evidence) from network meta-analysis (NMA). Application of the original GRADE guidance, published in 2014, in a number of NMAs has resulted in advances that strengthen its conceptual basis and make the process more efficient. This guidance will be useful for systematic review authors who aim to assess the certainty of all pairwise comparisons from an NMA and who are familiar with the basic concepts of NMA and the traditional GRADE approach for pairwise meta-analysis. Two principles of the original GRADE NMA guidance are that we need to rate the certainty of the evidence for each pairwise comparison within a network separately and that in doing so we need to consider both the direct and indirect evidence. We present, discuss, and illustrate four conceptual advances: (1) consideration of imprecision is not necessary when rating the direct and indirect estimates to inform the rating of NMA estimates, (2) there is no need to rate the indirect evidence when the certainty of the direct evidence is high and the contribution of the direct evidence to the network estimate is at least as great as that of the indirect evidence, (3) we should not trust a statistical test of global incoherence of the network to assess incoherence at the pairwise comparison level, and (4) in the presence of incoherence between direct and indirect evidence, the certainty of the evidence of each estimate can help decide which estimate to believe.

The GRADE Working Group clarifies the construct of certainty of evidence

OBJECTIVE

To clarify the grading of recommendations assessment, development and evaluation (GRADE) definition of certainty of evidence and suggest possible approaches to rating certainty of the evidence for systematic reviews, health technology assessments, and guidelines.

STUDY DESIGN AND SETTING

This work was carried out by a project group within the GRADE Working Group, through brainstorming and iterative refinement of ideas, using input from workshops, presentations, and discussions at GRADE Working Group meetings to produce this document, which constitutes official GRADE guidance.

RESULTS

Certainty of evidence is best considered as the certainty that a true effect lies on one side of a specified threshold or within a chosen range. We define possible approaches for choosing threshold or range. For guidelines, what we call a fully contextualized approach requires simultaneously considering all critical outcomes and their relative value. Less-contextualized approaches, more appropriate for systematic reviews and health technology assessments, include using specified ranges of magnitude of effect, for example, ranges of what we might consider no effect, trivial, small, moderate, or large effects.

CONCLUSION

It is desirable for systematic review authors, guideline panelists, and health technology assessors to specify the threshold or ranges they are using when rating the certainty in evidence.