Going from evidence to recommendations: Can GRADE get us there?

Improving Guideline Development Processes: Integrating Evidence Estimation and Decision-Analytical Frameworks

RATIONALE, AIMS AND OBJECTIVES

Despite using state-of-the-art methodologies like Grades of Recommendation, Assessment, Development and Evaluation (GRADE), current guideline development frameworks still rely heavily on panellists' intuitive integration of evidence related to the benefits and harms/burdens of health interventions. This leads to the 'black-box' and 'integration' problems, highlighting the lack of transparency in guideline decision-making. Combined with humans' limited capacity to process the large volumes of information presented in Summary of Findings (SoF) tables-the primary output of systematic reviews that underpin guideline recommendations-this reliance on non-explicit processes raises concerns about the trustworthiness of clinical practice guidelines.

METHODS

SoF tables provide the best available evidence, derived from frequentist or Bayesian estimation frameworks. Decision analysis, which integrates both types of estimates but considers intervention consequences, is the only analytical approach that combines multiple outcomes (benefits, harms and costs) into a single metric to support decision-making. Such analysis seeks to identify the optimal decision by balancing harms, benefits and uncertainties. This paper leverages the PICO format (Population, Intervention, Comparison(s), Outcome) as a conceptual basis for deriving SoF tables. Subsequently, we propose a solution to GRADE's "black-box" and "integration" problems by matching PICO-based SoF with decision models.

RESULTS

We succeeded in connecting the PICO framework to simple decision-analytical models, restricted to time frames supported by empirically verifiable evidence, to calculate which competing intervention offers the greatest benefit (net differences in expected utility; ΔEU). The single metric [ΔEU] enabled a simple, transparent and easy-to-understand assessment of the superiority of competing management strategies across multiple outcomes (considering both benefits and harms), addressing the 'black-box' and 'integration' problems. Completing a SoF-based decision model takes about 10 min. Not surprisingly, the recommendations based on ΔEU may differ from the intuitive recommendations of panels.

CONCLUSION

We propose that incorporating the straightforward and transparent modelling into guideline panels' decision-making processes will enhance their intuitive judgements, resulting in more trustworthy recommendations. Given the simplicity of calculating ΔEU, we advocate for its immediate inclusion in systematic reviews and SoF tables.

Challenges and Stimulating Factors for the Incorporation of Economic Considerations in Clinical Practice Guidelines: A Scoping Review

RATIONALE, AIMS AND OBJECTIVES

The incorporation of economic considerations in clinical practice guidelines (CPGs) could help promote cost-conscious decision-making in healthcare. Though healthcare expenditures increase, and resources are becoming scarcer, the extent to which economic considerations are incorporated into CPGs remains limited. This scoping review aims to identify the challenges and potential stimulating factors to incorporate economic considerations in CPGs.

METHOD

This scoping review was conducted following the Joanna Briggs Institute Methodology and findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines. A systematic search was conducted in eight databases considering literature published from July 2017 and onwards. Data extraction was conducted via an iterative and inductive approach to identify challenges and potential stimulating factors from the included reports. Included documents focused on the (para)medical field and reported on CPG development and economic considerations.

RESULTS

The search identified 2445 documents from which 33 documents were included for analysis. The analysis identified five challenges: discourse surrounding economic considerations in CPGs, methodological ambiguities, scarcity of (high-quality) economic evidence, transferability of evidence, and resource constraints. Additionally, three potential stimulating factors were identified: acceptance, economic evidence knowledge, and guidance on incorporating economic considerations.

CONCLUSION

These findings reflect the complexity of incorporating economic considerations in CPGs. The identified challenges highlight the need for clearer guidance (i.e. by training) and standardised methodologies for incorporating economic considerations in CPGs. The potential stimulating factors provide a roadmap for future efforts to enhance the integration of economic evidence in CPGs. Collaborative initiatives between health economists, CPG developers, and other stakeholders are essential to drive progress in this area and promote cost-conscious decision-making in healthcare.

Technical specification for developing a clinical practice guideline for the integration of traditional Chinese medicine and Western medicine

Developing a clinical practice guideline (CPG) for integrating traditional Chinese medicine (TCM) and Western medicine (WM) requires the accurate identification, collation, and integration of all available evidence on TCM and WM in a comprehensive, meaningful, and resource-efficient manner. This entails framing appropriate clinical questions, retrieving and synthesizing evidence from multiple resources, and providing concise and complete recommendations for specific diseases. However, some existing CPGs for integrating TCM and WM lack deep and organic integration. As the effective preparation of a CPG for integrating TCM and WM typically involves a complex set of principles, methodology, and steps, we believe that a cohesive, step-by-step guide on how to prepare a CPG for integrating TCM and WM is essential. To facilitate the design and development of a robust CPG, we present a clear and concise methodology, outlining relevant principles and procedures, supported by references for guidance. This technical specification aims to simplify the methodology for preparing a CPG for integrating TCM and WM; provide healthcare professionals and researchers with methodologically sound tools; and enhance the quality of CPGs for integrating TCM and WM. This technical specification may help elucidate this complex process, facilitate evaluation of the quality of published CPGs for integrating TCM and WM, and improve the understanding and application of recommendations for the combined and integrated use of TCM and WM in a new system.

A clinician's guide to performing a case series study

BACKGROUND

Whilst some guidance exists, the literature is relatively scarce on designing and reporting on case series studies for non-surgical techniques/interventions or interventions that may be considered outside the medical model. This commentary presents a set of thirteen design attributes and an adapted checklist for consideration by clinicians when considering a case series design focused on a non-surgical intervention.

Shifting from Theoretical Best Evidence to Practical Best Evidence: an Approach to Overcome Structural Conservatism of Evidence-Based Medicine and Health Policy

There is disparity in the healthcare sector between the extent of innovation in medical products (e. g., drugs) and healthcare structures. The reason is not a lack of ideas, concepts, or (quasi-) experimental studies on structural innovations. Instead, we argue that the slow implementation of structural innovations has created this disparity partly because evidence-based medicine (EBM) instruments are well suited to evaluate product innovations but less suited to evaluate structural innovations. This article argues that the unintentional interplay between EBM, which has changed significantly over time to become primarily theoretical, on the one hand, and caution and inertia in health policy, on the other, has resulted in structural conservatism. Structural conservatism is present when healthcare structures persistently and essentially resist innovation. We interpret this phenomenon as an unintended consequence of deliberate EBM action. Therefore, we propose a new assessment framework to respond to structural innovations in healthcare, centered on the differentiation between the theoretical best (possible) evidence, the practical best (possible) evidence, and the best available evidence.

Decision theoretical foundations of clinical practice guidelines: an extension of the ASH thrombophilia guidelines

ABSTRACT

Decision analysis can play an essential role in informing practice guidelines. The American Society of Hematology (ASH) thrombophilia guidelines have made a significant step forward in demonstrating how decision modeling integrated within Grading of Recommendations Assessment, Developing, and Evaluation (GRADE) methodology can advance the field of guideline development. Although the ASH model was transparent and understandable, it does, however, suffer from certain limitations that may have generated potentially wrong recommendations. That is, the panel considered 2 models separately: after 3 to 6 months of index venous thromboembolism (VTE), the panel compared thrombophilia testing (A) vs discontinuing anticoagulants (B) and testing (A) vs recommending indefinite anticoagulation to all patients (C), instead of considering all relevant options simultaneously (A vs B vs C). Our study aimed to avoid what we refer to as the omitted choice bias by integrating 2 ASH models into a single unifying threshold decision model. We analyzed 6 ASH panel's recommendations related to the testing for thrombophilia in settings of "provoked" vs "unprovoked" VTE and low vs high bleeding risk (total 12 recommendations). Our model disagreed with the ASH guideline panels' recommendations in 4 of the 12 recommendations we considered. Considering all 3 options simultaneously, our model provided results that would have produced sounder recommendations for patient care. By revisiting the ASH guidelines methodology, we have not only improved the recommendations for thrombophilia but also provided a method that can be easily applied to other clinical problems and promises to improve the current guidelines' methodology.

Improving methods of clinical practice guidelines: From guidelines to pathways to fast-and-frugal trees and decision analysis to develop individualised patient care

BACKGROUND

Current methods for developing clinical practice guidelines have several limitations: they are characterised by the "black box" operation-a process with defined inputs and outputs but an incomplete understanding of its internal workings; they have "the integration problem"-a lack of framework for explicitly integrating factors such as patient preferences and trade-offs between benefits and harms; they generate one recommendation at a time that typically are not connected in a coherent analytical framework; and they apply to "average" patients, while clinicians and their patients seek advice tailored to individual circumstances.

METHODS

We propose augmenting the current guideline development method by converting evidence-based pathways into fast-and-frugal decision trees (FFTs) and integrating them with generalised decision curve analysis to formulate clear, individualised management recommendations.

RESULTS

We illustrate the process by developing recommendations for the management of heparin-induced thrombocytopenia (HIT). We converted evidence-based pathways for HIT, developed by the American Society of Hematology, into an FFT. Here, we consider only thrombotic complications and major bleeding. We leveraged the predictive potential of FFTs to compare the effects of argatroban, bivalirudin, fondaparinux, and direct oral anticoagulants (DOACs) using generalised decision curve analysis. We found that DOACs were superior to other treatments if the FFT-predicted probability of HIT exceeded 3%.

CONCLUSIONS

The proposed analytical framework connects guidelines, pathways, FFTs, and decision analysis, offering risk-tailored personalised recommendations and addressing current guideline development critiques.

Generalised decision curve analysis for explicit comparison of treatment effects

RATIONALE

Decision curve analysis (DCA) helps integrate prediction models with treatment assessments to guide personalised therapeutic choices among multiple treatment options. However, the current versions of DCA do not explicitly model treatment effects in the analysis but implicitly or holistically assess therapeutic benefits and harms. In addition, the existing DCA cannot allow the comparison of multiple treatments using a standard metric.

AIMS AND OBJECTIVES

To develop a generalised version of DCA (gDCA) by decomposing holistically assessed net benefits and harms into patient preferences versus empirical evidence (as obtained in the trials, meta-analyses of clinical studies, etc.) to allow individualised comparison of single or multiple treatments using a common metric.

METHODS

We reformulated DCA by (1) decomposing holistic, implicit utilities into specific utilities related to treatment effects and patient's relative values (RV) about disease outcomes versus treatment harms, (2) explicitly modelling each treatment effect at the level of probabilities and/or utilities (outcomes) in a decision tree, and (3) avoiding scaling effects employed in the original DCA to enable comparison of treatment effects against the common metrics. We used data from a published network meta-analysis of randomised trials to inform the use of statin treatment according to Framingham Risk Model.

RESULTS

We illustrate the analysis by modelling the effects of three statins in the primary prevention of cardiovascular disease. We performed simultaneous comparisons against standard metrics (RV) for all treatments. We examined for which RV values, a predictive model for guiding personalised treatment, outperformed the strategies of treating everyone or treating no one. We found that the magnitude of benefits (efficacy) seems more important than the simple ratio of efficacy/harms.

CONCLUSION

We describe gDCA for evaluating single or multiple treatments to help tailor therapy toward individual risk characteristics. gDCA further helps integrate the principles of evidence-based medicine with decision analysis.

The use of GRADE-CERQual in qualitative evidence synthesis: an evaluation of fidelity and reporting

BACKGROUND

GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative Research) is a methodological approach to systematically and transparently assess how much confidence decision makers can place in individual review findings from qualitative evidence syntheses. The number of reviews applying GRADE-CERQual is rapidly expanding in guideline and other decision-making contexts. The objectives of this evaluation were, firstly, to describe the uptake of GRADE-CERQual in qualitative evidence synthesis by review authors and, secondly, to assess both reporting of and fidelity to the approach.

METHODS

The evaluation had two parts. Part 1 was a citation analysis and descriptive overview of the literature citing GRADE-CERQual. Authors worked together to code and chart the citations, first by title and abstract and second by full text. Part 2 was an assessment and analysis of fidelity to, and reporting of, the GRADE-CERQual approach in included reviews. We developed fidelity and reporting questions and answers based on the most recent guidance for GRADE-CERQual and then used NVivo12 to document assessments in a spreadsheet and code full-text PDF articles for any concerns that had been identified. Our assessments were exported to Excel and we applied count formulae to explore patterns in the data. We employed a qualitative content analysis approach in NVivo12 to sub-coding all the data illustrating concerns for each reporting and fidelity criteria.

RESULTS

233 studies have applied the GRADE-CERQual approach, with most (n = 225, 96.5%) in the field of health research. Many studies (n = 97/233, 41.6%) were excluded from full fidelity and reporting assessment because they demonstrated a serious misapplication of GRADE-CERQual, for example interpreting it as a quality appraisal tool for primary studies or reviews. For the remaining studies that applied GRADE-CERQual to assess confidence in review findings, the main areas of reporting concern involved terminology, labelling and completeness. Fidelity concerns were identified in more than half of all studies assessed.

CONCLUSIONS

GRADE-CERQual is being used widely within qualitative evidence syntheses and there are common reporting and fidelity issues. Most of these are avoidable and we highlight these as gaps in knowledge and guidance for applying the GRADE-CERQual approach.

Exploring the feasibility of using the ICER Evidence Rating Matrix for Comparative Clinical Effectiveness in assessing treatment benefit and certainty in the clinical evidence on orphan therapies for paediatric indications

BACKGROUND

The evaluation of clinical evidence takes account of health benefit (efficacy and safety) and the degree of certainty in the estimate of benefit. In orphan indications practical and ethical challenges in conducting clinical trials, particularly in paediatric patients, often limit the available evidence, rendering structured evaluation challenging. While acknowledging the paucity of evidence, regulators and reimbursement authorities compare the efficacy and safety of alternative treatments for a given indication, often in the context of the benefits of other treatments for similar or different conditions. This study explores the feasibility of using the Institute for Clinical and Economic Review (ICER) Evidence Rating Matrix for Comparative Clinical Effectiveness in structured assessment of both the magnitude of clinical benefit (net health benefit, NHB) and the certainty of the effect estimate in a sample of orphan therapies for paediatric indications.

RESULTS

Eleven systemic therapies with European Medicines Agency (EMA) orphan medicinal product designation, licensed for 16 paediatric indications between January 2017 and March 2020 were identified using OrphaNet and EMA databases and were selected for evaluation with the ICER Evidence Rating Matrix: burosumab; cannabidiol; cerliponase alfa; chenodeoxycholic acid (CDCA); dinutuximab beta; glibenclamide; metreleptin; nusinersen; tisagenlecleucel; velmanase alfa; and vestronidase alfa. EMA European Public Assessment Reports, PubMed, EMBASE, the Cochrane Library, Clinical Key, and conference presentations from January 2016 to April 2021 were searched for evidence on efficacy and safety. Two of the identified therapies were graded as "substantial" NHB: dinutuximab beta (neuroblastoma maintenance) and nusinersen (Type I SMA), and one as "comparable" NHB (CDCA). The NHB grade of the remaining therapies fell between "comparable" and "substantial". No therapies were graded as having negative NHB. The certainty of the estimate ranged from "high" (dinutuximab beta in neuroblastoma maintenance) to "low" (CDCA, metreleptin and vestronidase alfa). The certainty of the other therapies was graded between "low" and "high". The ICER Evidence Rating Matrix overall rating "A" (the highest) was given to two therapies, "B+" to 6 therapies, "C+" to five therapies, and "I" (the lowest) to three therapies. The scores varied between rating authors with mean agreement over all indications of 71.9% for NHB, 56.3% for certainty and 68.8% for the overall rating.

CONCLUSIONS

Using the ICER Matrix to grade orphan therapies according to their treatment benefit and certainty is feasible. However, the assessment involves subjective judgements based on heterogenous evidence. Tools such as the ICER Matrix might aid decision makers to evaluate treatment benefit and its certainty when comparing therapies across indications.

Therapeutic effects of repetitive transcranial magnetic stimulation on cognitive impairment in stroke patients: a systematic review and meta-analysis

Background

In recent years, repetitive transcranial magnetic stimulation (rTMS) has emerged as a noninvasive and painless treatment for post-stroke cognitive impairment (PSCI). However, few studies have analyzed the intervention parameters of cognitive function and the effectiveness and safety of rTMS for treating patients with PSCI. Thus, this meta-analysis aimed to analyze the interventional parameters of rTMS and evaluate the safety and effectiveness of rTMS for treating patients with PSCI.

Methods

According to the PRISMA guidelines, we searched the Web of Science, PubMed, EBSCO, Cochrane Library, PEDro, and Embase to retrieve randomized controlled trials (RCTs) of rTMS for the treatment of patients with PSCI. Studies were screened according to the inclusion and exclusion criteria, and two reviewers independently performed literature screening, data extraction, and quality assessment. RevMan 5.40 software was used for data analysis.

Results

12 RCTs involving 497 patients with PSCI met the inclusion criteria. In our analysis, rTMS had a positive therapeutic effect on cognitive rehabilitation in patients with PSCI (P < 0.05). Both high-frequency rTMS and low-frequency rTMS were effective in improving the cognitive function of patients with PSCI by stimulating the dorsolateral prefrontal cortex (DLPFC), but their efficacy was not statistically different (P > 0.05).

Conclusions

rTMS treatment on the DLPFC can improve cognitive function in patients with PSCI. There is no significant difference in the treatment effect of high-frequency rTMS and low-frequency rTMS in patients with PSCI between high-frequency and low-frequency rTMS.

Systematic review registration

https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=323720, identifier CRD 42022323720.

Decomposing clinical practice guidelines panels' deliberation into decision theoretical constructs

RATIONALE, AIMS AND OBJECTIVES: The development of clinical practice guidelines (CPG) suffers from the lack of an explicit and transparent framework for synthesising the key elements necessary to formulate practice recommendations. We matched deliberations of the American Society of Haematology (ASH) CPG panel for the management of pulmonary embolism (PE) with the corresponding decision-theoretical constructs to assess agreement of the panel recommendations with explicit decision modelling.

METHODS

Five constructs were identified of which three were used to reformulate the panel's recommendations: (1) standard, expected utility threshold (EUT) decision model; (2) acceptable regret threshold model (ARg) to determine the frequency of tolerable false negative (FN) or false positive (FP) recommendations, and (3) fast-and-frugal tree (FFT) decision trees to formulate the entire strategy for management of PE. We compared four management strategies: withhold testing versus d-dimer → computerized pulmonary angiography (CTPA) ('ASH-Low') versus CTPA→ d-dimer ('ASH-High') versus treat without testing.

RESULTS

Different models generated different recommendations. For example, according to EUT, testing should be withheld for prior probability PE < 0.13%, a clinically untenable threshold which is up to 15 times (2/0.13) below the ASH guidelines threshold of ruling out PE (at post probability of PE ≤ 2%). Three models only agreed that the 'ASH low' strategy should be used for the range of pretest probabilities of PE between 0.13% and 13.27% and that the 'ASH high' management should be employed in a narrow range of the prior PE probabilities between 90.85% and 93.07%. For all other prior probabilities of PE, choosing one model did not ensure coherence with other models.

CONCLUSIONS

CPG panels rely on various decision-theoretical strategies to develop its recommendations. Decomposing CPG panels' deliberation can provide insights if the panels' deliberation retains a necessary coherence in developing guidelines. CPG recommendations often do not agree with the EUT decision analysis, widely used in medical decision-making modelling.

Evaluation of a fast-and-frugal clinical decision algorithm ('pathways') on clinical outcomes in hospitalised patients with COVID-19 treated with anticoagulants

RATIONALE, AIMS AND OBJECTIVES

Critics have charged that evidence-based medicine (EBM) overemphasises algorithmic rules over unstructured clinical experience and intuition, but the role of structured decision support systems in improving health outcomes remains uncertain. We aim to assess if delivery of anticoagulant prophylaxis in hospitalised patients with COVID-19 according to an algorithm based on evidence-based clinical practice guideline (CPG) improved clinical outcomes compared with administration of anticoagulant treatment given at individual practitioners' discretion.

METHODS

An observational design consisting of the analysis of all acutely ill, consecutive patients (n = 1783) with confirmed COVID-19 diagnosis admitted between 10 March 2020 to 11 January 2022 to an US academic center. American Society of Haematology CPG for anticoagulant prophylaxis in hospitalised patients with COVID-19 was converted into a clinical pathway and translated into fast-and-frugal decision (FFT) tree ('algorithm'). We compared delivery of anticoagulant prophylaxis in hospitalised patients with COVID-19 according to the FFT algorithm with administration of anticoagulant treatment given at individual practitioners' discretion.

RESULTS

In an adjusted analysis, using combination of Lasso (least absolute shrinkage and selection operator) and propensity score based weighting [augmented inverse-probability weighting] statistical techniques controlling for cluster data, the algorithm did not reduce death, venous thromboembolism, or major bleeding, but helped avoid longer hospital stay [number of patients needed to be treated (NNT) = 40 (95% CI: 23-143), indicating that for every 40 patients (23-143) managed on FFT algorithm, one avoided staying in hospital longer than 10 days] and averted admission to intensive-care unit (ICU) [NNT = 19 (95% CI: 13-40)]. All model's selected covariates were well balanced. The results remained robust to sensitivity analyses used to test the stability of the findings.

CONCLUSIONS

When delivered using a structured FFT algorithm, CPG shortened the hospital stay and help avoided admission to ICU, but it did not affect other relevant outcomes.

Clinical recommendations: The role of mechanisms in the GRADE framework

The Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework has become one of the most influential frameworks for assessing quality of research and developing clinical recommendations. The GRADE framework has been presented as an evolution in the Evidence-Based Medicine (EBM) movement. Both GRADE and EBM emphasize effect estimates derived from population-level clinical trials and, as a consequence, devalue the role of mechanisms as the basis for clinical decisions. Although mechanisms do not hold the epistemic privilege of rigorous clinical trials in EBM reasoning, this paper will argue that mechanisms appear to be important in the use and application of GRADE, as described in the literature. The seemingly necessary role of mechanisms in the development of clinical recommendations has, so far, received little attention and is not explicitly featured in the literature describing GRADE. The analysis of the GRADE framework presented in this paper reveals an apparent tension between EBM's willingness to downplay mechanisms and what seems their inevitable use in GRADE. In this paper, we take the position that if mechanistic reasoning is inevitable in the use of GRADE, then the instructional literature on the framework would benefit from more explicit discussion of how to consider and integrate mechanisms.

Meta-research evaluating redundancy and use of systematic reviews when planning new studies in health research: a scoping review

BACKGROUND

Several studies have documented the production of wasteful research, defined as research of no scientific importance and/or not meeting societal needs. We argue that this redundancy in research may to a large degree be due to the lack of a systematic evaluation of the best available evidence and/or of studies assessing societal needs.

OBJECTIVES

The aim of this scoping review is to (A) identify meta-research studies evaluating if redundancy is present within biomedical research, and if so, assessing the prevalence of such redundancy, and (B) to identify meta-research studies evaluating if researchers had been trying to minimise or avoid redundancy.

ELIGIBILITY CRITERIA

Meta-research studies (empirical studies) were eligible if they evaluated whether redundancy was present and to what degree; whether health researchers referred to all earlier similar studies when justifying and designing a new study and/or when placing new results in the context of earlier similar trials; and whether health researchers systematically and transparently considered end users' perspectives when justifying and designing a new study.

SOURCES OF EVIDENCE

The initial overall search was conducted in MEDLINE, Embase via Ovid, CINAHL, Web of Science, Social Sciences Citation Index, Arts & Humanities Citation Index, and the Cochrane Methodology Register from inception to June 2015. A 2nd search included MEDLINE and Embase via Ovid and covered January 2015 to 26 May 2021. No publication date or language restrictions were applied.

CHARTING METHODS

Charting methods included description of the included studies, bibliometric mapping, and presentation of possible research gaps in the identified meta-research.

RESULTS

We identified 69 meta-research studies. Thirty-four (49%) of these evaluated the prevalence of redundancy and 42 (61%) studies evaluated the prevalence of a systematic and transparent use of earlier similar studies when justifying and designing new studies, and/or when placing new results in context, with seven (10%) studies addressing both aspects. Only one (1%) study assessed if the perspectives of end users had been used to inform the justification and design of a new study. Among the included meta-research studies evaluating whether redundancy was present, only two of nine health domains (medical areas) and only two of 10 research topics (different methodological types) were represented. Similarly, among the included meta-research studies evaluating whether researchers had been trying to minimise or avoid redundancy, only one of nine health domains and only one of 10 research topics were represented.

CONCLUSIONS THAT RELATE TO THE REVIEW QUESTIONS AND OBJECTIVES

Even with 69 included meta-research studies, there was a lack of information for most health domains and research topics. However, as most included studies were evaluating across different domains, there is a clear indication of a high prevalence of redundancy and a low prevalence of trying to minimise or avoid redundancy. In addition, only one meta-research study evaluated whether the perspectives of end users were used to inform the justification and design of a new study.

SYSTEMATIC REVIEW REGISTRATION

Protocol registered at Open Science Framework: https://osf.io/3rdua/ (15 June 2021).

E-synthesis for carcinogenicity assessments: A case study of processed meat

RATIONALE, AIMS AND OBJECTIVES

Recent controversies about dietary advice concerning meat demonstrate that aggregating the available evidence to assess a putative causal link between food and cancer is a challenging enterprise.

METHODS

We show how a tool developed for assessing putative causal links between drugs and adverse drug reactions, E-Synthesis, can be applied for food carcinogenicity assessments. The application is demonstrated on the putative causal relationship between processed meat consumption and cancer.

RESULTS

The output of the assessment is a Bayesian probability that processed meat consumption causes cancer. This Bayesian probability is calculated from a Bayesian network model, which incorporates a representation of Bradford Hill's Guidelines as probabilistic indicators of causality. We show how to determine probabilities of indicators of causality for food carcinogenicity assessments based on assessments of the International Agency for Research on Cancer.

CONCLUSIONS

We find that E-Synthesis is a tool well-suited for food carcinogenicity assessments, as it enables a graphical representation of lines and weights of evidence, offers the possibility to make a great number of judgements explicit and transparent, outputs a probability of causality suitable for decision making and is flexible to aggregate different kinds of evidence.

Clinical effectiveness of nimodipine for the prevention of poor outcome after aneurysmal subarachnoid hemorrhage: A systematic review and meta-analysis

Objective

In clinical practice, nimodipine is used to control cerebral vasospasm (CVS), which is one of the major causes of severe disability and mortality in patients with aneurysmal subarachnoid hemorrhage (aSAH). However, the exact efficacy of nimodipine use for patients with aSAH is still controversial due to the lack of sufficient and up-to-date evidence.

Methods

In this meta-analysis, the latest databases of the Cochrane Central Register of Controlled Trials, PubMed-Medline, Web of Science, Embase, Scopus, and OVID-Medline were comprehensively searched for retrieving all randomized controlled trials (RCTs) regarding the efficacy of nimodipine in patients with aSAH. The primary outcome was a poor outcome, and the secondary outcomes were mortality and cerebral vasospasm (CVS). After detailed statistical analysis of different outcome variables, further evidence quality evaluation and recommendation grade assessment were carried out.

Results

Approximately 13 RCTs met the inclusion criteria, and a total of 1,727 patients were included. Meta-analysis showed that a poor outcome was significantly reduced in the nimodipine group [RR, 0.69 (0.60–0.78); I² = 29%]. Moreover, nimodipine also dramatically decreased the mortality [RR, 0.50 (0.32–0.78); I² = 62%] and the incidence of CVS [RR, 0.68 (0.46–0.99); I² = 57%]. Remarkably, we found a poor outcome and mortality were both significantly lower among patients with aSAH, with the mean age < 50 than that mean age ≥ 50 by subgroup analysis. Furthermore, the evidence grading of a poor outcome and its age subgroup in this study was assessed as high.

Conclusion

Nimodipine can significantly reduce the incidence of a poor outcome, mortality, and CVS in patients with aSAH. Moreover, we strongly recommend that patients with aSAH, especially those younger than 50 years old, should use nimodipine as early as possible in order to achieve a better clinical outcome, whether oral medication or endovascular direct medication.

Systematic review registration

www.york.ac.uk/inst/crd, identifier: CRD42022334619.

The Effectiveness of High-Frequency Repetitive Transcranial Magnetic Stimulation on Patients with Neuropathic Orofacial Pain: A Systematic Review of Randomized Controlled Trials

Background

Repetitive transcranial magnetic stimulation (rTMS) has been widely used in the treatment of neuropathic orofacial pain (NOP). The consistency of its therapeutic efficacy with the optimal protocol is highly debatable.

Objective

To assess the effectiveness of rTMS on pain intensity, psychological conditions, and quality of life (QOL) in individuals with NOP based on randomized controlled trials (RCTs).

Methods

We carefully screened and browsed 5 medical databases from inception to January 1, 2022. The study will be included that use of rTMS as the intervention for patients with NOP. Two researchers independently completed record retrieval, data processing, and evaluation of methodological quality. Quality and evidence were assessed using the PEDro scores and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system.

Results

Six RCTs with 214 participants were included in this systematic review: 2 studies were considered level 1 evidence, and 4 were considered level 2 evidence. Six studies found that high-frequency rTMS had a pain-relieving effect, while 4 studies found no improvement in psychological conditions and QOL. Quality of evidence (GRADE system) ranged from moderate to high. No significant side effects were found.

Conclusions

There is moderate-to-high evidence to prove that high-frequency rTMS is effective in reducing pain in individuals with NOP, but it has no significant positive effect on psychological conditions and QOL. High-frequency rTMS can be used as an alternative treatment for pain in individuals with NOP, but further studies will be conducted to unify treatment parameters, and the sample size will be expanded to explore its influence on psychological conditions and QOL.

High quality (certainty) evidence changes less often than low-quality evidence, but the magnitude of effect size does not systematically differ between studies with low versus high-quality evidence

RATIONALE, AIMS, AND OBJECTIVES

It is generally believed that evidence from low quality of evidence generate inaccurate estimates about treatment effects more often than evidence from high (certainty) quality evidence (CoE). As a result, we would expect that (a) estimates of effects of health interventions initially based on high CoE change less frequently than the effects estimated by lower CoE (b) the estimates of magnitude of effect size differ between high and low CoE. Empirical assessment of these foundational principles of evidence-based medicine has been lacking.

METHODS

We reviewed the Cochrane Database of Systematic Reviews from January 2016 through May 2021 for pairs of original and updated reviews for change in CoE assessments based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method. We assessed the difference in effect sizes between the original versus updated reviews as a function of change in CoE, which we report as a ratio of odds ratio (ROR). We compared ROR generated in the studies in which CoE changed from very low/low (VL/L) to moderate/high (M/H) versus M/H to VL/L. Heterogeneity and inconsistency were assessed using the tau and I² statistic. We also assessed the change in precision of effect estimates (by calculating the ratio of standard errors) (seR), and the absolute deviation in estimates of treatment effects (aROR).

RESULTS

Four hundred and nineteen pairs of reviews were included of which 414 (207 × 2) informed the CoE appraisal and 384 (192 × 2) the assessment of effect size. We found that CoE originally appraised as VL/L had 2.1 [95% confidence interval (CI): 1.19-4.12; p = 0.0091] times higher odds to be changed in the future studies than M/H CoE. However, the effect size was not different (p = 1) when CoE changed from VL/L → M/H [ROR = 1.02 (95% CI: 0.74-1.39)] compared with M/H → VL/L (ROR = 1.02 [95% CI: 0.44-2.37]). Similar overlap in aROR between the VL/L → M/H versus M/H → VL/L subgroups was observed [median (IQR): 1.12 (1.07-1.57) vs. 1.21 (1.12-2.43)]. We observed large inconsistency across ROR estimates (I²  = 99%). There was larger imprecision in treatment effects when CoE changed from VL/L → M/H (seR = 1.46) than when it changed from M/H → VL/L (seR = 0.72).

CONCLUSIONS

We found that low-quality evidence changes more often than high CoE. However, the effect size did not systematically differ between the studies with low versus high CoE. The finding that the effect size did not differ between low and high CoE indicate urgent need to refine current EBM critical appraisal methods.

Has anything changed in Evidence-Based Medicine?

The Evidence-Based Medicine (EBM) movement, undoubtably one of the most successful movements in medicine, questions dogma and "clinical authority" and combines the "best available evidence" with clinical expertise and patient values in order to provide the best care for the individual patient. Although since its inception in the 1990s its strong theoretical foundations remain unaltered, a lot has changed in its practical implementation due to the electronic explosion of information and the unprecedented COVID-19 crisis. The purpose of this article is to succinctly provide the reader with an update on the major changes in EBM, including the important most recent ones that were "fast-tracked" due to the COVID-19 challenge.

Cytomegalovirus-A Risk Factor for Childhood Hearing Loss: A Systematic Review

OBJECTIVE

Permanent hearing loss is an important public health issue in children with consequences for language, social, and academic functioning. Early hearing detection, intervention, and monitoring are important in mitigating the impact of permanent childhood hearing loss. Congenital cytomegalovirus (CMV) infection is a leading cause of hearing loss. The purpose of this review was to synthesize the evidence on the association between CMV infection and permanent childhood hearing loss.

DESIGN

We performed a systematic review and examined scientific literature from the following databases: MEDLINE, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R), Embase, and CINAHL. The primary outcome was permanent bilateral or unilateral hearing loss with congenital onset or onset during childhood (birth to 18 years). The secondary outcome was progressive hearing loss. We included studies reporting data on CMV infection. Randomized controlled trials, quasi-experimental studies, nonrandomized comparative and noncomparative studies, and case series were considered. Data were extracted and the quality of individual studies was assessed with the Qualitative Assessment Tool for Quantitative Studies (McMaster University). The quality and strength of the evidence were graded using the Grading of Recommendations Assessment, Development and Evaluation (GRADE). A narrative synthesis was completed.

RESULTS

Sixty-five articles were included in the review. Prevalence of hearing loss at birth was over 33% among symptomatic CMV-infected newborns and less than 15% in asymptomatic infections. This difference in prevalence was maintained during childhood with more than 40% prevalence reported for symptomatic and less than 30% for asymptomatic CMV. Late-onset and progressive hearing loss appear to be characteristic of congenital CMV infections. Definitions of hearing loss, degree of loss, and reporting of laterality varied across studies. All degrees and both bilateral and unilateral loss were reported, regardless of symptomatic and asymptomatic status at birth, and no conclusions about the characteristics of hearing loss could be drawn. Various patterns of hearing loss were reported including stable, progressive, and fluctuating, and improvement in hearing (sometimes to normal hearing) was documented. These changes were reported in children with symptomatic/asymptomatic congenital CMV infection, presenting with congenital/early onset/late-onset hearing loss and in children treated and untreated with antiviral medication.

CONCLUSIONS

Symptomatic and asymptomatic congenital CMV infection should be considered a risk factor for hearing loss at birth and during childhood and for progressive hearing loss. Therefore, CMV should be included as a risk factor in screening and surveillance programs and be taken into account in clinical follow-up of children with hearing loss.

A few panel members dominated guideline development meeting discussions: Social network analysis

OBJECTIVES

To identify patterns of interactions that may influence guideline panels' decision-making.

STUDY DESIGN AND SETTING

Social network analysis (SNA) to describe the conversation network in a guideline development meeting in United States.

RESULTS

We analyzed one two-day guideline panel meeting that included 20 members who developed a guideline using the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach. The conversation pattern of the guideline panel indicated a well-connected network (density=0.59, clustering coefficient=0.82). GRADE topics on quality of evidence and benefits versus harms accounted for 46%; non-GRADE factors accounted for 30% of discussion. The chair, co-chair and methodologist initiated 53% and received 60% of all communications in the meeting; 42% of their communications occurred among themselves. SNA metrics (eigenvector, betweenness and closeness) indicated that these individuals also exerted highest influence on discussion, controlled information flow and were at the center of all communications. Members were more likely to continue previous discussion with the same individuals after both morning breaks (r=0.54, P<0.005; r=0.17, P=0.04), and after the last break on day 2 (r=0.44, P=0.015).

CONCLUSION

Non-GRADE factors such as breaks, and the members' roles, affect guideline development more than previously recognized. Collectively, the chair, co-chair and methodologist dominated the discussion.

EA3: A softmax algorithm for evidence appraisal aggregation

Real World Evidence (RWE) and its uses are playing a growing role in medical research and inference. Prominently, the 21st Century Cures Act—approved in 2016 by the US Congress—permits the introduction of RWE for the purpose of risk-benefit assessments of medical interventions. However, appraising the quality of RWE and determining its inferential strength are, more often than not, thorny problems, because evidence production methodologies may suffer from multiple imperfections. The problem arises to aggregate multiple appraised imperfections and perform inference with RWE. In this article, we thus develop an evidence appraisal aggregation algorithm called EA³. Our algorithm employs the softmax function—a generalisation of the logistic function to multiple dimensions—which is popular in several fields: statistics, mathematical physics and artificial intelligence. We prove that EA³ has a number of desirable properties for appraising RWE and we show how the aggregated evidence appraisals computed by EA³ can support causal inferences based on RWE within a Bayesian decision making framework. We also discuss features and limitations of our approach and how to overcome some shortcomings. We conclude with a look ahead at the use of RWE.

Professional judgement in clinical practice (part 2): knowledge into practice

RATIONALE, AIMS, AND OBJECTIVES

Though strong evidence-based medicine is assertive in its claims, an insufficient theoretical basis and patchwork of arguments provide a good case that rather than introducing a new paradigm, EBM is resisting a shift to actually revolutionary complexity theory and other emergent approaches. This refusal to pass beyond discredited positivism is manifest in strong EBM's unsuccessful attempts to continually modify its already inadequate previous modifications, as did the defenders of the Ptolemaic astronomical model who increased the number of circular epicycles until the entire epicycle-deferent system proved untenable.

METHODS

Narrative Review.

RESULTS

The analysis in Part 1 of this three part series showed epistemological confusion as strong EBM plays the discredited positivistic tradition out to the end, thus repeating in a medical sphere and vocabulary the major assumptions and inadequacies that have appeared in the trajectory of modern science. Paper 2 in this series examines application, attending to strong EBM's claim of direct transferability of EBM research findings to clinical settings and its assertion of epistemological normativity. EBM's contention that it provides the "only valid" approach to knowledge and action is questioned by analyzing the troubled story of proposed hierarchies of the quality of research findings (especially of RCTs, with other factors marginalized), which falsely identifies evaluating findings with operationally utilizing them in clinical recommendations and decision-making. Further, its claim of carrying over its normative guidelines to cover the ethical responsibilities of researchers and clinicians is questioned.

Artificial intelligence methods for a Bayesian epistemology-powered evidence evaluation

RATIONALE, AIMS AND OBJECTIVES

The diversity of types of evidence (eg, case reports, animal studies and observational studies) makes the assessment of a drug's safety profile into a formidable challenge. While frequentist uncertain inference struggles in aggregating these signals, the more flexible Bayesian approaches seem better suited for this quest. Artificial Intelligence (AI) offers great promise to these approaches for information retrieval, decision support, and learning probabilities from data.

METHODS

E-Synthesis is a Bayesian framework for drug safety assessments built on philosophical principles and considerations. It aims to aggregate all the available information, in order to provide a Bayesian probability of a drug causing an adverse reaction. AI systems are being developed for evidence aggregation in medicine, which increasingly are automated.

RESULTS

We find that AI can help E-Synthesis with information retrieval, usability (graphical decision-making aids), learning Bayes factors from historical data, assessing quality of information and determining conditional probabilities for the so-called 'indicators' of causation for E-Synthesis. Vice versa, E-Synthesis offers a solid methodological basis for (semi-)automated evidence aggregation with AI systems.

CONCLUSIONS

Properly applied, AI can help the transition of philosophical principles and considerations concerning evidence aggregation for drug safety to a tool that can be used in practice.

Certainty of evidence and intervention's benefits and harms are key determinants of guidelines' recommendations

OBJECTIVE

Many factors are postulated to affect guidelines developments. We set out to identify the key determinants.

STUDY DESIGN AND SETTING

a) Web-based survey of 12 panels of 153 "voting" members who issued 2941 recommendations; b) qualitative analysis of 13 panels of 311 attendees (panel members, systematic review teams and observers).

RESULTS

Compared with "no recommendations", when intervention's benefit outweigh harms (BH-balance), probability of issuing strong recommendations in favor of intervention was 0.22 (95%CI: 0.08 to 0.36) when certainty of evidence (CoE) was very low; 0.5 (95%CI:0.36 to 0.63) when low; 0.74 (95%CI 0.61 to 0.87) when moderate and 0.85 (95%CI:0.71 to 1.00) when high. No other postulated factor significantly affected recommendations. The findings are consistent with a J- curve model when recommendations are issued in favor but not against an intervention. Panelists often changed their judgments as a result of the meeting discussion (67% for CoE to 92% for balance between benefits and harms). The panels spent over 50% of their time debating CoE; the chairs and co-chairs dominated discussion.

CONCLUSIONS

CoE and BH-balance are key determinants of recommendations in favor of an intervention. Chairs and co-chairs dominate discussion. Panelists often change their judgments as a result of panel deliberation.

Associations of workflow disruptions in the operating room with surgical outcomes: a systematic review and narrative synthesis

BACKGROUND

Performance in the operating room is an important determinant of surgical safety. Flow disruptions (FDs) represent system-related performance problems that affect the efficiency of the surgical team and have been associated with a risk to patient safety. Despite the growing evidence base on FDs, a systematic synthesis has not yet been published.

OBJECTIVE

Our aim was to identify, evaluate and summarise the evidence on relationships between intraoperative FD events and provider, surgical process and patient outcomes.

METHODS

We systematically searched databases MEDLINE, Embase and PsycINFO (last update: September 2019). Two reviewers independently screened the resulting studies at the title/abstract and full text stage in duplicate, and all inconsistencies were resolved through discussion. We assessed the risk of bias of included studies using established and validated tools. We summarised effects from included studies through a narrative synthesis, stratified based on predefined surgical outcome categories, including surgical process, provider and patient outcomes.

RESULTS

We screened a total of 20 481 studies. 38 studies were found to be eligible. Included studies were highly heterogeneous in terms of methodology, medical specialty and context. Across studies, 20.5% of operating time was attributed to FDs. Various other process, patient and provider outcomes were reported. Most studies reported negative or non-significant associations of FDs with surgical outcomes.

CONCLUSION

Apart from the identified relationship of FDs with procedure duration, the evidence base concerning the impact of FDs on provider, surgical process and patient outcomes is limited and heterogeneous. We further provide recommendations concerning use of methods, relevant outcomes and avenues for future research on associated effects of FDs in surgery.

The effect of lactoferrin supplementation on death or major morbidity in very low birthweight infants (LIFT): a multicentre, double-blind, randomised controlled trial

BACKGROUND

Very low birthweight or preterm infants are at increased risk of adverse outcomes including sepsis, necrotising enterocolitis, and death. We assessed whether supplementing the enteral diet of very low-birthweight infants with lactoferrin, an antimicrobial protein, reduces all-cause mortality or major morbidity.

METHODS

We did a multicentre, double-blind, pragmatic, randomised superiority trial in 14 Australian and two New Zealand neonatal intensive care units. Infants born weighing less than 1500 g and aged less than 8 days, were eligible and randomly assigned (1:1) using minimising web-based randomisation to receive once daily 200 mg/kg pasteurised bovine lactoferrin supplements or no lactoferrin supplement added to breast or formula milk until 34 weeks' post-menstrual age (or for 2 weeks, if longer), or until discharge from the study hospital if that occurred first. Designated nurses preparing the daily feeds were not masked to group assignment, but other nurses, doctors, parents, caregivers, and investigators were unaware. The primary outcome was survival to hospital discharge or major morbidity (defined as brain injury, necrotising enterocolitis, late-onset sepsis at 36 weeks' post-menstrual age, or retinopathy treated before discharge) assessed in the intention-to-treat population. Safety analyses were by treatment received. We also did a prespecified, PRISMA-compliant meta-analysis, which included this study and other relevant randomised controlled trials, to estimate more precisely the effects of lactoferrin supplementation on late-onset sepsis, necrotising enterocolitis, and survival. This trial is registered with the Australian and New Zealand Clinical Trials Registry, ACTRN12611000247976.

FINDINGS

Between June 27, 2014, and Sept 1, 2017, we recruited 1542 infants; 771 were assigned to the intervention group and 771 to the control group. One infant who had consent withdrawn before beginning lactoferrin treatment was excluded from analysis. In-hospital death or major morbidity occurred in 162 (21%) of 770 infants in the intervention group and in 170 (22%) of 771 infants in the control group (relative risk [RR] 0·95, 95% CI 0·79-1·14; p=0·60). Three suspected unexpected serious adverse reactions occurred; two in the lactoferrin group, namely unexplained late jaundice and inspissated milk syndrome, but were not attributed to the intervention and one in the control group had fatal inspissated milk syndrome. Our meta-analysis identified 13 trials completed before Feb 18, 2020, including this Article, in 5609 preterm infants. Lactoferrin supplements significantly reduced late-onset sepsis (RR 0·79, 95% CI 0·71-0·88; p<0·0001; I²=58%), but not necrotising enterocolitis or all-cause mortality.

INTERPRETATION

Lactoferrin supplementation did not improve death or major morbidity in this trial, but might reduce late-onset sepsis, as found in our meta-analysis of over 5000 infants. Future collaborative studies should use products with demonstrated biological activity, be large enough to detect moderate and clinically important effects reliably, and assess greater doses of lactoferrin in infants at increased risk, such as those not exclusively receiving breastmilk or infants of extremely low birthweight.

FUNDING

Australian National Health and Medical Research Council.

An epistemological problem for integration in EBM

Evidence-based medicine (EBM) calls for medical practitioners to "integrate" our best available evidence into clinical practice. A significant amount of the literature on EBM takes this integration to be unproblematic, focusing on questions like how to interpret evidence and engage with patient values, rather than critically looking at how these features of EBM can be implemented together. Other authors have also commented on this gap in the literature, for example, identifying the lack of clarity about how patient preferences and evidence from trials is supposed to be integrated in practice. In this paper, I look at this issue from an epistemological perspective, (looking at how different types of knowledge in EBM can be used to make sounds judgements). In particular, I introduce an epistemological issue for this integration problem, which I call the epistemic integration problem. This is essentially the problem of how we can use information that is both general (eg, about a population sample) and descriptive (eg, about what expected outcomes are) to reach clinical judgements that are individualized (applying to a particular patient) and normative (about what is best for their health).

Reviewing the Mechanistic Evidence Assessors E-Synthesis and EBM+: A Case Study of Amoxicillin and Drug Reaction with Eosinophilia and Systemic Symptoms (DRESS)

Background:

Basic science has delivered unprecedented insights into intricate relationships on the smallest scales within well-controlled environments. Addressing pressing societal decision problems requires an understanding of systems on larger scales in real-world situations.

Objective:

To assess how well the evidence assessors E-Synthesis and EBM+ assess basic science findings to support medical decision making.

Method:

We demonstrate the workings of E-Synthesis and EBM+ on a case study: the suspected causal connection between the widely-used drug amoxicillin (AMX) and the putative adverse drug reaction: Drug Reaction with Eosinophilia and Systemic Symptoms (DRESS).

Results:

We determine an increase in the probability that AMX can cause DRESS within the E-Synthesis approach and using the EBM+ standards assess the basic science findings as supporting the existence of a mechanism linking AMX and DRESS.

Conclusions:

While progress is made towards developing methodologies which allow the incorporation of basic science research in the decision making process for pressing societal questions, there is still considerable need for further developments. A continued dialogue between basic science researchers and methodologists, philosophers and statisticians seems to offer the best prospects for developing and evaluating continuously evolving methodologies.

What confidence should we have in GRADE?

RATIONALE, AIMS, AND OBJECTIVES

Confidence (or belief) that a therapy is effective is essential to practicing clinical medicine. GRADE, a popular framework for developing clinical recommendations, provides a means for assigning how much confidence one should have in a therapy's effect estimate. One's level of confidence (or "degree of belief") can also be modelled using Bayes theorem. In this paper, we look through both a GRADE and Bayesian lens to examine how one determines confidence in the effect estimate.

METHODS

Philosophical examination.

RESULTS

The GRADE framework uses a criteria-based method to assign a quality of evidence level. The criteria pertain mostly to considerations of methodological rigour, derived from a modified evidence-based medicine evidence hierarchy. The four levels of quality relate to the level of confidence one should have in the effect estimate. The Bayesian framework is not bound by a predetermined set of criteria. Bayes theorem shows how a rational agent adjusts confidence (ie, degree of belief) in the effect estimate on the basis of the available evidence. Such adjustments relate to the principles of incremental confirmation and evidence proportionism. Use of the Bayesian framework reveals some potential pitfalls in GRADE's criteria-based thinking on confidence that are out of step with our intuitions on evidence.

CONCLUSIONS

A rational thinker uses all available evidence to formulate beliefs. The GRADE criteria seem to suggest that we discard some of that information when other, more favoured information (eg, derived from clinical trials) is available. The GRADE framework should strive to ensure that the whole evidence base is considered when determining confidence in the effect estimate. The incremental value of such evidence on determining confidence in the effect estimate should be assigned in a manner that is theoretically or empirically justified, such that confidence is proportional to the evidence, both for and against it.

Treating real people: Science and humanity

Something important is happening in applied, interdisciplinary research, particularly in the field of applied health research. The vast array of papers in this edition are evidence of a broad change in thinking across an impressive range of practice and academic areas. The problems of complexity, the rise of chronic conditions, overdiagnosis, co-morbidity, and multi-morbidity are serious and challenging, but we are rising to that challenge. Key conceptions regarding science, evidence, disease, clinical judgement, and health and social care are being revised and their relationships reconsidered: Boundaries are indeed being redrawn; reasoning is being made "fit for practice." Ideas like "person-centred care" are no longer phrases with potential to be helpful in some yet-to-be-clarified way: Theorists and practitioners are working in collaboration to give them substantive import and application.

From MARS to MAGIC: The remarkable journey through time and space of the Grading of Recommendations Assessment, Development and Evaluation initiative

For over 30 years, "evidence-based" clinical guidelines remained entrenched in an oversimplified, design-based, framework for rating the strength of evidence supporting clinical recommendations. The approach frequently equated the rating of evidence with that of the recommendations themselves. "Grading Recommendations Assessment, Development and Evaluation (GRADE)" has emerged as a proposed antidote to obsolete guideline methodology. GRADE sponsors and collaborators are in the process of attempting to amplify and extend the framework to encompass implementation and adaptation of guidelines, above and beyond the evaluation and rating of clinical research. Alternative schemes and models for such extensions are beginning to appear. This commentary reviews the strengths and weaknesses of GRADE with reference to other recent critiques. It considers the GRADE Working Group's "evidence-to-decision" extension of the evidence rating framework, together with proposed alternatives. It identifies pitfalls of the GRADE system's cooptation of relational processes necessary to the interpretation and uptake of recommendations that properly belong to end-users. It also identifies dangers inherent in blurring important boundaries between clinical and policy applications of guidelines. Finally, it addresses criticisms regarding the lack of a theoretical framework supporting the different facets of the GRADE approach and proposes a social constructivist orientation to clinical guideline development and use. Recommendations are offered to potential guideline developers and users regarding how to draw upon the strengths of the GRADE framework without succumbing to its pitfalls.

The evolution of GRADE (part 2): Still searching for a theoretical and/or empirical basis for the GRADE framework

RATIONALE, AIMS, AND OBJECTIVES

The GRADE framework has been widely adopted as the preferred method for developing clinical practice recommendations. In the first article of our three part series examining the evolution of GRADE, we showed an absence (in the first two versions of GRADE) of a theoretical basis and/or empirical data to support why the presented criteria for determining the quality of evidence regarding the effect estimate and the components under consideration for determining the strength of the recommendation were included and other criteria/components excluded. Furthermore, often, it was not clear how to operationalize the included criteria/components (and integrate them) when using the framework. In part 2 of this series, we examine if version 3 of GRADE offered improvements on previous versions with respect to a justification scheme and how to operationalize the framework's criteria/components.

METHODS

Narrative review.

RESULTS

Our examination suggests that version 3 has done little to improve on the justification scheme that sustains GRADE. Still absent is a justification (theoretical and/or empirical) for why the criteria/components were chosen. Likewise, version 3 is still lacking clarity regarding how to implement and integrate the criteria/considerations in the framework (ie, operationalize the framework) when determining the quality of evidence or strength of recommendation. Transparency is now emphasized as the merit of GRADE. However, we are offered no theoretical justification for how the use of GRADE should achieve transparency or empirical evidence to support that transparency is achieved.

CONCLUSIONS

While version 3 reveals acknowledgement by the authors of GRADE that the framework is a work in progress, it still lacks a justification scheme (theoretical and/or empirical) to sustain it and clarity in its criteria/components to operationalize it. As was suggested in part 1, such issues limit one's ability to scientifically assess the appropriateness of GRADE for its stated purpose.

How do we know if a clinical practice guideline is good? A response to Djulbegovic and colleagues' use of fast-and-frugal decision trees to improve clinical care strategies

Clinical practice guidelines (CPGs) and clinical pathways have become important tools for improving the uptake of evidence-based care. Where CPGs are good, adherence to the recommendations within is thought to result in improved patient outcomes. However, the usefulness of such tools for improving patient important outcomes depends both on adherence to the guideline and whether or not the CPG in question is good. This begs the question of what it is that makes a CPG good? In this issue of the Journal, Djulbegovic and colleagues offer a theory to help guide the development of CPGs. The "fast-and-frugal tree" (FFT) heuristic theory is purported to provide the theoretical structure needed to quantitatively assess clinical guidelines in practice, something that the lack of theory to guide CPG development has precluded. In this paper, I examine the role of FFTs in providing an adequate theoretical framework for developing CPGs. In my view, positioning guideline development within the FFT framework may help with problems related to adherence. However, I believe that FTTs fall short in providing panel members with the theoretical basis needed to justify which factors should be considered when developing a CPG, how information on those factors derived from research studies should be interpreted, and how those factors should be integrated into the recommendation.

The evolution of GRADE (part 1): Is there a theoretical and/or empirical basis for the GRADE framework?

RATIONALE, AIMS, AND OBJECTIVES

The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) framework has been presented as the best method available for developing clinical recommendations. GRADE has undergone a series of modifications. Here, we present the first part of a three article series examining the evolution of GRADE. Our purpose is to explore if (and if so, how) GRADE provides: (1) a justification (ie, theoretical and/or empirical) for why the criteria/components under consideration in the system are included (and other factors excluded), as well as why some criteria/components where added/modified in the evolution process, (2) clear and functional (ie, how to operationalize them) definitions of the included criteria/components, and (3) instruction and justification for how all the criteria/components are to be integrated when determining a recommendation. In part 1 of the series, we examine the first two versions of GRADE.

METHODS

Narrative review.

RESULTS

The justification scheme that sustains GRADE is not articulated in the first two versions of the framework. Why some criteria/components were included, and others excluded, is not justified theoretically nor is empirical support provided to suggest that the framework as presented includes that which is needed to produce valid recommendations. The first two versions of GRADE show a lack of clear instruction on how to operationalize the criteria for assessing the quality of evidence and the components for making a recommendation (including how to integrate the criteria/components at each step), which leaves substantial room for judgement on the part of the user of GRADE for guideline development.

CONCLUSIONS

This article revealed an absence of a justification (theoretical and/or empirical) to support important aspects of the GRADE framework, as well as a lack of clear instruction on how to operationalize the criteria and components in the framework. These issues limit one's ability to scientifically assess the appropriateness of GRADE for determining clinical recommendations.

Transforming clinical practice guidelines and clinical pathways into fast-and-frugal decision trees to improve clinical care strategies

BACKGROUND

Contemporary delivery of health care is inappropriate in many ways, largely due to suboptimal Q5 decision-making. A typical approach to improve practitioners' decision-making is to develop evidence-based clinical practice guidelines (CPG) by guidelines panels, who are instructed to use their judgments to derive practice recommendations. However, mechanisms for the formulation of guideline judgments remains a "black-box" operation-a process with defined inputs and outputs but without sufficient knowledge of its internal workings.

METHODS

Increased explicitness and transparency in the process can be achieved by implementing CPG as clinical pathways (CPs) (also known as clinical algorithms or flow-charts). However, clinical recommendations thus derived are typically ad hoc and developed by experts in a theory-free environment. As any recommendation can be right (true positive or negative), or wrong (false positive or negative), the lack of theoretical structure precludes the quantitative assessment of the management strategies recommended by CPGs/CPs.

RESULTS

To realize the full potential of CPGs/CPs, they need to be placed on more solid theoretical grounds. We believe this potential can be best realized by converting CPGs/CPs within the heuristic theory of decision-making, often implemented as fast-and-frugal (FFT) decision trees. This is possible because FFT heuristic strategy of decision-making can be linked to signal detection theory, evidence accumulation theory, and a threshold model of decision-making, which, in turn, allows quantitative analysis of the accuracy of clinical management strategies.

CONCLUSIONS

Fast-and-frugal provides a simple and transparent, yet solid and robust, methodological framework connecting decision science to clinical care, a sorely needed missing link between CPGs/CPs and patient outcomes. We therefore advocate that all guidelines panels express their recommendations as CPs, which in turn should be converted into FFTs to guide clinical care.