Defining decision thresholds for judgments on health benefits and harms using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence to Decision (EtD) frameworks: a protocol for a randomised methodological study (GRADE-THRESHOLD)

Immunomodulators and immunosuppressants for progressive multiple sclerosis: a network meta-analysis

BACKGROUND

In recent years a broader range of immunomodulatory and immunosuppressive treatment options have emerged for people with progressive forms of multiple sclerosis (PMS). While consensus supports these options as reducing relapses, their relative benefit and safety profiles remain unclear due to a lack of direct comparison trials.

OBJECTIVES

To compare through network meta-analysis the efficacy and safety of alemtuzumab, azathioprine, cladribine, cyclophosphamide, daclizumab, dimethylfumarate, diroximel fumarate, fingolimod, fludarabine, glatiramer acetate, immunoglobulins, interferon beta 1-a and beta 1-b, interferon beta-1b (Betaferon), interferon beta-1a (Avonex, Rebif), laquinimod, leflunomide, methotrexate, minocycline, mitoxantrone, mycophenolate mofetil, natalizumab, ocrelizumab, ofatumumab, ozanimod, pegylated interferon beta-1a, ponesimod, rituximab, siponimod, corticosteroids, and teriflunomide for PMS.

SEARCH METHODS

We searched CENTRAL, MEDLINE, and Embase up to August 2022, as well as ClinicalTrials.gov and the WHO ICTRP.

SELECTION CRITERIA

Randomised controlled trials (RCTs) that studied one or more treatments as monotherapy, compared to placebo or to another active agent, for use in adults with PMS.

DATA COLLECTION AND ANALYSIS

Two review authors independently selected studies and extracted data. We performed data synthesis by pair-wise and network meta-analysis. We assessed the certainty of the body of evidence according to GRADE.

MAIN RESULTS

We included 23 studies involving a total of 10,167 participants. The most frequent (39% of studies) reason for a rating of high risk of bias was sponsor role in study authorship and data management and analysis. Other concerns were performance, attrition, and selective reporting bias, with 8.7% of studies at high risk of bias for all three of these domains. The common comparator for network analysis was placebo. Relapses over 12 months: assessed in one study (318 participants). None of the treatments assessed showed moderate or high certainty evidence compared to placebo. Relapses over 24 months: assessed in six studies (1622 participants). The number of people with clinical relapses is probably trivially reduced with rituximab (risk ratio (RR) 0.60, 95% confidence interval (CI) 0.19 to 1.95; moderate certainty evidence). None of the remaining treatments assessed showed moderate or high certainty evidence compared to placebo. Relapses over 36 months: assessed in four studies (2095 participants). The number of people with clinical relapses is probably trivially reduced with interferon beta-1b (RR 0.82, 95% CI 0.73 to 0.93; moderate certainty evidence). None of the remaining treatments assessed showed moderate or high certainty evidence compared to placebo. Disability worsening over 24 months: assessed in 11 studies (5284 participants). None of the treatments assessed showed moderate or high certainty evidence compared to placebo. Disability worsening over 36 months: assessed in five studies (2827 participants). None of the treatments assessed showed moderate or high certainty evidence compared to placebo. Serious adverse events: assessed in 15 studies (8019 participants). None of the treatments assessed showed moderate or high certainty evidence compared to placebo. Discontinuation due to adverse events: assessed in 21 studies (9981 participants). The number of people who discontinued treatment due to adverse events is trivially increased with interferon beta-1a (odds ratio (OR) 2.93, 95% CI 1.64 to 5.26; high certainty evidence). The number of people who discontinued treatment due to adverse events is probably trivially increased with rituximab (OR 4.00, 95% CI 0.84 to 19.12; moderate certainty evidence); interferon beta-1b (OR 2.98, 95% CI 1.92 to 4.61; moderate certainty evidence); immunoglobulins (OR 1.95, 95% CI 0.99 to 3.84; moderate certainty evidence); glatiramer acetate (OR 3.98, 95% CI 1.48 to 10.72; moderate certainty evidence); natalizumab (OR 1.02, 95% CI 0.55 to 1.90; moderate certainty evidence); siponimod (OR 1.53, 95% CI 0.98 to 2.38; moderate certainty evidence); fingolimod (OR 2.29, 95% CI 1.46 to 3.60; moderate certainty evidence), and ocrelizumab (OR 1.24, 95% CI 0.54 to 2.86; moderate certainty evidence). None of the remaining treatments assessed showed moderate or high certainty evidence compared to placebo.

AUTHORS' CONCLUSIONS

The number of people with PMS with relapses is probably slightly reduced with rituximab at two years, and interferon beta-1b at three years, compared to placebo. Both drugs are also probably associated with a slightly higher proportion of withdrawals due to adverse events, as are immunoglobulins, glatiramer acetate, natalizumab, fingolimod, siponimod, and ocrelizumab; we have high confidence that this is the case with interferon beta-1a. We found only low or very low certainty evidence relating to disability progression for the included disease-modifying treatments compared to placebo, largely due to imprecision. We are also uncertain about the effect of interventions on serious adverse events, also because of imprecision. These findings are due in part to the short follow-up of the included RCTs, which lacked detection of less common severe adverse events. Moreover, the funding source of many included studies may have introduced bias into the results. Future research on PMS should include head-to-head rather than placebo-controlled trials, with a longer follow-up of at least three years. Given the relative rarity of PMS, controlled, non-randomised studies on large samples may usefully integrate data from pivotal RCTs. Outcomes valuable and meaningful to people with PMS should be consistently adopted and measured to permit the evaluation of relative effectiveness among treatments.

COMPAR-EU Recommendations on Self-Management Interventions in Type 2 Diabetes Mellitus

Self-management interventions (SMIs) offer a promising approach to actively engage patients in the management of their chronic diseases. Within the scope of the COMPAR-EU project, our goal is to provide evidence-based recommendations for the utilisation and implementation of SMIs in the care of adult individuals with type 2 diabetes mellitus (T2DM). A multidisciplinary panel of experts, utilising a core outcome set (COS), identified critical outcomes and established effect thresholds for each outcome. The panel formulated recommendations using the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach, a transparent and rigorous framework for developing and presenting the best available evidence for the formulation of recommendations. All recommendations are based on systematic reviews (SR) of the effects and of values and preferences, a contextual analysis, and a cost-effectiveness analysis. The COMPAR-EU panel is in favour of using SMIs rather than usual care (UC) alone (conditional, very low certainty of the evidence). Furthermore, the panel specifically is in favour of using ten selected SMIs, rather than UC alone (conditional, low certainty of the evidence), mostly encompassing education, self-monitoring, and behavioural techniques. The panel acknowledges that, for most SMIs, moderate resource requirements exist, and cost-effectiveness analyses do not distinctly favour either the SMI or UC. Additionally, it recognises that SMIs are likely to enhance equity, deeming them acceptable and feasible for implementation.

Testosterone replacement in men with sexual dysfunction

BACKGROUND

Clinical practice guidelines recommend testosterone replacement therapy (TRT) for men with sexual dysfunction and testosterone deficiency. However, TRT is commonly promoted in men without testosterone deficiency and existing trials often do not clearly report participants' testosterone levels or testosterone-related symptoms. This review assesses the potential benefits and harms of TRT in men presenting with complaints of sexual dysfunction.

OBJECTIVES

To assess the effects of testosterone replacement therapy compared to placebo or other medical treatments in men with sexual dysfunction.

SEARCH METHODS

We performed a comprehensive search of CENTRAL (the Cochrane Library), MEDLINE, EMBASE, and the trials registries ClinicalTrials.gov and World Health Organization International Clinical Trials Registry Platform, with no restrictions on language of publication or publication status, up to 29 August 2023.

SELECTION CRITERIA

We included randomized controlled trials (RCTs) in men (40 years or over) with sexual dysfunction. We excluded men with primary or secondary hypogonadism. We compared testosterone or testosterone with phosphodiesterase-5 inhibitors (PDEI5I) to placebo or PDE5I alone.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened the literature, assessed the risk of bias, extracted data, and rated the certainty of evidence (CoE) according to GRADE using a minimally contextualized approach. We performed statistical analyses using a random-effects model and interpreted them according to standard Cochrane methodology. Predefined primary outcomes were self-reported erectile dysfunction assessed by a validated instrument, sexual quality of life assessed by a validated instrument, and cardiovascular mortality. Secondary outcomes were treatment withdrawal due to adverse events, prostate-related events, and lower urinary tract symptoms (LUTS). We distinguished between short-term (up to 12 months) and long-term (> 12 months) outcomes.

MAIN RESULTS

We identified 43 studies with 11,419 randomized participants across three comparisons: testosterone versus placebo, testosterone versus PDE5I, and testosterone with PDE5I versus PDE5I alone. This abstract focuses on the most relevant comparison of testosterone versus placebo. Testosterone versus placebo (up to 12 months) Based on a predefined sensitivity analysis of studies at low risk of bias, and an analysis combing data from the similar International Index of Erectile Function (IIEF-EF) and IIEF-5 instruments, TRT likely results in little to no difference in erectile function assessed with the IIEF-EF (mean difference (MD) 2.37, 95% confidence interval (CI) 1.67 to 3.08; I² = 0%; 6 RCTs, 2016 participants; moderate CoE) on a scale from 6 to 30 with larger values reflecting better erectile function. We assumed a minimal clinically important difference (MCID) of greater than or equal to 4. TRT likely results in little to no change in sexual quality of life assessed with the Aging Males' Symptoms scale (MD -2.31, 95% CI -3.63 to -1.00; I² = 0%; 5 RCTs, 1030 participants; moderate CoE) on a scale from 17 to 85 with larger values reflecting worse sexual quality of life. We assumed a MCID of greater than or equal to 10. TRT also likely results in little to no difference in cardiovascular mortality (risk ratio (RR) 0.83, 95% CI 0.21 to 3.26; I² = 0%; 10 RCTs, 3525 participants; moderate CoE). Based on two cardiovascular deaths in the placebo group and an assumed MCID of 3%, this would correspond to no additional deaths per 1000 men (95% CI 1 fewer to 4 more). TRT also likely results in little to no difference in treatment withdrawal due to adverse events, prostate-related events, or LUTS. Testosterone versus placebo (later than 12 months) We are very uncertain about the longer-term effects of TRT on erectile dysfunction assessed with the IIEF-EF (MD 4.20, 95% CI -2.03 to 10.43; 1 study, 42 participants; very low CoE). We did not find studies reporting on sexual quality of life or cardiovascular mortality. We are very uncertain about the effect of testosterone on treatment withdrawal due to adverse events. We found no studies reporting on prostate-related events or LUTS.

AUTHORS' CONCLUSIONS

In the short term, TRT probably has little to no effect on erectile function, sexual quality of life, or cardiovascular mortality compared to a placebo. It likely results in little to no difference in treatment withdrawals due to adverse events, prostate-related events, or LUTS. In the long term, we are very uncertain about the effects of TRT on erectile function when compared to placebo; we did not find data on its effects on sexual quality of life or cardiovascular mortality. The certainty of evidence ranged from moderate (signaling that we are confident that the reported effect size is likely to be close to the true effect) to very low (indicating that the true effect is likely to be substantially different). The findings of this review should help to inform future guidelines and clinical decision-making at the point of care.

Immunomodulators and immunosuppressants for relapsing-remitting multiple sclerosis: a network meta-analysis

BACKGROUND

Different therapeutic strategies are available for the treatment of people with relapsing-remitting multiple sclerosis (RRMS), including immunomodulators, immunosuppressants and biological agents. Although each one of these therapies reduces relapse frequency and slows disability accumulation compared to no treatment, their relative benefit remains unclear. This is an update of a Cochrane review published in 2015.

OBJECTIVES

To compare the efficacy and safety, through network meta-analysis, of interferon beta-1b, interferon beta-1a, glatiramer acetate, natalizumab, mitoxantrone, fingolimod, teriflunomide, dimethyl fumarate, alemtuzumab, pegylated interferon beta-1a, daclizumab, laquinimod, azathioprine, immunoglobulins, cladribine, cyclophosphamide, diroximel fumarate, fludarabine, interferon beta 1-a and beta 1-b, leflunomide, methotrexate, minocycline, mycophenolate mofetil, ofatumumab, ozanimod, ponesimod, rituximab, siponimod and steroids for the treatment of people with RRMS.

SEARCH METHODS

CENTRAL, MEDLINE, Embase, and two trials registers were searched on 21 September 2021 together with reference checking, citation searching and contact with study authors to identify additional studies. A top-up search was conducted on 8 August 2022.

SELECTION CRITERIA

Randomised controlled trials (RCTs) that studied one or more of the available immunomodulators and immunosuppressants as monotherapy in comparison to placebo or to another active agent, in adults with RRMS.

DATA COLLECTION AND ANALYSIS

Two authors independently selected studies and extracted data. We considered both direct and indirect evidence and performed data synthesis by pairwise and network meta-analysis. Certainty of the evidence was assessed by the GRADE approach.

MAIN RESULTS

We included 50 studies involving 36,541 participants (68.6% female and 31.4% male). Median treatment duration was 24 months, and 25 (50%) studies were placebo-controlled. Considering the risk of bias, the most frequent concern was related to the role of the sponsor in the authorship of the study report or in data management and analysis, for which we judged 68% of the studies were at high risk of other bias. The other frequent concerns were performance bias (34% judged as having high risk) and attrition bias (32% judged as having high risk). Placebo was used as the common comparator for network analysis. Relapses over 12 months: data were provided in 18 studies (9310 participants). Natalizumab results in a large reduction of people with relapses at 12 months (RR 0.52, 95% CI 0.43 to 0.63; high-certainty evidence). Fingolimod (RR 0.48, 95% CI 0.39 to 0.57; moderate-certainty evidence), daclizumab (RR 0.55, 95% CI 0.42 to 0.73; moderate-certainty evidence), and immunoglobulins (RR 0.60, 95% CI 0.47 to 0.79; moderate-certainty evidence) probably result in a large reduction of people with relapses at 12 months. Relapses over 24 months: data were reported in 28 studies (19,869 participants). Cladribine (RR 0.53, 95% CI 0.44 to 0.64; high-certainty evidence), alemtuzumab (RR 0.57, 95% CI 0.47 to 0.68; high-certainty evidence) and natalizumab (RR 0.56, 95% CI 0.48 to 0.65; high-certainty evidence) result in a large decrease of people with relapses at 24 months. Fingolimod (RR 0.54, 95% CI 0.48 to 0.60; moderate-certainty evidence), dimethyl fumarate (RR 0.62, 95% CI 0.55 to 0.70; moderate-certainty evidence), and ponesimod (RR 0.58, 95% CI 0.48 to 0.70; moderate-certainty evidence) probably result in a large decrease of people with relapses at 24 months. Glatiramer acetate (RR 0.84, 95%, CI 0.76 to 0.93; moderate-certainty evidence) and interferon beta-1a (Avonex, Rebif) (RR 0.84, 95% CI 0.78 to 0.91; moderate-certainty evidence) probably moderately decrease people with relapses at 24 months. Relapses over 36 months findings were available from five studies (3087 participants). None of the treatments assessed showed moderate- or high-certainty evidence compared to placebo. Disability worsening over 24 months was assessed in 31 studies (24,303 participants). Natalizumab probably results in a large reduction of disability worsening (RR 0.59, 95% CI 0.46 to 0.75; moderate-certainty evidence) at 24 months. Disability worsening over 36 months was assessed in three studies (2684 participants) but none of the studies used placebo as the comparator. Treatment discontinuation due to adverse events data were available from 43 studies (35,410 participants). Alemtuzumab probably results in a slight reduction of treatment discontinuation due to adverse events (OR 0.39, 95% CI 0.19 to 0.79; moderate-certainty evidence). Daclizumab (OR 2.55, 95% CI 1.40 to 4.63; moderate-certainty evidence), fingolimod (OR 1.84, 95% CI 1.31 to 2.57; moderate-certainty evidence), teriflunomide (OR 1.82, 95% CI 1.19 to 2.79; moderate-certainty evidence), interferon beta-1a (OR 1.48, 95% CI 0.99 to 2.20; moderate-certainty evidence), laquinimod (OR 1.49, 95 % CI 1.00 to 2.15; moderate-certainty evidence), natalizumab (OR 1.57, 95% CI 0.81 to 3.05), and glatiramer acetate (OR 1.48, 95% CI 1.01 to 2.14; moderate-certainty evidence) probably result in a slight increase in the number of people who discontinue treatment due to adverse events. Serious adverse events (SAEs) were reported in 35 studies (33,998 participants). There was probably a trivial reduction in SAEs amongst people with RRMS treated with interferon beta-1b as compared to placebo (OR 0.92, 95% CI 0.55 to 1.54; moderate-certainty evidence).

AUTHORS' CONCLUSIONS

We are highly confident that, compared to placebo, two-year treatment with natalizumab, cladribine, or alemtuzumab decreases relapses more than with other DMTs. We are moderately confident that a two-year treatment with natalizumab may slow disability progression. Compared to those on placebo, people with RRMS treated with most of the assessed DMTs showed a higher frequency of treatment discontinuation due to AEs: we are moderately confident that this could happen with fingolimod, teriflunomide, interferon beta-1a, laquinimod, natalizumab and daclizumab, while our certainty with other DMTs is lower. We are also moderately certain that treatment with alemtuzumab is associated with fewer discontinuations due to adverse events than placebo, and moderately certain that interferon beta-1b probably results in a slight reduction in people who experience serious adverse events, but our certainty with regard to other DMTs is lower. Insufficient evidence is available to evaluate the efficacy and safety of DMTs in a longer term than two years, and this is a relevant issue for a chronic condition like MS that develops over decades. More than half of the included studies were sponsored by pharmaceutical companies and this may have influenced their results. Further studies should focus on direct comparison between active agents, with follow-up of at least three years, and assess other patient-relevant outcomes, such as quality of life and cognitive status, with particular focus on the impact of sex/gender on treatment effects.

GRADE guidance 37: rating imprecision in a body of evidence on test accuracy

OBJECTIVES

To provide guidance on rating imprecision in a body of evidence assessing the accuracy of a single test. This guide will clarify when Grading of Recommendations Assessment, Development and Evaluation (GRADE) users should consider rating down the certainty of evidence by one or more levels for imprecision in test accuracy.

STUDY DESIGN AND SETTING

A project group within the GRADE working group conducted iterative discussions and presentations at GRADE working group meetings to produce this guidance.

RESULTS

Before rating the certainty of evidence, GRADE users should define the target of their certainty rating. GRADE recommends setting judgment thresholds defining what they consider a very accurate, accurate, inaccurate, and very inaccurate test. These thresholds should be set after considering consequences of testing and effects on people-important outcomes. GRADE's primary criterion for judging imprecision in test accuracy evidence is considering confidence intervals (i.e., CI approach) of absolute test accuracy results (true and false, positive, and negative results in a cohort of people). Based on the CI approach, when a CI appreciably crosses the predefined judgment threshold(s), one should consider rating down certainty of evidence by one or more levels, depending on the number of thresholds crossed. When the CI does not cross judgment threshold(s), GRADE suggests considering the sample size for an adequately powered test accuracy review (optimal or review information size [optimal information size (OIS)/review information size (RIS)]) in rating imprecision. If the combined sample size of the included studies in the review is smaller than the required OIS/RIS, one should consider rating down by one or more levels for imprecision.

CONCLUSION

This paper extends previous GRADE guidance for rating imprecision in single test accuracy systematic reviews and guidelines, with a focus on the circumstances in which one should consider rating down one or more levels for imprecision.

A novel framework for incorporating patient values and preferences in making guideline recommendations: guideline panel surveys

OBJECTIVE

Universally acknowledged standards for trustworthy guidelines include the necessity to ground recommendations in patient values and preferences. When information is limited-which is typically the case-guideline panels often find it difficult to explicitly integrate patient values and preferences into their recommendations. Our objective was to develop and evaluate a framework for systematically navigating guideline panels in incorporating patient values and preferences in making recommendations.

STUDY DESIGN AND SETTING

In the context of developing a guideline for colorectal cancer screening, we generated an initial framework for creating panel surveys to elicit guideline panelists' views of patient values and preferences and to inform panel discussions on recommendations. With further applications in guidelines of diverse topic areas, we dynamically refined the framework through iterative discussions and consensus.

RESULTS

The finial framework consists of five steps for creating and implementing panel surveys. The surveys can serve three objectives following from the quantitative information regarding patient values and preferences that guideline panels usually require. An accompanying video provides detailed instructions of the survey.

CONCLUSION

The framework for creating and implementing panel surveys offers explicit guidance for guideline panels considering transparently and systematically incorporating patient values and preferences into guideline recommendations.

Photodynamic Diagnosis-Assisted Transurethral Resection - Lights Out?

In a randomized trial and accompanying cost-effectiveness analysis, conducted in people with suspected urothelial carcinoma of the bladder, Heer at al.1 in this issue of NEJM Evidence compare transurethral resection aided by intravesical installation of a photosensitizer with the same procedure guided by standard white light. The study's main finding was that the photosensitizer-based approach was not associated with a reduction of time to recurrence of tumor. Moreover, patients undergoing the photosensitizer-based procedure had only a small absolute risk reduction (3.8 percentage points; 95% confidence interval [CI], -13.4 to 6.6) of tumor recurrence in the trial time frame.

Using explicit thresholds for benefits and harms in partially contextualized GRADE guidelines. Pilot experience from a living COVID-19 guideline

Objectives

Guideline panels must assess the magnitude of health benefits and harms to develop sensible recommendations. However, they rarely use explicit thresholds. In this paper we report on the piloting and the use thresholds for benefits and harms.

Study Design and Setting

We piloted the use of thresholds in a Chilean COVID-19 living guideline. For each of the critical outcomes, we asked panelists to suggest values of the thresholds for large, moderate, small, or trivial or no effect. We collected this information through a survey and an on-line discussion.

Results

Twelve panelists decided on thresholds for three critical outcomes (mortality, need for mechanical ventilation and serious adverse events). For all outcomes, an absolute risk reduction was considered larger with more than 50 events, moderate with less than 50 events, small with less than 25 events, and trivial with less than 10 events. Having these a priori thresholds in place significantly impacted on the development of recommendations.

Conclusion

Explicit thresholds were a valuable addition to the judgment of the certainty in the evidence, to decide the direction and strength of the recommendation and to evaluate the need for update. We believe this is a line of research worth perusing.