Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants

An international Delphi consensus process to determine a common data element and core outcome set for frailty: FOCUS (The Frailty Outcomes Consensus Project)

Background

Despite increased recognition of frailty and its importance, high quality evidence to guide decision-making is lacking. There has been variation in reported data elements and outcomes which makes it challenging to interpret results across studies as well as to generalize research findings. The creation of a frailty core set, consisting of a minimum set of data elements and outcomes to be measured in all frailty studies, would allow for findings from research and translational studies to be collectively analyzed to better inform care and decision-making. To achieve this, the Frailty Outcomes Consensus Project was developed to reach consensus from the international frailty community on a set of common data elements and core outcomes for frailty.

Methods

An international steering committee developed the methodology and the consensus process to be followed. The committee formulated the initial list of data elements and outcomes. Participants from across the world were invited to take part in the Delphi consensus process. The Delphi consisted of three rounds. Following review of data after three rounds, a final ranking round of data elements and outcomes was conducted. A required retention rate of 80% between rounds was set a priori.

Results

One hundred and eighty-four panelists from 25 different countries participated in the first round of the Delphi consensus process. This included researchers, clinicians, administrators, older adults, and caregivers. The retention rate between rounds was achieved. Data elements and outcomes forming primary and secondary core sets were identified, within the domains of participant characteristics, physical performance, physical function, physical health, cognition and mental health, socioenvironmental circumstances, frailty measures, and other.

Conclusion

It is anticipated that implementation and uptake of the frailty core set will enable studies to be collectively analyzed to better inform care for persons living with frailty and ultimately improve their outcomes. Future work will focus on identification of measurement tools to be used in the application of the frailty core set.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12877-022-02993-w.

Usefulness of Cochrane Reviews in Clinical Guideline Development-A Survey of 585 Recommendations

The Danish Health Authority develops clinical practice guidelines to support clinical decision-making based on the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system and prioritizes using Cochrane reviews. The objective of this study was to explore the usefulness of Cochrane reviews as a source of evidence in the development of clinical recommendations. Evidence-based recommendations in guidelines published by the Danish Health Authority between 2014 and 2021 were reviewed. For each recommendation, it was noted if and how Cochrane reviews were utilized. In total, 374 evidence-based recommendations and 211 expert consensus recommendations were published between 2014 and 2021. Of the 374 evidence-based recommendations, 106 included evidence from Cochrane reviews. In 28 recommendations, all critical and important outcomes included evidence from Cochrane reviews. In 36 recommendations, a minimum of all critical outcomes included evidence from Cochrane reviews, but not all important outcomes. In 33 recommendations, some but not all critical outcomes included evidence from Cochrane reviews. Finally, in nine recommendations, some of the important outcomes included evidence from Cochrane reviews. In almost one-third of the evidence-based recommendations, Cochrane reviews were used to inform clinical recommendations. This evaluation should inform future evaluations of Cochrane review uptake in clinical practice guidelines concerning outcomes important for clinical decision-making.

Automated Identification of Common Disease-Specific Outcomes for Comparative Effectiveness Research Using ClinicalTrials.gov: Algorithm Development and Validation Study

Background

Common disease-specific outcomes are vital for ensuring comparability of clinical trial data and enabling meta analyses and interstudy comparisons. Traditionally, the process of deciding which outcomes should be recommended as common for a particular disease relied on assembling and surveying panels of subject-matter experts. This is usually a time-consuming and laborious process.

Objective

The objectives of this work were to develop and evaluate a generalized pipeline that can automatically identify common outcomes specific to any given disease by finding, downloading, and analyzing data of previous clinical trials relevant to that disease.

Methods

An automated pipeline to interface with ClinicalTrials.gov’s application programming interface and download the relevant trials for the input condition was designed. The primary and secondary outcomes of those trials were parsed and grouped based on text similarity and ranked based on frequency. The quality and usefulness of the pipeline’s output were assessed by comparing the top outcomes identified by it for chronic obstructive pulmonary disease (COPD) to a list of 80 outcomes manually abstracted from the most frequently cited and comprehensive reviews delineating clinical outcomes for COPD.

Results

The common disease-specific outcome pipeline successfully downloaded and processed 3876 studies related to COPD. Manual verification indicated that the pipeline was downloading and processing the same number of trials as were obtained from the self-service ClinicalTrials.gov portal. Evaluating the automatically identified outcomes against the manually abstracted ones showed that the pipeline achieved a recall of 92% and precision of 79%. The precision number indicated that the pipeline was identifying many outcomes that were not covered in the literature reviews. Assessment of those outcomes indicated that they are relevant to COPD and could be considered in future research.

Conclusions

An automated evidence-based pipeline can identify common clinical trial outcomes of comparable breadth and quality as the outcomes identified in comprehensive literature reviews. Moreover, such an approach can highlight relevant outcomes for further consideration.

Patient-important outcomes in systematic reviews: Poor quality of evidence

Background

Cochrane reviewers are strongly encouraged to evaluate the quality of evidence for the most important outcomes by using the GRADE approach and to report these results in a Summary of Findings (SoF) table. We aimed to assess whether outcomes reported in the SoF table of Cochrane reviews could be considered patient-important outcomes (PIOs) and the quality of the available evidence for these outcomes.

Methods

We performed a methodological review of Cochrane reviews published between March 2011 and September 2014. For a random sample of Cochrane reviews reporting a SoF table, we extracted all outcomes reported in this table and evaluated whether they could be considered PIOs (i.e., mortality, other clinical events, adverse events, function, pain, quality of life and therapeutic decisions). Then, we collected the quality of evidence for every outcome in these SoF tables.

Results

We included 290 reviews issued by 47 of the 53 Cochrane Review Groups. Every SoF table included a median of 5 outcomes, for a total of 1414 outcomes; 1089 (77%) could be considered PIOs. Almost all reviews (n = 278, 96%) included at least one PIO in their SoF table. The quality of evidence for the outcomes was high for 12% (n = 168), moderate for 28% (n = 402) and low or very low for 45% (n = 640). Less than one quarter of reviews (n = 63) included at least one PIO with high-quality evidence that favoured a benefit of the experimental intervention evaluated in half of them (n = 34 reviews).

Conclusions

Many outcomes reported in the SoF table of recent Cochrane reviews can be considered PIOs. However, the quality of available evidence remains limited for these outcomes.

Vitamin D supplementation during pregnancy: state of the evidence from a systematic review of randomised trials

Objectives To estimate the effects of vitamin D supplementation during pregnancy on 11 maternal and 27 neonatal/infant outcomes; to determine frequencies at which trial outcome data were missing, unreported, or inconsistently reported; and to project the potential contributions of registered ongoing or planned trials.Design Systematic review and meta-analysis of randomised controlled trials; systematic review of registered but unpublished trials.Data sources Medline, Embase, PubMed, Cochrane Database of Systematic Reviews, and Cochrane Central Register of Controlled Trials from inception to September 2017; manual searches of reference lists of systematic reviews identified in the electronic search; and online trial registries for unpublished, ongoing, or planned trials.Eligibility criteria for study selection Trials of prenatal vitamin D supplementation with randomised allocation and control groups administered placebo, no vitamin D, or vitamin D ≤600 IU/day (or its equivalent), and published in a peer reviewed journal.Results 43 trials (8406 participants) were eligible for meta-analyses. Median sample size was 133 participants. Vitamin D increased maternal/cord serum concentration of 25-hydroxyvitamin D, but the dose-response effect was weak. Maternal clinical outcomes were rarely ascertained or reported, but available data did not provide evidence of benefits. Overall, vitamin D increased mean birth weight of 58.33 g (95% confidence interval 18.88 g to 97.78 g; 37 comparisons) and reduced the risk of small for gestational age births (risk ratio 0.60, 95% confidence interval 0.40 to 0.90; seven comparisons), but findings were not robust in sensitivity and subgroup analyses. There was no effect on preterm birth (1.0, 0.77 to 1.30; 15 comparisons). There was strong evidence that prenatal vitamin D reduced the risk of offspring wheeze by age 3 years (0.81, 0.67 to 0.98; two comparisons). For most outcomes, meta-analyses included data from a minority of trials. Only eight of 43 trials (19%) had an overall low risk of bias. Thirty five planned/ongoing randomised controlled trials could contribute 12 530 additional participants to future reviews.Conclusions Most trials on prenatal vitamin D published by September 2017 were small and of low quality. The evidence to date seems insufficient to guide clinical or policy recommendations. Future trials should be designed and powered to examine clinical endpoints, including maternal conditions related to pregnancy (such as pre-eclampsia), infant growth, and respiratory outcomes.Systematic review registration PROSPERO CRD42016051292.

Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS

BACKGROUND

The usefulness of clinical trials and systematic reviews is compromised when they report different outcomes. We compared outcomes in reviews of HIV/AIDS and the trials included in the reviews.

STUDY DESIGN AND SETTING

We examined all Cochrane reviews of HIV/AIDS (as of June 2013) that included ≥1 trial and the trials that the reviews included. We compared outcomes within subgroups defined by type of intervention: clinical management, biomedical prevention, behavioral prevention, and health services.

RESULTS

We included 84 reviews that encompassed 524 trials. Although the median number of outcomes per trial (8) and per review (7.5) was similar, the trials reported a considerably greater number of unique outcomes than the reviews (779 vs. 218), ranging from 2.3 times greater (clinical management) to 5.4 times greater (behavioral prevention). High proportions of trial outcomes were not in any review: 68% (clinical management) to 83% (behavioral prevention). Lower proportions of review outcomes were not in any trial: 11% (clinical management) to 39% (health services).

CONCLUSION

Outcomes in trials and reviews are not well aligned for appropriate inclusion of trial results in reviews and meta-analyses. Differences in perspectives, goals, and constraints between trialists and reviewers may explain differences in outcomes they consider important.

Surfactant instillation in spontaneously breathing preterm infants: a systematic review and meta-analysis

Less invasive surfactant therapies (LIST) use surfactant instillation through a thin tracheal catheter in spontaneously breathing infants. This review and meta-analysis investigates respiratory outcomes for preterm infants with respiratory distress syndrome treated with LIST rather than administration of surfactant through an endotracheal tube. Randomised controlled trial (RCT) full texts provided outcome data for bronchopulmonary dysplasia (BPD), death or BPD, early CPAP failure, invasive ventilation requirements and usual neonatal morbidities. Relative risks (RR) from pooled data, with subgroup analyses, were obtained from a Mantel-Haenszel analysis using a random effect model. Six RCTs evaluated LIST: 4 vs InSurE and 1 each vs delayed or immediate intubation for surfactant. LIST resulted in decreased risks of BPD (RR = 0.71 [0.52-0.99]; NNT = 21), death or BPD (RR = 0.74 [0.58-0.94]; NNT = 15) and early CPAP failure or invasive ventilation requirements (RR = 0.67 [0.53-0.84]; NNT = 8 and RR = 0.69 [0.53-0.88]; NNT = 6). Compared to InSurE, LIST decreased the risks of BPD or death (RR = 0.63 [0.44-0.92]; NNT = 11) and of early CPAP failure (RR = 0.71 [0.53-0.96]; NNT = 11). Common neonatal morbidities were not different.

CONCLUSIONS

Respiratory management with LIST decreases the risks of BPD and BPD or death, and the need for invasive ventilation. This strategy appears safe, but long-term follow-up is lacking. What is Known: • Initial management of preterm infants with CPAP decreases the risk of death or BPD, but many still require surfactant or invasive ventilation. • Surfactant can be instilled through a tracheal thin catheter while the infant breathes on CPAP, but improvement in BPD is inconsistent between studies. What is New: • Less invasive surfactant therapy (LIST) strategies decrease the risks of BPD, of death or BPD, and of CPAP failure compared to strategies where surfactant is administered through an endotracheal tube. • LIST strategies decrease the risks of the composite outcome of BPD or death and of early CPAP failure when compared to "intubation-surfactant-extubation" approaches.

The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses

POLICY POINTS

Currently, there is massive production of unnecessary, misleading, and conflicted systematic reviews and meta-analyses. Instead of promoting evidence-based medicine and health care, these instruments often serve mostly as easily produced publishable units or marketing tools. Suboptimal systematic reviews and meta-analyses can be harmful given the major prestige and influence these types of studies have acquired. The publication of systematic reviews and meta-analyses should be realigned to remove biases and vested interests and to integrate them better with the primary production of evidence.

CONTEXT

Currently, most systematic reviews and meta-analyses are done retrospectively with fragmented published information. This article aims to explore the growth of published systematic reviews and meta-analyses and to estimate how often they are redundant, misleading, or serving conflicted interests.

METHODS

Data included information from PubMed surveys and from empirical evaluations of meta-analyses.

FINDINGS

Publication of systematic reviews and meta-analyses has increased rapidly. In the period January 1, 1986, to December 4, 2015, PubMed tags 266,782 items as "systematic reviews" and 58,611 as "meta-analyses." Annual publications between 1991 and 2014 increased 2,728% for systematic reviews and 2,635% for meta-analyses versus only 153% for all PubMed-indexed items. Currently, probably more systematic reviews of trials than new randomized trials are published annually. Most topics addressed by meta-analyses of randomized trials have overlapping, redundant meta-analyses; same-topic meta-analyses may exceed 20 sometimes. Some fields produce massive numbers of meta-analyses; for example, 185 meta-analyses of antidepressants for depression were published between 2007 and 2014. These meta-analyses are often produced either by industry employees or by authors with industry ties and results are aligned with sponsor interests. China has rapidly become the most prolific producer of English-language, PubMed-indexed meta-analyses. The most massive presence of Chinese meta-analyses is on genetic associations (63% of global production in 2014), where almost all results are misleading since they combine fragmented information from mostly abandoned era of candidate genes. Furthermore, many contracting companies working on evidence synthesis receive industry contracts to produce meta-analyses, many of which probably remain unpublished. Many other meta-analyses have serious flaws. Of the remaining, most have weak or insufficient evidence to inform decision making. Few systematic reviews and meta-analyses are both non-misleading and useful.

CONCLUSIONS

The production of systematic reviews and meta-analyses has reached epidemic proportions. Possibly, the large majority of produced systematic reviews and meta-analyses are unnecessary, misleading, and/or conflicted.

Standardized Outcomes in Nephrology-Transplantation: A Global Initiative to Develop a Core Outcome Set for Trials in Kidney Transplantation

Background

Although advances in treatment have dramatically improved short-term graft survival and acute rejection in kidney transplant recipients, long-term graft outcomes have not substantially improved. Transplant recipients also have a considerably increased risk of cancer, cardiovascular disease, diabetes, and infection, which all contribute to appreciable morbidity and premature mortality. Many trials in kidney transplantation are short-term, frequently use unvalidated surrogate endpoints, outcomes of uncertain relevance to patients and clinicians, and do not consistently measure and report key outcomes like death, graft loss, graft function, and adverse effects of therapy. This diminishes the value of trials in supporting treatment decisions that require individual-level multiple tradeoffs between graft survival and the risk of side effects, adverse events, and mortality. The Standardized Outcomes in Nephrology-Transplantation initiative aims to develop a core outcome set for trials in kidney transplantation that is based on the shared priorities of all stakeholders.

Methods

This will include a systematic review to identify outcomes reported in randomized trials, a Delphi survey with an international multistakeholder panel (patients, caregivers, clinicians, researchers, policy makers, members from industry) to develop a consensus-based prioritized list of outcome domains and a consensus workshop to review and finalize the core outcome set for trials in kidney transplantation.

Conclusions

Developing and implementing a core outcome set to be reported, at a minimum, in all kidney transplantation trials will improve the transparency, quality, and relevance of research; to enable kidney transplant recipients and their clinicians to make better-informed treatment decisions for improved patient outcomes.

Covariate adjustments in randomized controlled trials increased study power and reduced biasedness of effect size estimation

OBJECTIVES

This study aims to show that under several assumptions, in randomized controlled trials (RCTs), unadjusted, crude analysis will underestimate the Cohen's d effect size of the treatment, and an unbiased estimate of effect size can be obtained only by adjusting for all predictors of the outcome.

STUDY DESIGN AND SETTING

Four simulations were performed to examine the effects of adjustment on the estimated effect size of the treatment and power of the analysis. In addition, we analyzed data from the Advanced Cognitive Training for Independent and Vital Elderly (ACTIVE) study (older adults aged 65-94), an RCT with three treatment arms and one control arm.

RESULTS

We showed that (1) the number of unadjusted covariates was associated with the effect size of the treatment; (2) the biasedness of effect size estimation was minimized if all covariates were adjusted for; (3) the power of the statistical analysis slightly decreased with the number of adjusted noise variables; and (4) exhaustively searching the covariates and noise variables adjusted for can lead to exaggeration of the true effect size. Analysis of the ACTIVE study data showed that the effect sizes adjusting for covariates of all three treatments were 7.39-24.70% larger than their unadjusted counterparts, whereas the effect size would be elevated by at most 57.92% by exhaustively searching the variables adjusted for.

CONCLUSION

All covariates of the outcome in RCTs should be adjusted for, and if the effect of a particular variable on the outcome is unknown, adjustment will do more good than harm.

A Core Outcome Set for Evaluation of Interventions to Prevent Preterm Birth

OBJECTIVE

To develop a consensus on a set of key clinical outcomes for the evaluation of preventive interventions for preterm birth in asymptomatic pregnant women.

METHODS

A two-stage web-based Delphi survey and a face-to-face meeting of key stakeholders were used to develop a consensus on a set of critical and important outcomes. We approached five stakeholder groups (parents, midwives, obstetricians, neonatologists, and researchers) from middle- and high-income countries. Outcomes subjected to the Delphi survey were identified by systematic literature review and stakeholder input. Survey participants scored each outcome on a 9-point Likert scale anchored between 1 (limited importance) and 9 (critical importance). They had the opportunity to reflect on total and stakeholder subgroup feedback between survey stages. For consensus, defined a priori, outcomes required at least 70% of participants of each stakeholder group to score them as "critical" and less than 15% as "limited."

RESULTS

A total of 228 participants from five stakeholder groups from three lower middle-income countries, seven upper middle-income countries, and 17 high-income countries were asked to score 31 outcomes. Of these participants, 195 completed the first survey and 174 the second. Consensus was reached on 13 core outcomes: four were related to pregnant women: maternal mortality, maternal infection or inflammation, prelabor rupture of membranes, and harm to mother from intervention. Nine were related to offspring: gestational age at birth, offspring mortality, birth weight, early neurodevelopmental morbidity, late neurodevelopmental morbidity, gastrointestinal morbidity, infection, respiratory morbidity, and harm to offspring from intervention.

CONCLUSION

This core outcome set for studies that evaluate prevention of preterm birth developed with an international multidisciplinary perspective will ensure that data from trials that assess prevention of preterm birth can be compared and combined.

DATABASE REGISTRATION

COMET Initiative, http://www.comet-initiative.org/studies/details/603, REGISTRATION NUMBER: 603.

Social network analysis identified central outcomes for core outcome sets using systematic reviews of HIV/AIDS

OBJECTIVES

Methods to develop core outcome sets, the minimum outcomes that should be measured in research in a topic area, vary. We applied social network analysis methods to understand outcome co-occurrence patterns in human immunodeficiency virus (HIV)/AIDS systematic reviews and identify outcomes central to the network of outcomes in HIV/AIDS.

STUDY DESIGN AND SETTING

We examined all Cochrane reviews of HIV/AIDS as of June 2013. We defined a tie as two outcomes (nodes) co-occurring in ≥2 reviews. To identify central outcomes, we used normalized node betweenness centrality (nNBC) (the extent to which connections between other outcomes in a network rely on that outcome as an intermediary). We conducted a subgroup analysis by HIV/AIDS intervention type (i.e., clinical management, biomedical prevention, behavioral prevention, and health services).

RESULTS

The 140 included reviews examined 1,140 outcomes, 294 of which were unique. The most central outcome overall was all-cause mortality (nNBC = 23.9). The most central and most frequent outcomes differed overall and within subgroups. For example, "adverse events (specified)" was among the most central but not among the most frequent outcomes, overall.

CONCLUSION

Social network analysis methods are a novel application to identify central outcomes, which provides additional information potentially useful for developing core outcome sets.

Are current standards of reporting quality for clinical trials sufficient in addressing important sources of bias?

Determining the quality of a randomized clinical trial (RCT) is necessary for decision-makers to determine the believability and applicability of the trial findings. Issues that are likely to affect the utility of RCT evidence include issues of bias, random error and applicability. In this article we focus primarily on issues of bias and examine the evidence for whether reporting methodological items, including allocation concealment, sequence generation, and blinding of participants can be relied upon as evidence of bias. We present the findings of a systematic review of meta-epidemiological studies and a simulation study demonstrating that commonly examined sources of bias likely play little role in treatment exaggeration. We discuss other issues that may additionally influence trial outcomes including sample size, publication bias, and expertise of trialists. We conclude by discussing strategies to moderate the effect of known biases in assessing overall estimates of treatment effects.