Case studies to explore the optimal use of randomized and nonrandomized studies in evidence syntheses that use GRADE

Including non-randomized studies of interventions in meta-analyses of randomized controlled trials changed the estimates in more than a third of the studies: evidence from an empirical analysis

OBJECTIVES

There is a growing trend to include nonrandomized studies of interventions (NRSIs) in meta-analyses of randomized controlled trials (RCTs) for health decision-making. The study aimed to quantify the impact of integrating NRSI on the evidence derived from RCTs within the same systematic review.

STUDY DESIGN AND SETTING

We searched PubMed for systematic reviews published between December 9, 2017, and December 9, 2022, that included both RCTs and NRSIs under the same outcome. Using the DerSimonian-Laird random-effects model, we reanalyzed the pooled estimates to compare those derived from RCTs with those from combined RCTs and NRSIs. We examined changes in point estimates, subgroup differences, statistical heterogeneity, and the weight of RCTs in pooled estimates. Results were defined as being in qualitative agreement if both estimates demonstrated statistical significance in the same direction or if neither achieved statistical significance.

RESULTS

A total of 220 eligible systematic reviews were identified and 217 meta-analyses were reanalyzed. Qualitative disagreement between RCTs only and pooled estimates combining RCTs and NRSIs was observed in 78 meta-analyses (35.9%), of which 69 (88.5%) gained statistical significance after the inclusion of NRSIs. Point estimates in 58 meta-analyses (26.7%) failed to meet predefined agreement criteria, and statistically significant subgroup differences between RCTs and NRSIs were identified in 32 meta-analyses (14.8%). The incorporation of NRSIs raised the heterogeneity from 21.8% to 36.9%, whereas RCTs accounted for a median weight of 33.9% in the pooled estimates.

CONCLUSION

These findings highlight the need for caution in conducting and interpreting meta-analyses combining RCTs and NRSIs, particularly in scenarios where RCTs yield nonsignificant results whereas the inclusion of NRSIs achieves statistical significance.

PLAIN LANGUAGE SUMMARY

Although randomized controlled trials (RCTs) remain the gold standard for clinical evidence, they are often insufficient to address complex clinical questions. Nonrandomized studies of interventions (NRSIs), leveraging real-world clinical data, are increasingly used to supplement RCT findings. Despite growing interest in integrating NRSIs into meta-analyses with RCTs, the clinical and statistical implications of this approach remain uncertain. To address this gap, we conducted a systematic evaluation of how NRSI inclusion impacts meta-analytic results by analyzing 220 systematic reviews that combined RCTs and NRSIs under the same outcome. Our analysis revealed that incorporating NRSIs altered effect estimates in over one-third of cases, with 88.5% of meta-analyses achieving statistical significance only after NRSI inclusion-a finding with critical implications for decision-making. In addition, NRSI integration elevated statistical heterogeneity, although RCTs accounted for less than one-third of the weight in pooled estimates. These findings collectively underscore the necessity for robust evaluation and cautious interpretation when merging NRSI data with RCTs in meta-analyses.

Integrating randomized controlled trials and non-randomized studies of interventions to assess the effect of rare events: a Bayesian re-analysis of two meta-analyses

BACKGROUND

There is a growing trend to include non-randomised studies of interventions (NRSIs) in rare events meta-analyses of randomised controlled trials (RCTs) to complement the evidence from the latter. An important consideration when combining RCTs and NRSIs is how to address potential bias and down-weighting of NRSIs in the pooled estimates. The aim of this study is to explore the use of a power prior approach in a Bayesian framework for integrating RCTs and NRSIs to assess the effect of rare events.

METHODS

We proposed a method of specifying the down-weighting factor based on judgments of the relative magnitude (no information, and low, moderate, serious and critical risk of bias) of the overall risk of bias for each NRSI using the ROBINS-I tool. The methods were illustrated using two meta-analyses, with particular interest in the risk of diabetic ketoacidosis (DKA) in patients using sodium/glucose cotransporter-2 (SGLT-2) inhibitors compared with active comparators, and the association between low-dose methotrexate exposure and melanoma.

RESULTS

No significant results were observed for these two analyses when the data from RCTs only were pooled (risk of DKA: OR = 0.82, 95% confidence interval (CI): 0.25-2.69; risk of melanoma: OR = 1.94, 95%CI: 0.72-5.27). When RCTs and NRSIs were directly combined without distinction in the same meta-analysis, both meta-analyses showed significant results (risk of DKA: OR = 1.50, 95%CI: 1.11-2.03; risk of melanoma: OR = 1.16, 95%CI: 1.08-1.24). Using Bayesian analysis to account for NRSI bias, there was a 90% probability of an increased risk of DKA in users receiving SGLT-2 inhibitors and an 91% probability of an increased risk of melanoma in patients using low-dose methotrexate.

CONCLUSIONS

Our study showed that including NRSIs in a meta-analysis of RCTs for rare events could increase the certainty and comprehensiveness of the evidence. The estimates obtained from NRSIs are generally considered to be biased, and the possible influence of NRSIs on the certainty of the combined evidence needs to be carefully investigated.

A Bayesian bias-adjusted random-effects model for synthesizing evidence from randomized controlled trials and nonrandomized studies of interventions

OBJECTIVE

An important consideration when combining RCTs and NRSIs is how to address their potential biases in the pooled estimates. This study aimed to propose a Bayesian bias-adjusted random effects model for the synthesis of evidence from RCTs and NRSIs.

METHODS

We present a Bayesian bias-adjusted random effects model based on power prior method, which combines the likelihood contribution of the NRSIs, raised to the power parameter of alpha, with the likelihood of the RCT data, modeled with an additive bias. The method was illustrated using a meta-analysis on the association between low-dose methotrexate exposure and melanoma. We also combined RCTs and NRSIs using the naïve data synthesis.

RESULTS

The results including only RCTs has a posterior median and 95% credible interval (CrI) of 1.18 (0.31-4.04), the posterior probability of any harm (> 1.0) and a meaningful association (> 1.15) were 0.61 and 0.52, respectively. The posterior median and 95% CrI based on the naïve data synthesis resulted in 1.17 (0.96-1.47), and the posterior probability of any harm and a meaningful association were 0.96 and 0.60, respectively. For the Bayesian bias-adjusted analysis, the median OR was 1.16 (95% CrI: 0.83-1.71), and the posterior probabilities of any and a meaningful clinical association were 0.88 and 0.53, respectively.

CONCLUSIONS

The results indicated that integrating NRSIs into meta-analysis could increase the certainty of the body of evidence. However, directly combining RCTs and NRSIs in the same meta-analysis without distinction may lead to misleading conclusions.

Evaluating agreement between evidence from randomised controlled trials and cohort studies in nutrition: a meta-research replication study

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study. We searched Medline, Epistemonikos and the Cochrane Database of Systematic Reviews for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies. Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements. We included 82 BoE-pairs from 51 systematic reviews. For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99 to 1.10, I2 = 59%, τ2 = 0.02, prediction interval (PI) 0.77 to 1.41). For continuous outcomes, the pooled DSMD was  - 0.09 (95% CI  - 0.26 to 0.09, PI  - 0.55 to 0.38). Subgroup analyses yielded that differences in type of intake/exposure were drivers towards disagreement. We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.Trial registration: CRD42021278908.

To feed or not to feed during therapeutic hypothermia in asphyxiated neonates: a systematic review and meta-analysis

The practice of withholding feed during therapeutic hypothermia (TH) in neonates with hypoxemic ischemic encephalopathy (HIE) is based on conventions rather than evidence. Recent studies suggest that enteral feeding might be safe during TH. We systematically compared the benefits and harms of enteral feeding in infants undergoing TH for HIE. We searched electronic databases and trial registries (MEDLINE, CINAHL, Embase, Web of Science, and CENTRAL) until December 15, 2022, for studies comparing enteral feeding and non-feeding strategies. We performed a random-effects meta-analysis using RevMan 5.4 software. The primary outcome was the incidence of stage II/III necrotizing enterocolitis (NEC). Other outcomes included the incidence of any stage NEC, mortality, sepsis, feed intolerance, time to full enteral feeds, and hospital stay. Six studies ((two randomized controlled trials (RCTs) and four nonrandomized studies of intervention (NRSIs)) enrolling 3693 participants were included. The overall incidence of stage II/III NEC was very low (0.6%). There was no significant difference in the incidence of stage II/III NEC in RCTs (2 trials, 192 participants; RR, 1.20; 95% CI: 0.53 to 2.71, I², 0%) and NRSIs (3 studies, no events in either group). In the NRSIs, infants in the enteral feeding group had significantly lower sepsis rates (four studies, 3500 participants, RR, 0.59; 95% CI: 0.51 to 0.67, I²-0%) and lower all-cause mortality (three studies, 3465 participants, RR: 0.43; 95% CI: 0.33 to 0.57, I²-0%) than the infants in the "no feeding" group. However, no significant difference in mortality was observed in RCTs (RR: 0.70; 95% CI: 0.28 to 1.74, I²-0%). Infants in the enteral feeding group achieved full enteral feeding earlier, had higher breastfeeding rates at discharge, received parenteral nutrition for a shorter duration, and had shorter hospital stays than the control group.  Conclusion: In late preterm and term infants with HIE, enteral feeding appears safe and feasible during the cooling phase of TH. However, there is insufficient evidence to guide the timing of initiation, volume, and feed advancement. What is Known: • Many neonatal units withhold enteral feeding during therapeutic hypothermia, fearing an increased risk of complications (feed intolerance and necrotizing enterocolitis). • The overall risk of necrotizing enterocolitis in late-preterm and term infants is extremely low (< 1%). What is New: • Enteral feeding during therapeutic hypothermia is safe and does not increase the risk of necrotizing enterocolitis, hypoglycemia, or feed intolerance. It may reduce the incidence of sepsis and all-cause mortality until discharge.