Evaluating the impact of including non-randomised studies of interventions in meta-analysis of randomised controlled trials: a protocol for a meta-epidemiological study

Including non-randomized studies of interventions in meta-analyses of randomized controlled trials changed the estimates in more than a third of the studies: evidence from an empirical analysis

OBJECTIVES

There is a growing trend to include nonrandomized studies of interventions (NRSIs) in meta-analyses of randomized controlled trials (RCTs) for health decision-making. The study aimed to quantify the impact of integrating NRSI on the evidence derived from RCTs within the same systematic review.

STUDY DESIGN AND SETTING

We searched PubMed for systematic reviews published between December 9, 2017, and December 9, 2022, that included both RCTs and NRSIs under the same outcome. Using the DerSimonian-Laird random-effects model, we reanalyzed the pooled estimates to compare those derived from RCTs with those from combined RCTs and NRSIs. We examined changes in point estimates, subgroup differences, statistical heterogeneity, and the weight of RCTs in pooled estimates. Results were defined as being in qualitative agreement if both estimates demonstrated statistical significance in the same direction or if neither achieved statistical significance.

RESULTS

A total of 220 eligible systematic reviews were identified and 217 meta-analyses were reanalyzed. Qualitative disagreement between RCTs only and pooled estimates combining RCTs and NRSIs was observed in 78 meta-analyses (35.9%), of which 69 (88.5%) gained statistical significance after the inclusion of NRSIs. Point estimates in 58 meta-analyses (26.7%) failed to meet predefined agreement criteria, and statistically significant subgroup differences between RCTs and NRSIs were identified in 32 meta-analyses (14.8%). The incorporation of NRSIs raised the heterogeneity from 21.8% to 36.9%, whereas RCTs accounted for a median weight of 33.9% in the pooled estimates.

CONCLUSION

These findings highlight the need for caution in conducting and interpreting meta-analyses combining RCTs and NRSIs, particularly in scenarios where RCTs yield nonsignificant results whereas the inclusion of NRSIs achieves statistical significance.

PLAIN LANGUAGE SUMMARY

Although randomized controlled trials (RCTs) remain the gold standard for clinical evidence, they are often insufficient to address complex clinical questions. Nonrandomized studies of interventions (NRSIs), leveraging real-world clinical data, are increasingly used to supplement RCT findings. Despite growing interest in integrating NRSIs into meta-analyses with RCTs, the clinical and statistical implications of this approach remain uncertain. To address this gap, we conducted a systematic evaluation of how NRSI inclusion impacts meta-analytic results by analyzing 220 systematic reviews that combined RCTs and NRSIs under the same outcome. Our analysis revealed that incorporating NRSIs altered effect estimates in over one-third of cases, with 88.5% of meta-analyses achieving statistical significance only after NRSI inclusion-a finding with critical implications for decision-making. In addition, NRSI integration elevated statistical heterogeneity, although RCTs accounted for less than one-third of the weight in pooled estimates. These findings collectively underscore the necessity for robust evaluation and cautious interpretation when merging NRSI data with RCTs in meta-analyses.

The Effects of Exercise Training on Body Composition and Cardiometabolic Risk Factors in Type 1 Diabetes Mellitus: A Systematic Review and Meta-Analysis

INTRODUCTION AND AIM

We performed a systematic review and meta-analysis to investigate the effects of exercise training on body composition and cardiometabolic health in patients with Type 1 diabetes (T1D).

METHOD

A search in three main databases including PubMed, Web of Science, and Scopus was conducted from the inception of this review until June 2024 to identify randomized control trials investigating the effects of exercise training compared to a control on body composition and cardiometabolic risk factors in patients with T1D. The data were pooled using random effects models to calculate weighted mean differences (WMDs), standardized mean differences (SMDs), and 95% confidence intervals (CIs).

RESULTS

Overall, 25 studies involving 1120 patients with T1D were included in the meta-analysis. Exercise training decreased body mass index (BMI) [WMD: -0.18 kg.m2, p = 0.02], fasting glucose [WMD: -14.97 mg/dl, p = 0.01], and HbA1c [WMD: -0.49%, p = 0.003], and increased VO2max/peak [WMD: 2.76 mL/kg/min, p = 0.001] as compared with controls. Exercise training had no effect on body fat percentage or lean body mass, lipid profiles, or blood pressure. Subgroup analysis indicated that age, exercise mode, and intervention duration were the main moderators for the beneficial effects of exercise training.

CONCLUSIONS

In patients with T1D, exercise training is effective for decreasing body weight and cardiometabolic risk factors.

Integration of non-randomized studies with randomized controlled trials in meta-analyses of clinical studies: a meta-epidemiological study on effect estimation of interventions

BACKGROUNDS

Syntheses of non-randomized studies of interventions (NRSIs) and randomized controlled trials (RCTs) are increasingly used in decision-making. This study aimed to summarize when NRSIs are included in evidence syntheses of RCTs, with a particular focus on the methodological issues associated with combining NRSIs and RCTs.

METHODS

We searched PubMed to identify clinical systematic reviews published between 9 December 2017 and 9 December 2022, randomly sampling reviews in a 1:1 ratio of Core and non-Core clinical journals. We included systematic reviews with RCTs and NRSIs for the same clinical question. Clinical scenarios for considering the inclusion of NRSIs in eligible studies were classified. We extracted the methodological characteristics of the included studies, assessed the concordance of estimates between RCTs and NRSIs, calculated the ratio of the relative effect estimate from NRSIs to that from RCTs, and evaluated the impact on the estimates of pooled estimates when NRSIs are included.

RESULTS

Two hundred twenty systematic reviews were included in the analysis. The clinical scenarios for including NRSIs were grouped into four main justifications: adverse outcomes (n = 140, 63.6%), long-term outcomes (n = 36, 16.4%), the applicability of RCT results to broader populations (n = 11, 5.0%), and other (n = 33, 15.0%). When conducting a meta-analysis, none of these reviews assessed the compatibility of the different types of evidence prior, 203 (92.3%) combined estimates from RCTs and NRSIs in the same meta-analysis. Of the 203 studies, 169 (76.8%) used crude estimates of NRSIs, and 28 (13.8%) combined RCTs and multiple types of NRSIs. Seventy-seven studies (35.5%) showed "qualitative disagree" between estimates from RCTs and NRSIs, and 101 studies (46.5%) found "important difference". The integration of NRSIs changed the qualitative direction of estimates from RCTs in 72 out of 200 studies (36.0%).

CONCLUSIONS

Systematic reviews typically include NRSIs in the context of assessing adverse or long-term outcomes. The inclusion of NRSIs in a meta-analysis of RCTs has a substantial impact on effect estimates, but discrepancies between RCTs and NRSIs are often ignored. Our proposed recommendations will help researchers to consider carefully when and how to synthesis evidence from RCTs and NRSIs.

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Phenobarbital treatment of alcohol withdrawal in the emergency department: A systematic review and meta-analysis

OBJECTIVE

Despite frequent treatment of alcohol withdrawal syndrome (AWS) in the emergency department (ED), evidence for phenobarbital (PB) as an ED alternative therapy is mixed. We conducted a systematic review and meta-analysis comparing safety and efficacy of PB to benzodiazepines (BZDs) for treatment of AWS in the ED.

METHODS

We searched articles and references published in English in PubMed, Web of Science, and Embase from inception through May 2022. We included randomized trials and cohort studies comparing treatment with PB to BZD controls and excluded studies focused on non-AWS conditions. Review was conducted by two blinded investigators and a third author; eight of 59 (13.6%) abstracts met inclusion criteria for review and meta-analysis using a random-effects model. Treatment superiority was evaluated through utilization, pharmacologic, and clinical outcomes. Primary outcomes for meta-analysis were the proportion of patients (1) admitted to the intensive care unit (ICU), (2) admitted to the hospital, (3) readmitted to the ED after discharge, and (4) who experienced adverse events.

RESULTS

Eight studies (two randomized controlled trials, six retrospective cohorts) comprised data from 1507 patients in 2012 treatment encounters for AWS. All studies were included in meta-analysis for adverse events, seven for hospital admission, five for ICU admission, and three for readmission to the ED after discharge. Overall methodological quality was low-moderate, risk of bias moderate-high, and statistical heterogeneity moderate. Pooled relative risk of ICU admission for those treated with PB versus BZD was 0.92 (95% confidence interval [CI] 0.54-1.55). Risk for admission to the hospital was 0.98 (95% CI 0.89-1.07) and for any adverse event was 1.1 (95% CI 0.78-1.57); heterogeneity prevented meta-analysis for ED readmission.

CONCLUSIONS

The current literature base does not show that treatment with PB significantly reduces ICU admissions, hospital admissions, ED readmissions, or adverse events in ED patients with AWS compared with BZDs alone.