Meta-analysis of randomized trials: looking back and looking ahead

Evidence-Based Decision-Making 2: Systematic Reviews and Meta-Analysis

The number of studies published in the biomedical literature has dramatically increased over the last few decades. This massive proliferation of literature makes clinical medicine increasingly complex, and information from multiple studies is often needed to inform a particular clinical decision. However, available studies often vary in their design, methodological quality, and population studied, and may define the research question of interest quite differently. This can make it challenging to synthesize the conclusions of multiple studies. In addition, since even highly cited trials may be challenged over time, clinical decision-making requires ongoing reconciliation of studies which provide different answers to the same question. Because it is often impractical for readers to track down and review all the primary studies, systematic reviews and meta-analyses are an important source of evidence on the diagnosis, prognosis and treatment of any given disease. This chapter summarizes methods for conducting and reading systematic reviews and meta-analyses, as well as describes potential advantages and disadvantages of these publications.

Appraising the Quality of Systematic Reviews for Age-Related Macular Degeneration Interventions: A Systematic Review

Importance

Age-related macular degeneration (AMD) is a leading cause of vision impairment. It is imperative that AMD care is timely, appropriate, and evidence-based. It is thus essential that AMD systematic reviews are robust; however, little is known about the quality of this literature.

Objectives

To investigate the methodological quality of systematic reviews of AMD intervention studies, and to evaluate their use for guiding evidence-based care.

Evidence Review

This systematic review adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. All studies that self-identified as a systematic review in their title or abstract or were categorized as a systematic review from a medical subject heading and investigated the safety, efficacy and/or effectiveness of an AMD intervention were included. Comprehensive electronic searches were performed in Ovid MEDLINE, Embase, and the Cochrane Library from inception to March 2017. Two reviewers independently assessed titles and abstracts, then full-texts for eligibility. Quality was assessed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool. Study characteristics (publication year, type of intervention, journal, citation rate, and funding source) were extracted.

Findings

Of 983 citations retrieved, 71 studies (7.6%) were deemed eligible. The first systematic review relating to an AMD intervention was published in 2003. More than half were published since 2014. Methodological quality was highly variable. The mean (SD) AMSTAR score was 5.8 (3.2) of 11.0, with no significant improvement over time (r = -0.03; 95% CI, -0.26 to 0.21; P = .83). Cochrane systematic reviews were overall of higher quality than reviews in other journals (mean [SD] AMSTAR score, 9.9 [1.2], n = 15 vs 4.7 [2.2], n = 56; P < .001). Overall, there was poor adherence to referring to an a priori design (22 articles [31%]) and reporting conflicts of interest in both the review and included studies (16 articles [23%]). Reviews funded by government grants and/or institutions were generally of higher quality than industry-sponsored reviews or where the funding source was not reported.

Conclusions and Relevance

There are gaps in the conduct of systematic reviews in the field of AMD. Enhanced endorsement of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement by refereed journals may improve review quality and improve the dissemination of reliable evidence relating to AMD interventions to clinicians.

Are child-centric aspects in newborn and child health systematic review and meta-analysis protocols and reports adequately reported?-two systematic reviews

BACKGROUND

Evidence suggests that newborn and child health systematic reviews and meta-analyses exhibit poor quality in reporting. The "Preferred Reporting Items in Systematic Review and Meta-Analysis" (PRISMA) and PRISMA-Protocols (PRISMA-P) checklists have been developed to improve the reporting of systematic review results and protocols, respectively. We aimed to evaluate the clarity and transparency in reporting of child-centric items in child health systematic reviews (SRs) and SR protocols and to identify areas where reporting could be strengthened.

METHODS

Two preliminary lists of potential child-centric reporting items were used to examine current reporting. The Cochrane, DARE, MEDLINE, and EMBASE libraries were searched from 2010 to 2014 for systematic reviews that included children. Each report and protocol that met the inclusion criteria had their quality of reporting assessed by their reporting of child-centric items. Quality of reporting was assessed per whether one third, one to two thirds, or more than two thirds of papers complied with potential child-centric potential modifications/extensions to PRISMA and were analyzed by the following: (i) paper type (i.e., report vs. protocol), (ii) publication type (i.e., Cochrane vs. non-Cochrane), and (iii) population type (i.e., child-only vs. mixed populations vs. family/maternal).

RESULTS

Of the 414 eligible articles, 248 reports and 76 protocols were included. In 21 of 24 potential SR reporting items and 13 of 14 potential SR protocol reporting items, less than two thirds of papers met the child-centric reporting item requirements. Mixed population studies displayed significantly poorer reporting in comparison to child-only and family/maternal intervention studies for 11 potential SR reporting items (p < 0.05) and five potential SR protocol items (p < 0.05). When comparing non-Cochrane to Cochrane reports and protocols, five items in both lists were found to perform significantly poorer in non-Cochrane reports (p < 0.05). Significant differences in reporting quality were found in three of 14 items shared between the potential SR reporting items and potential SR protocol reporting items (p < 0.05).

CONCLUSIONS

Newborn and child health systematic reviews and meta-analyses exhibit incomplete reporting, thereby hindering prudent decision-making by healthcare providers and policy makers. These results provide a rationale for the implementation of child-centric extensions and modifications to current PRISMA and PRISMA-P, such as to improve reporting in this population.

Meta-Analyses of Human Cell-Based Cardiac Regeneration Therapies: What Can Systematic Reviews Tell Us About Cell Therapies for Ischemic Heart Disease?

Controversies from basic science, discrepancies from clinical trials, and divergent results from meta-analyses have recently arisen in the field of cell therapies for cardiovascular repair and regeneration. Noticeably, there are almost as many systematic reviews and meta-analyses published as there are well-conducted clinical studies. But how do we disentangle the confusion they have raised? This article addresses why results obtained from systematic reviews and meta-analyses of human cell-based cardiac regeneration therapies are still valid to inform the design of future clinical trials. It also addresses how meta-analyses are not free from limitations and how important it is to assess the quality of the evidence and the quality of the systematic reviews and finally how stronger conclusions can be drawn when several pieces of evidence converge.

Performance of informative priors skeptical of large treatment effects in clinical trials: A simulation study

One of the main advantages of Bayesian analyses of clinical trials is their ability to formally incorporate skepticism about large treatment effects through the use of informative priors. We conducted a simulation study to assess the performance of informative normal, Student- t, and beta distributions in estimating relative risk (RR) or odds ratio (OR) for binary outcomes. Simulation scenarios varied the prior standard deviation (SD; level of skepticism of large treatment effects), outcome rate in the control group, true treatment effect, and sample size. We compared the priors with regards to bias, mean squared error (MSE), and coverage of 95% credible intervals. Simulation results show that the prior SD influenced the posterior to a greater degree than the particular distributional form of the prior. For RR, priors with a 95% interval of 0.50-2.0 performed well in terms of bias, MSE, and coverage under most scenarios. For OR, priors with a wider 95% interval of 0.23-4.35 had good performance. We recommend the use of informative priors that exclude implausibly large treatment effects in analyses of clinical trials, particularly for major outcomes such as mortality.

Evidence-based decision-making 2: Systematic reviews and meta-analysis

The number of studies published in the biomedical literature has dramatically increased over the last few decades. This massive proliferation of literature makes clinical medicine increasingly complex, and information from multiple studies is often needed to inform a particular clinical decision. However, available studies often vary in their design, methodological quality, populations studied and may define the research question of interest quite differently, which can make it challenging to synthesize their conclusions. In addition, since even highly cited trials may be challenged over time, clinical decision-making requires ongoing reconciliation of studies which provide different answers to the same question. Because it is often impractical for readers to track down and review all the primary studies, systematic reviews and meta-analyses are an important source of evidence on the diagnosis, prognosis, and treatment of any given disease. This chapter summarizes methods for conducting and reading systematic reviews and meta-analyses, as well as describing potential advantages and disadvantages of these publications.

The surgical treatment for portal hypertension: a systematic review and meta-analysis

Aim. To compare the effectiveness of surgical procedures (selective or nonselective shunt, devascularization, and combined shunt and devascularization) in preventing recurrent variceal bleeding and other complications in patients with portal hypertension. Methods. A systematic literature search of the Medline and Cochrane Library databases was carried out, and a meta-analysis was conducted according to the guidelines of the Quality of Reporting Meta-Analyses (QUOROM) statement. Results. There were a significantly higher reduction in rebleeding, yet a significantly more common encephalopathy (P = 0.05) in patients who underwent the shunt procedure compared with patients who had only a devascularization procedure. Further, there were no significant differences in rebleeding, late mortality, and encephalopathy between selective versus non-selective shunt. Next, the decrease of portal vein pressure, portal vein diameter, and free portal pressure in patients who underwent combined treatment with shunt and devascularization was more pronounced compared with patients who were treated with devascularization alone (P < 0.05). Conclusions. This meta-analysis shows clinical advantages of combined shunt and devascularization over devascularization in the prevention of recurrent variceal bleeding and other complications in patients with portal hypertension.

Pouch Roux-en-Y vs No Pouch Roux-en-Y following total gastrectomy: a meta-analysis based on 12 studies

After a total resection of the stomach, the continuity of the gastrointestinal tract can be restored either by Roux-en-Y esophagojejunostomy with or without a pouch. There is still no consensus on the best reconstruction technique. The aim of this report was to derive a more precise estimation of Roux-en-Y esophagojejunostomy with a pouch compared with Roux-en-Y esophagojejunostomy without a pouch. Studies were identified by PubMed and Embase searches, and the inclusion criteria were randomized controlled trials (RCTs) comparing reconstruction techniques between Roux-en-Y with and without a pouch. A total of 12 studies including 1,018 patients were included. The meta-analysis shows that pouch Roux-en-Y does not significantly increase total postoperative complications, anastomotic leakage or mortality. Importantly, there is no significant difference in 5-year survival rates between the two groups. Patients with Roux-en-Y esophagojejunostomy complained significantly less of reflux symptoms and dumping syndrome, and had significantly less severe reflux esophagitis. Quality of life was significantly improved in patients with Roux-en-Y esophagojejunostomy with a pouch compared with patients who received Roux-en-Y reconstruction without a pouch. The results indicate the need for Roux-en-Y esophagojejunostomy with a pouch is a gastric substitute after total gastrectomy by comparison with Roux-en-Y esophagojejunostomy without a pouch.

Debunking Myths and Urban Legends About Meta-Analysis

Meta-analysis is the dominant approach to research synthesis in the organizational sciences. We discuss seven meta-analytic practices, misconceptions, claims, and assumptions that have reached the status of myths and urban legends (MULs). These seven MULs include issues related to data collection (e.g., consequences of choices made in the process of gathering primary-level studies to be included in a meta-analysis), data analysis (e.g., effects of meta-analytic choices and technical refinements on substantive conclusions and recommendations for practice), and the interpretation of results (e.g., meta-analytic inferences about causal relationships). We provide a critical analysis of each of these seven MULs, including a discussion of why each merits being classified as an MUL, their kernels of truth value, and what part of each MUL represents misunderstanding. As a consequence of discussing each of these seven MULs, we offer best-practice recommendations regarding how to conduct meta-analytic reviews.

Pouch vs. no pouch following total gastrectomy: meta-analysis and systematic review

OBJECTIVES

Whether reconstruction after total gastrectomy for gastric malignancies should be done with or without a pouch is a controversial issue in clinical research. There is still no consensus on the reconstruction technique of choice. The aim of this report was to assess the value of pouch formation as a gastric substitute after total gastrectomy compared with reconstruction techniques without a pouch.

METHODS

A systematic literature search of the Medline database and the Cochrane Library was carried out and a meta-analysis executed according to the Quality of Reporting Meta-Analyses (QUOROM) statement. Only randomized controlled trials (RCTs) comparing reconstruction techniques with and without a pouch were eligible for inclusion. All trials were independently assessed by two authors. Data on perioperative parameters, postgastrectomy symptoms, eating capability, body weight, and quality of life were extracted from the RCTs for meta-analysis using random-effects models for the calculation of pooled estimates of treatment effects.

RESULTS

Nine RCTs comparing Roux-en-Y reconstructions with and without pouch and four RCTs comparing jejunal interpositions with and without a pouch were included. The results of the meta-analyses show that additional pouch formation does not significantly increase morbidity or mortality and does not considerably extend the operating time or the hospital stay. Patients with a pouch complained significantly less of dumping and heartburn and showed a significantly better food intake postoperatively. Quality of life was significantly improved in patients with a pouch compared with patients without a pouch. This difference even increased over time from 6 to 12 and 24 months postoperatively.

CONCLUSIONS

This meta-analysis highlights some clinical advantages of pouch reconstruction after total gastrectomy.

Primer on systematic review and meta-analysis

A systematic review uses an explicitly defined process to comprehensively identify all studies pertaining to a specific focused question, appraise the methods of the studies, summarize the results, identify reasons for different findings across studies, and cite limitations of current knowledge. Meta-analyses usually combine aggregate-level data reported in each primary study, which may provide a more precise estimate of the "true effect" than any individual study. However, the conclusions may be limited by between-trial heterogeneity, publication bias, or deficits in the conduct or reporting of individual primary studies.

Dietary magnesium intake and risk of cardiovascular disease among women

This study assessed the hypothesis that greater magnesium intake is associated with reduced risk for cardiovascular disease (CVD), including myocardial infarction (MI) and stroke, in a large prospective cohort of women. In 1993, a semi-quantitative food frequency questionnaire was used to assess magnesium intake in 39,876 female health professionals aged 39 to 89 years who had no history of CVD or cancer. During a median of 10 years of follow-up, 1,037 incident cases of CVD were identified, including 280 nonfatal MIs and 368 strokes. After adjustment for age and randomized treatment status, magnesium intake was not significantly associated with risk for incident CVD. Comparing the highest quintile of magnesium intake (median 433 mg/day) with the lowest quintile (median 255 mg/day), the relative risks were 0.87 (95% confidence interval [CI] 0.72 to 1.05, p for trend = 0.24) for total CVD, 0.88 (95% CI 0.70 to 1.12, p for trend = 0.34) for coronary heart disease (CHD), 1.03 (95% CI 0.72 to 1.49, p for trend = 0.96) for nonfatal MI, 1.11 (95% CI 0.61 to 2.00, p for trend = 0.95) for CVD death, and 0.87 (95% CI 0.64 to 1.18, p for trend = 0.55) for total stroke. Additional adjustment for other CVD risk factors did not materially change the observed null associations. In conclusion, the results do not support the hypothesis that magnesium intake reduces the development of CHD, although a modest inverse association with stroke cannot be ruled out.

Beyond classical meta-analysis: can inadequately reported studies be included?

Classical meta-analysis requires the same data from each clinical trial, thus data-reporting must be of a high-quality. Imputation methods are used to include studies that provide incomplete information on variability and the fixed and random effects of a drug. Regression models can be used to include studies other than randomized placebo-controlled studies. In the example outlined here, the use of non-randomized single-arm studies and studies against comparator treatments has little influence on the estimation of the treatment effect in comparison with placebo, an effect that is based on the randomized placebo-controlled studies. The inclusion of other studies serves to increase the precision of the effect of the treatment compared with baseline. Although multiple imputation techniques enable a larger number of studies to be included, which will typically increase the precision of the estimated effect, a careful sensitivity analysis is also required.

Assessing heterogeneity in meta-analyses of Helicobacter pylori infection-related clinical studies: a critical appraisal

OBJECTIVE

To critically assess the meta-analyses of Helicobacter pylori infection-related clinical studies, particularly the handling of between-study heterogeneity.

METHODS

A qualitative, all-language, systematic literature search was performed in Medline, PubMed, BioMed Central and Embase up to February 2003, supplemented by a manual search of major relevant journals. Assessment was according to modified criteria for literature searching, eligibility criteria, validity assessment, data extraction and presentation. Five parameters were used to assess the quality of the meta-analyses in handling between-study heterogeneity.

RESULTS

Of 84 potentially relevant citations, 47 were systematic reviews and of them 38 were meta-analyses. Of these 38 studies, 15 (39.5%) had conducted a literature search of multiple databases and 34 (89.5%) had conducted a supplementary manual search. The eligibility criteria were clearly presented in 81.6% of studies, but the quality of the primary studies was assessed in only 26.3%. The process and strategy for data extraction was reported in 57.9% of all studies; 19 (50%) studies planned statistical tests of between-study homogeneity and the results were reported in 18, but the level of statistical significance was reported in only 11 (57.9%). The selection of and justification for a statistical model was presented in 39.5% and 26.3% of studies, respectively. Among the 11 meta-analyses in which statistical between-study heterogeneity was reported, 54.5% ignored the statistical findings and proceeded to pool the study results. The implications of between-study heterogeneity were discussed in only 8 studies.

CONCLUSIONS

Many methodological flaws were identified in the meta-analyses of H. pylori-related clinical studies, particularly for assessing, reporting and interpreting between-study heterogeneity. This warrants consistent and urgent adherence by reviewers and journal editors to the methodological guidelines for meta-analyses.

Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction

BACKGROUND

There has been extensive discussion of the apparent conflict between meta-analyses and a mega-trial investigating the benefits of intravenous magnesium following myocardial infarction, in which the early trial results have been said to be 'too good to be true'.

METHODS

We apply Bayesian methods of meta-analysis to the trials available before and after the publication of the ISIS-4 results. We show how scepticism can be formally incorporated into an analysis as a Bayesian prior distribution, and how Bayesian meta-analysis models allow appropriate exploration of hypotheses that the treatment effect depends on the size of the trial or the risk in the control group.

RESULTS

Adoption of a sceptical prior would have led early enthusiasm for magnesium to be suitably tempered, but only if combined with a random effects meta-analysis, rather than the fixed effect analysis that was actually conducted.

CONCLUSIONS

We argue that neither a fixed effect nor a random effects analysis is appropriate when the mega-trial is included. The Bayesian framework provides many possibilities for flexible exploration of clinical hypotheses, but there can be considerable sensitivity to apparently innocuous assumptions.

Reamed versus nonreamed intramedullary nailing of lower extremity long bone fractures: a systematic overview and meta-analysis

OBJECTIVE

To determine the effect of reamed versus nonreamed intramedullary (IM) nailing of lower extremity long bone fractures on the rates of nonunion, implant failure, malunion, compartment syndrome, pulmonary embolus, and infection.

DESIGN

Quantitative systematic review of prospective, randomized controlled trials.

DATA IDENTIFICATION

MEDLINE and SCISEARCH computer searches provided lists of published randomized clinical trials from 1969 to 1998. Extensive hand searches of major orthopaedic journals, bibliographies of major orthopaedic texts, and personal files identified additional studies.

STUDY SELECTION AND DATA EXTRACTION

Of 676 citations initially identified, sixty proved potentially eligible, of which four published and five unpublished randomized trials met all eligibility criteria. Each of three investigators assessed study quality and abstracted relevant data.

RESULTS

The pooled relative risk of reamed versus nonreamed nails (nine trials, n = 646 patients) was 0.33 [95% confidence interval (CI), 0.16 to 0.68; p = 0.004]. The absolute risk difference in nonunion rates with reamed IM nailing was 7.0 percent (95% CI, 1 to 11 percent). Thus, one nonunion could be prevented for every fourteen patients treated with reamed IM nailing [number needed to treat (NNT) = 14.28]. The risk ratios for secondary outcome measures were: implant failure, 0.30 (95% CI, 0.16 to 0.58; p < 0.001); malunion, 1.06 (95% CI, 0.32 to 3.57); pulmonary embolus, 1.10 (95% CI, 0.26 to 4.76); compartment syndrome, 0.45 (95% CI, 0.13 to 1.56); and infection, 0.98 (95% CI, 0.21 to 4.76). Sensitivity analyses suggested that reported rates of nonunion and implant failure were higher in studies of lower quality. The type of long bone fractured (tibia or femur), the degree of soft tissue injury (open or closed), study quality, and whether a study was published or unpublished did not significantly alter the relative risk of nonunion between reamed and nonreamed IM nailing.

CONCLUSIONS

There is evidence from a pooled analysis of randomized trials that reamed IM nailing of lower extremity long bone fractures significantly reduces rates of nonunion and implant failure in comparison with nonreamed nailing.

New antiepileptic drugs: comparison of key clinical trials

PURPOSE

Data accrued from clinical trials of five new antiepileptic drugs (AEDs) are compared for efficacy in reducing seizures and self-reported adverse events as a basis of selection among new AEDs. Drawbacks to use of these data also are demonstrated.

METHODS

A review of double-blind, placebo-controlled clinical trials of a new AED or placebo added to a standard AED provided data on reduction of complex partial seizures (CPSs). Success is > or =50% fewer CPSs with a new AED or placebo; Overall Improvement is the success rate with drug minus the success rate with placebo. Adverse events were tabulated from product-labeling lists of COSTART items (incidence, > or =5%). The Summary Complaint score is the total number of reports of individual events for each AED.

RESULTS

Efficacy data demonstrate differences in Overall Improvement rates among five new AEDs and placebos (p = 0.001). However, rates of response to placebo also differed significantly among trials (p = 0.01). Adverse events predominantly affect central nervous system, psychiatric, and general body systems. However, patients in the placebo control groups did not consistently report adverse effects. Summary Complaint scores differ among the five new AEDs, but variability in use of COSTART terms nullifies comparisons.

CONCLUSIONS

Comparisons of data for five new AEDs provide information for selection among treatments when a second drug is needed to improve control of CPSs. However, significant differences among the control groups and other problems make comparisons between trials problematic. The final choice should be based on the need of the individual patient for superior seizure control versus minimal adverse effects.