Meta-analysis: is moving the goal post the answer?

Determining the conclusiveness of a meta-analysis

The pursuit of conclusive evidence related to an unanswered foreground (decision-making) question has been the driving factor behind multiple ongoing and planned randomized controlled trials as well as meta-analyses. However, a fundamental challenge lies in establishing robust methods for ascertaining whether a collection of synthesized trials has yielded a definitive answer to that foreground question through the process of meta-analysis. This article explores the evolution of methods that attempt to address this challenge. These methods have primarily focused on defining and measuring the sufficiency and stability of evidence within a meta-analytic context. Cumulative meta-analysis and trial sequential analysis are the tools currently used, but they both come with limitations and challenges. We further discuss methods aimed at evaluating the evolution of effects over time more directly, such as the recursive cumulative meta-analysis. The latter method can be considered a better alternative, as it serves to demonstrate whether there is a true underlying treatment effect to which the meta-analysis is converging. However, recursive cumulative meta-analysis falls short of a specific indicator that establishes whether convergence has been reached. We coin the term exit for a meta-analysis where convergence can be demonstrated. Developing methods to determine the exit status of a meta-analysis is the next priority in research synthesis methods, as it will indicate that the research journey has concluded on a particular foreground question with no expectation of a different result with the addition of future trials.

Magnesium sulphate for fetal neuroprotection at imminent risk for preterm delivery: a systematic review with meta-analysis and trial sequential analysis

BACKGROUND

Ordinary meta-analyses indicate that magnesium sulphate (MgSO₄ ) treatment in women at imminent risk for preterm delivery decreases the offspring's risk of cerebral palsy (CP). However, repetitive testing of cumulative data calls for statistical caution, e.g. by trial sequential analysis (TSA), for which there are previously insufficient samples to draw a firm conclusion. Recently, a randomised controlled trial (RCT) provided additional data that potentially increased the sample size such that a new TSA might detect a statistically significant effect.

OBJECTIVES

To assess the possible fetal neuroprotective effect of MgSO₄ for women at imminent risk for preterm delivery in an updated systematic review with meta-analysis and TSA.

SEARCH STRATEGY

We searched MEDLINE, Embase, Cochrane and ClinicalTrials.gov on 8 October 2019. The search strategy clustered terms describing the MgSO₄ intervention and preterm delivery.

SELECTION CRITERIA

RCTs.

DATA COLLECTION AND ANALYSIS

Two reviewers extracted the data. Summary relative risks (RRs) and 95% confidence intervals (CIs) were calculated using fixed-effects models. A TSA was applied to the primary outcome, CP. The quality of the evidence was assessed using GRADE. The protocol was registered in PROSPERO (registration: CRD42019151441).

MAIN RESULTS

We identified six eligible trials (5917 women). MgSO₄ intervention in women at imminent risk for preterm birth decreased the offspring's CP risk (meta-analysis RR 0.68, 95% CI 0.54-0.85; TSA RR 0.69, 95% CI 0.48-0.97).

CONCLUSIONS

This systematic review with meta-analysis and TSA shows conclusively that MgSO₄ , when given to women at imminent risk for preterm delivery, decreases the offspring's CP risk.

TWEETABLE ABSTRACT

Antenatal magnesium sulphate decreases the risk of cerebral palsy in children born preterm.

Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses

BACKGROUND

Random error may cause misleading evidence in meta-analyses. The required number of participants in a meta-analysis (i.e. information size) should be at least as large as an adequately powered single trial. Trial sequential analysis (TSA) may reduce risk of random errors due to repetitive testing of accumulating data by evaluating meta-analyses not reaching the information size with monitoring boundaries. This is analogous to sequential monitoring boundaries in a single trial.

METHODS

We selected apparently conclusive (P </= 0.05) Cochrane neonatal meta-analyses. We applied heterogeneity-adjusted and unadjusted TSA on these meta-analyses by calculating the information size, the monitoring boundaries, and the cumulative Z-statistic after each trial. We identified the proportion of meta-analyses that did not reach the required information size and the proportion of these meta-analyses in which the Z-curve did not cross the monitoring boundaries.

RESULTS

Of 54 apparently conclusive meta-analyses, 39 (72%) did not reach the heterogeneity-adjusted information size required to accept or reject an intervention effect of 25% relative risk reduction. Of these 39, 19 meta-analyses (49%) were considered inconclusive, because the cumulative Z-curve did not cross the monitoring boundaries. The median number of participants required to reach the required information size was 1591 (range, 339-6149). TSA without heterogeneity adjustment largely confirmed these results.

CONCLUSIONS

Many apparently conclusive Cochrane neonatal meta-analyses may become inconclusive when the statistical analyses take into account the risk of random error due to repetitive testing.

Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses

OBJECTIVES

To evaluate meta-analyses with trial sequential analysis (TSA). TSA adjusts for random error risk and provides the required number of participants (information size) in a meta-analysis. Meta-analyses not reaching information size are analyzed with trial sequential monitoring boundaries analogous to interim monitoring boundaries in a single trial.

STUDY DESIGN AND SETTING

We applied TSA on meta-analyses performed in Cochrane Neonatal reviews. We calculated information sizes and monitoring boundaries with three different anticipated intervention effects of 30% relative risk reduction (TSA(30%)), 15% (TSA(15%)), or a risk reduction suggested by low-bias risk trials of the meta-analysis corrected for heterogeneity (TSA(LBHIS)).

RESULTS

A total of 174 meta-analyses were eligible; 79 out of 174 (45%) meta-analyses were statistically significant (P<0.05). In the significant meta-analyses, TSA(30%) showed firm evidence in 61%. TSA(15%) and TSA(LBHIS) found firm evidence in 33% and 73%, respectively. The remaining significant meta-analyses had potentially spurious evidence of effect. In the 95 statistically nonsignificant (P>or=0.05) meta-analyses, TSA(30%) showed absence of evidence in 80% (insufficient information size). TSA(15%) and TSA(LBHIS) found that 95% and 91% had absence of evidence. The remaining nonsignificant meta-analyses had evidence of lack of effect.

CONCLUSION

TSA reveals insufficient information size and potentially false positive results in many meta-analyses.

Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis

BACKGROUND AND OBJECTIVE

Cumulative meta-analyses are prone to produce spurious P<0.05 because of repeated testing of significance as trial data accumulate. Information size in a meta-analysis should at least equal the sample size of an adequately powered trial. Trial sequential analysis (TSA) corresponds to group sequential analysis of a single trial and may be applied to meta-analysis to evaluate the evidence.

STUDY DESIGN AND SETTING

Six randomly selected neonatal meta-analyses with at least five trials reporting a binary outcome were examined. Low-bias heterogeneity-adjusted information size and information size determined from an assumed intervention effect of 15% were calculated. These were used for constructing trial sequential monitoring boundaries. We assessed the cumulative z-curves' crossing of P=0.05 and the boundaries.

RESULTS

Five meta-analyses showed early potentially spurious P<0.05 values. In three significant meta-analyses the cumulative z-curves crossed both boundaries, establishing firm evidence of an intervention effect. In two nonsignificant meta-analyses the cumulative z-curves crossed P=0.05, but never the boundaries, demonstrating early potentially spurious P<0.05 values. In one nonsignificant meta-analysis the cumulative z-curves never crossed P=0.05 or the boundaries.

CONCLUSION

TSAs may establish when firm evidence is reached in meta-analysis.

Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data

Meta-analyses of randomized evidence may include published, unpublished, and updated data in an ongoing estimation process that continuously accommodates more data. Synthesis may be performed either with group data or with meta-analysis of individual patient data (MIPD). Although MIPD with updated data is considered the gold standard of evidence, there is a need for a careful study of the impact different sources of data have on a meta-analysis and of the change in the treatment effect estimates over sequential information steps. Unpublished data and late-appearing data may be different from early-appearing data. Updated information after the end of the main study follow-up may be affected by cross-overs, missing information, and unblinding. The estimated treatment effect may thus depend on the completeness and updating of the available evidence. To address these issues, we present recursive cumulative meta-analysis (RCM) as an extension of cumulative metaanalysis. Recursive cumulative meta-analysis is based on the principle of recalculating the results of a cumulative meta-analysis with each new or updated piece of information and focuses on the evolution of the treatment effect as a more complete and updated picture of the evidence becomes available. An examination of the perturbations of the cumulative treatment effect over sequential information steps may signal the presence of bias or heterogeneity in a meta-analysis. Recursive cumulative meta-analysis may suggest whether there is a true underlying treatment effect to which the meta-analysis is converging and how treatment effects are sequentially altered by new or modified evidence. The method is illustrated with an example from the conduct of an MIPD on acyclovir in human immunodeficiency virus infection. The relative strengths and limitations of both metaanalysis of group data and MIPD are discussed through the RCM perspective.