Causally Interpretable Meta-analysis: Application in Adolescent HIV Prevention

When, why and how are estimated effects transported between populations? A scoping review of studies applying transportability methods

Transportability methods can improve the external validity of estimated effects by accounting for effect heterogeneity due to differently distributed covariates between populations. This scoping review aims to provide an overview of when, why and how transportability methods have been applied. We systematically searched MEDLINE (Ovid), Embase, Web of Science, EconLit and Google Scholar for studies published between 2010 and December 18, 2024. Studies using transportability methods in a numerical application for at least partly non-overlapping source and target populations were included. We identified 3432 unique studies and included 64 studies applying transportability methods. Over two thirds of the included studies (44/64) introduced new methods. Less than one third of the included studies (20/64) were pure applications of transportability methods. Most applied studies (17/20) transported effect estimates from randomized controlled trials. Effects were transported to target populations with either complete (9/20) or no (9/20) treatment and outcome data or both (2/20). The most frequent aims of applied studies were to transport estimated effects to new populations (10/20) and to assess effect heterogeneity explainable by measured covariates (8/20). How transportability methods were applied varied widely between studies, for instance in the covariate selection approach and sensitivity analysis. Methodological studies with a transportability application presented new transportability estimators for randomized data (5/44), specific transportability applications (e.g., meta-analysis, mediation analysis; 21/44) and other methodological aspects (e.g., covariate selection, missing data handling; 18/44). Transportability methods are a useful tool for knowledge transfer between populations. More applications of transportability methods and guidance for their use are desirable.

Systematic review of applied transportability and generalizability analyses: A landscape analysis

Transportability and generalizability analysis are novel causal inference methods that quantitatively assess external validity. Currently, it is unclear how these analyses are applied in practice. To characterize applications and methods, we conducted a landscape analysis of applied transportability and generalizability analyses using a systematic literature search of PubMed, CINAHL and Embase supplemented with hand-searches. We identified 68 publications describing transportability and generalizability analyses conducted with 83 unique source-target dataset pairs and reporting 99 distinct analyses. The majority of source and target datasets were collected in the US (n = 63/83, 75.9 %; and n = 59/83, 71.1 %, respectively). These methods were most often applied to transport RCT findings to observational studies (n = 38/83; 45.8 %), or to another RCT (n = 20/83; 24.1 %). Several studies used transportability analysis outside the standard application, for example to identify effect modifiers or calibrate measurements within an RCT. Methods that used weights and individual-level patient data were most common (n = 56/99, 56.5 %; n = 80/83, 96.4 %, respectively). Reporting quality varied across studies. Transportability analysis has a wide range of applications including supporting decision-making by improving evidence relevance and improving trial design by identifying contextual effect modifiers and calibrating outcome measurements. Efforts are needed to standardize analysis and reporting of these methods to improve transparency and uptake.

Transportability of model-based estimands in evidence synthesis

In evidence synthesis, effect modifiers are typically described as variables that induce treatment effect heterogeneity at the individual level, through treatment-covariate interactions in an outcome model parametrized at such level. As such, effect modification is defined with respect to a conditional measure, but marginal effect estimates are required for population-level decisions in health technology assessment. For noncollapsible measures, purely prognostic variables that are not determinants of treatment response at the individual level may modify marginal effects, even where there is individual-level treatment effect homogeneity. With heterogeneity, marginal effects for measures that are not directly collapsible cannot be expressed in terms of marginal covariate moments, and generally depend on the joint distribution of conditional effect measure modifiers and purely prognostic variables. There are implications for recommended practices in evidence synthesis. Unadjusted anchored indirect comparisons can be biased in the absence of individual-level treatment effect heterogeneity, or when marginal covariate moments are balanced across studies. Covariate adjustment may be necessary to account for cross-study imbalances in joint covariate distributions involving purely prognostic variables. In the absence of individual patient data for the target, covariate adjustment approaches are inherently limited in their ability to remove bias for measures that are not directly collapsible. Directly collapsible measures would facilitate the transportability of marginal effects between studies by: (1) reducing dependence on model-based covariate adjustment where there is individual-level treatment effect homogeneity or marginal covariate moments are balanced; and (2) facilitating the selection of baseline covariates for adjustment where there is individual-level treatment effect heterogeneity.

Systematically missing data in causally interpretable meta-analysis

Causally interpretable meta-analysis combines information from a collection of randomized controlled trials to estimate treatment effects in a target population in which experimentation may not be possible but from which covariate information can be obtained. In such analyses, a key practical challenge is the presence of systematically missing data when some trials have collected data on one or more baseline covariates, but other trials have not, such that the covariate information is missing for all participants in the latter. In this article, we provide identification results for potential (counterfactual) outcome means and average treatment effects in the target population when covariate data are systematically missing from some of the trials in the meta-analysis. We propose three estimators for the average treatment effect in the target population, examine their asymptotic properties, and show that they have good finite-sample performance in simulation studies. We use the estimators to analyze data from two large lung cancer screening trials and target population data from the National Health and Nutrition Examination Survey (NHANES). To accommodate the complex survey design of the NHANES, we modify the methods to incorporate survey sampling weights and allow for clustering.

Addressing Systematic Missing Data in the Context of Causally Interpretable Meta-analysis

Evidence synthesis involves drawing conclusions from trial samples that may differ from the target population of interest, and there is often heterogeneity among trials in sample characteristics, treatment implementation, study design, and assessment of covariates. Stitching together this patchwork of evidence requires subject-matter knowledge, a clearly defined target population, and guidance on how to weigh evidence from different trials. Transportability analysis has provided formal identifiability conditions required to make unbiased causal inference in the target population. In this manuscript, we review these conditions along with an additional assumption required to address systematic missing data. The identifiability conditions highlight the importance of accounting for differences in treatment effect modifiers between the populations underlying the trials and the target population. We perform simulations to evaluate the bias of conventional random effect models and multiply imputed estimates using the pooled trials sample and describe causal estimators that explicitly address trial-to-target differences in key covariates in the context of systematic missing data. Results indicate that the causal transportability estimators are unbiased when treatment effect modifiers are accounted for in the analyses. Results also highlight the importance of carefully evaluating identifiability conditions for each trial to reduce bias due to differences in participant characteristics between trials and the target population. Bias can be limited by adjusting for covariates that are strongly correlated with missing treatment effect modifiers, including data from trials that do not differ from the target on treatment modifiers, and removing trials that do differ from the target and did not assess a modifier.

Efficient and robust methods for causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a target population

We present methods for causally interpretable meta-analyses that combine information from multiple randomized trials to draw causal inferences for a target population of substantive interest. We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results for transporting causal inferences from a collection of independent randomized trials to a new target population in which experimental data may not be available. We propose an estimator for the potential outcome mean in the target population under each treatment studied in the trials. The estimator uses covariate, treatment, and outcome data from the collection of trials, but only covariate data from the target population sample. We show that it is doubly robust in the sense that it is consistent and asymptotically normal when at least one of the models it relies on is correctly specified. We study the finite sample properties of the estimator in simulation studies and demonstrate its implementation using data from a multicenter randomized trial.

Transporting a Prediction Model for Use in a New Target Population

We considered methods for transporting a prediction model for use in a new target population, both when outcome and covariate data for model development are available from a source population that has a different covariate distribution compared with the target population and when covariate data (but not outcome data) are available from the target population. We discuss how to tailor the prediction model to account for differences in the data distribution between the source population and the target population. We also discuss how to assess the model's performance (e.g., by estimating the mean squared prediction error) in the target population. We provide identifiability results for measures of model performance in the target population for a potentially misspecified prediction model under a sampling design where the source and the target population samples are obtained separately. We introduce the concept of prediction error modifiers that can be used to reason about tailoring measures of model performance to the target population. We illustrate the methods in simulated data and apply them to transport a prediction model for lung cancer diagnosis from the National Lung Screening Trial to the nationally representative target population of trial-eligible individuals in the National Health and Nutrition Examination Survey.

Transportability Methods for Time-to-Event Outcomes: Application in Adjuvant Colon Cancer Trials

PURPOSE

Differences in the benefits of treatment on 5-year overall survival have been observed in 12 randomized phase III colon cancer adjuvant clinical trials from the ACCENT group. We investigated the reasons for these differences by incorporating the distribution of the observed covariates from each trial.

MATERIALS AND METHODS

We applied state-of-the-art transportability methods on the basis of causal inference, and compared them with a conventional meta-analysis approach to predict the treatment effect for the target population. Prediction errors were defined to evaluate whether the identifiability conditions necessary for causal inference were satisfied among the 12 trials, and to measure the performance of each method.

RESULTS

In the one-trial-at-a-time transportability analysis, the ranks of prediction errors for the target population were mostly consistent with the discrepancy in treatment effects among the 12 trials across the three models. The overall prediction errors between the leave-one-trial-out transportability method and the conventional individual participant data meta-analysis approach were very similar, and more than 40% lower than the overall prediction errors from the one-trial-at-a-time transportability method.

CONCLUSION

The discrepancy in treatment effects among the 12 trials is unlikely to arise from the choice of model specification or distribution of observed covariates but from the distribution of unobserved covariates or study-level features. The ability to quantify heterogeneity among the 12 trials was greatly reduced in both the leave-one-trial-out transportability method and the conventional meta-analysis approach compared with the one-trial-at-a-time transportability method.

Modern Meta-Analytic Methods in Prevention Science: Introduction to the Special Issue

Meta-analyses that statistically synthesize evidence from multiple research studies can play an important role in advancing evidence-informed prevention science. When done in the context of a well-conducted systematic review, meta-analysis is a powerful tool for synthesizing evidence and exploring research questions that are difficult to address in individual studies, such as the association of individual study limitations on intervention effect estimates, replicability of empirical findings, and variation of effect estimates across populations and settings. Alongside the rapid growth in the number of published reviews and meta-analyses, there has been a parallel growth in the development of meta-analytic techniques to handle the increasingly complex types of questions and types of evidence relevant to prevention science. Despite this rapid evolution of meta-analytic techniques and approaches, there is still a lag between the development of new techniques and their uptake by researchers in the field. This paper serves as a brief introduction to this special issue of Prevention Science, entitled "Modern Meta-Analytic Methods in Prevention Science," which highlights recent developments in meta-analytic methods and demonstrates their application to prevention research. This special issue makes an important contribution to the field by ensuring these methodological advances are widely accessible to prevention science researchers, thereby improving their uptake and utilization, and ultimately improving the utility and rigor of research syntheses for informing evidence-based decision making in prevention.

Leveraging Research Synthesis Methods to Support Evidence-Based Policy- and Decision-Making

This commentary discusses the potential utility of research syntheses for evidence-based policy- and decision-making, examining the papers that comprise the special issue on modern meta-analytic methods. Evidence and data have the potential to play a critical role in the development of policies and in the administration of programs that meet the social and economic needs of children and families. Novel, innovative, and methodologically rigorous methods that allow for comprehensive and systematic research synthesis, such as those disseminated in this special issue, can help inform the work of the federal government and the prevention science field at large. Overall, the papers hold promise for strengthening the rigor of existing approaches, illustrate novel approaches, and demonstrate the utility of information that research syntheses can produce. Collectively, the studies in this special issue advance the available toolbox of methods that can be used to support evidence-based policy- and decision-making.

Next-Generation Meta-analysis for Next-Generation Questions: Introducing the Prevention Science Special Issue on Modern Meta-analytic Methods

This commentary accompanies the special issue of Prevention Science on modern meta-analytic methods. The papers that comprise this special issue are considered in terms of the next-generation meta-analytic questions they support: questions about multivariate relationships, drawing on real-life data structures, with improved usability, and answered openly. The contributions to this special issue illustrate a range of methods to address these questions, including meta-analytic structural equation modelling; robust variance estimation and network meta-analysis methods; transportability and causal inference; Bayesian methods; and open science. This special issue collectively represents a step forward in the field's ability to address questions of use to improving human welfare through preventing ill health, supporting uptake of these next-generation methods by applied researchers in prevention science. Future methodological developments in meta-analysis should be synergistic with the questions prevention scientists seek to answer, both creating new possibilities and meeting the challenges of improving human health and wellbeing.