Diagnostic Assessment of Assumptions for External Validity: An Example Using Data in Metastatic Colorectal Cancer

Visualizing External Validity: Graphical Displays to Inform the Extension of Treatment Effects from Trials to Clinical Practice

BACKGROUND

In the presence of effect measure modification, estimates of treatment effects from randomized controlled trials may not be valid in clinical practice settings. The development and application of quantitative approaches for extending treatment effects from trials to clinical practice settings is an active area of research.

METHODS

In this article, we provide researchers with a practical roadmap and four visualizations to assist in variable selection for models to extend treatment effects observed in trials to clinical practice settings and to assess model specification and performance. We apply this roadmap and visualizations to an example extending the effects of adjuvant chemotherapy (5-fluorouracil vs. plus oxaliplatin) for colon cancer from a trial population to a population of individuals treated in community oncology practices in the United States.

RESULTS

The first visualization screens for potential effect measure modifiers to include in models extending trial treatment effects to clinical practice populations. The second visualization displays a measure of covariate overlap between the clinical practice populations and the trial population. The third and fourth visualizations highlight considerations for model specification and influential observations. The conceptual roadmap describes how the output from the visualizations helps interrogate the assumptions required to extend treatment effects from trials to target populations.

CONCLUSIONS

The roadmap and visualizations can inform practical decisions required for quantitatively extending treatment effects from trials to clinical practice settings.

Improving the external validity of Antenatal Late Preterm Steroids trial findings

BACKGROUND

The external validity of randomised trials can be compromised when trial participants differ from real-world populations. In the Antenatal Late Preterm Steroids (ALPS) trial of antenatal corticosteroids at late preterm ages, participants had systematically younger gestational ages than those outside the trial setting. As risk of respiratory morbidity (the primary trial outcome) is higher at younger gestations, absolute benefits of corticosteroids calculated in the trial population may overestimate real-world treatment benefits.

OBJECTIVES

To estimate the real-world absolute risk reduction and number-needed-to-treat (NNT) for antenatal corticosteroids at late preterm ages, accounting for gestational age differences between the ALPS and real-world populations.

METHODS

Individual participant data from the ALPS trial (which recruited 2831 women with imminent preterm birth at 34+0 to 36+5 weeks') was appended to population-based data for 15,741 women admitted for delivery between 34+0 and 36+5 weeks' from British Columbia, Canada, 2000-2013. We used logistic regression to calculate inverse odds of sampling weights for each trial participant and re-estimated treatment effects of corticosteroids on neonatal respiratory morbidity in ALPS participants, weighted to reflect the gestational age distribution of the population-based (real-world) sample.

RESULTS

The real-world absolute risk reduction was estimated to be -2.2 (95% CI -4.6, 0.0) cases of respiratory morbidity per 100, compared with -2.8 (95% CI -5.3, -0.3) in original trial data. Corresponding NNTs were 46 in the real-world setting vs 35 in the trial. Our focus on absolute measures also highlighted that the benefits of antenatal corticosteroids may be meaningfully greater at 34 weeks vs. 36 weeks (e.g., risk reductions of -3.7 vs. -1.2 per 100 respectively).

CONCLUSIONS

The absolute risk reductions and NNTs associated with antenatal corticosteroid administration at late preterm ages estimated in our study may be more appropriate for patient counselling as they better reflect the anticipated benefits of treatment when used in a real-world situation.

Two-stage matching-adjusted indirect comparison

BACKGROUND

Anchored covariate-adjusted indirect comparisons inform reimbursement decisions where there are no head-to-head trials between the treatments of interest, there is a common comparator arm shared by the studies, and there are patient-level data limitations. Matching-adjusted indirect comparison (MAIC), based on propensity score weighting, is the most widely used covariate-adjusted indirect comparison method in health technology assessment. MAIC has poor precision and is inefficient when the effective sample size after weighting is small.

METHODS

A modular extension to MAIC, termed two-stage matching-adjusted indirect comparison (2SMAIC), is proposed. This uses two parametric models. One estimates the treatment assignment mechanism in the study with individual patient data (IPD), the other estimates the trial assignment mechanism. The first model produces inverse probability weights that are combined with the odds weights produced by the second model. The resulting weights seek to balance covariates between treatment arms and across studies. A simulation study provides proof-of-principle in an indirect comparison performed across two randomized trials. Nevertheless, 2SMAIC can be applied in situations where the IPD trial is observational, by including potential confounders in the treatment assignment model. The simulation study also explores the use of weight truncation in combination with MAIC for the first time.

RESULTS

Despite enforcing randomization and knowing the true treatment assignment mechanism in the IPD trial, 2SMAIC yields improved precision and efficiency with respect to MAIC in all scenarios, while maintaining similarly low levels of bias. The two-stage approach is effective when sample sizes in the IPD trial are low, as it controls for chance imbalances in prognostic baseline covariates between study arms. It is not as effective when overlap between the trials' target populations is poor and the extremity of the weights is high. In these scenarios, truncation leads to substantial precision and efficiency gains but induces considerable bias. The combination of a two-stage approach with truncation produces the highest precision and efficiency improvements.

CONCLUSIONS

Two-stage approaches to MAIC can increase precision and efficiency with respect to the standard approach by adjusting for empirical imbalances in prognostic covariates in the IPD trial. Further modules could be incorporated for additional variance reduction or to account for missingness and non-compliance in the IPD trial.

Generalizability of ORIENT-11 trial results to a US standard of care cohort with advanced non-small-cell lung cancer

Aim: This retrospective study estimated efficacy and safety of sintilimab + pemetrexed + platinum (SPP) versus placebo + pemetrexed + platinum (PPP) in untreated locally advanced/metastatic, nonsquamous non-small-cell lung cancer (NSCLC), after adjusting each ORIENT-11 trial patient's contribution to ORIENT-11 data based on characteristics of a target US population. Materials & methods: The target US population (n = 557) was selected from a real-world deidentified advanced NSCLC database based on key ORIENT-11 eligibility criteria. Inverse probability weights for ORIENT-11 patients (n = 397) relative to US patients were calculated. Efficacy and safety of SPP versus PPP were adjusted by inverse probability weights. Results: After adjustment, progression-free survival remained superior for SPP. Other efficacy and safety outcomes were consistent. Conclusion: These results provide evidence on how the effects observed with SPP in ORIENT-11 could translate to a US population with untreated locally advanced/metastatic nonsquamous NSCLC.

Target validity: Bringing treatment of external validity in line with internal validity

Purpose of Review

"Target bias" is the difference between an estimate of association from a study sample and the causal effect in the target population of interest. It is the sum of internal and external bias. Given the extensive literature on internal validity, here, we review threats and methods to improve external validity.

Recent findings

External bias may arise when the distribution of modifiers of the effect of treatment differs between the study sample and the target population. Methods including those based on modeling the outcome, modeling sample membership, and doubly robust methods are available, assuming data on the target population is available.

Summary

The relevance of information for making policy decisions is dependent on both the actions that were studied and the sample in which they were evaluated. Combining methods for addressing internal and external validity can improve the policy relevance of study results.

Directed Acyclic Graphs, Effect Measure Modification, and Generalizability

Directed acyclic graphs (DAGs) have had a major impact on the field of epidemiology by providing straightforward graphical rules for determining when estimates are expected to lack causally interpretable internal validity. Much less attention has been paid, however, to what DAGs can tell researchers about effect measure modification and external validity. In this work, we describe 2 rules based on DAGs related to effect measure modification. Rule 1 states that if a variable, $P$, is conditionally independent of an outcome, $Y$, within levels of a treatment, $X$, then $P$ is not an effect measure modifier for the effect of $X$ on $Y$ on any scale. Rule 2 states that if $P$ is not conditionally independent of $Y$ within levels of $X$, and there are open causal paths from $X$ to $Y$ within levels of $P$, then $P$ is an effect measure modifier for the effect of $X$ on $Y$ on at least 1 scale (given no exact cancelation of associations). We then show how Rule 1 can be used to identify sufficient adjustment sets to generalize nested trials studying the effect of $X$ on $Y$ to the total source population or to those who did not participate in the trial.

Reweighting Oranges to Apples: Transported RE-LY Trial Versus Nonexperimental Effect Estimates of Anticoagulation in Atrial Fibrillation

BACKGROUND

Results from trials and nonexperimental studies are often directly compared, with little attention paid to differences between study populations. When target and trial population data are available, accounting for these differences through transporting trial results to target populations of interest provides useful perspective. We aimed to compare two-year risk differences (RDs) for ischemic stroke, mortality, and gastrointestinal bleeding in older adults with atrial fibrillation initiating dabigatran and warfarin when using trial transport methods versus nonexperimental methods.

METHODS

We identified Medicare beneficiaries who initiated warfarin or dabigatran from a 20% nationwide sample. To transport treatment effects observed in the randomized evaluation of long-term anticoagulation trial, we applied inverse odds weights to standardize estimates to two Medicare target populations of interest, initiators of: (1) dabigatran and (2) warfarin. Separately, we conducted a nonexperimental study in the Medicare populations using standardized morbidity ratio weighting to control measured confounding.

RESULTS

Comparing dabigatran to warfarin, estimated two-year RDs for ischemic stroke were similar with trial transport and nonexperimental methods. However, two-year mortality RDs were closer to the null when using trial transport versus nonexperimental methods for the dabigatran target population (transported RD: -0.57%; nonexperimental RD: -1.9%). Estimated gastrointestinal bleeding RDs from trial transport (dabigatran initiator RD: 1.8%; warfarin initiator RD: 1.9%) appeared more harmful than nonexperimental results (dabigatran initiator RD: 0.14%; warfarin initiator RD: 0.57%).

CONCLUSIONS

Differences in study populations can and should be considered quantitatively to ensure results are relevant to populations of interest, particularly when comparing trial with nonexperimental findings. See video abstract: http://links.lww.com/EDE/B703.

Reweighting Randomized Controlled Trial Evidence to Better Reflect Real Life - A Case Study of the Innovative Medicines Initiative

Evidence from randomized controlled trials available for timely health technology assessments of new pharmacological treatments and regulatory decision making may not be generalizable to local patient populations, often resulting in decisions being made under uncertainty. In recent years, several reweighting approaches have been explored to address this important question of generalizability to a target population. We present a case study of the Innovative Medicines Initiative to illustrate the inverse propensity score reweighting methodology, which may allow us to estimate the expected treatment benefit if a clinical trial had been run in a broader real‐world target population. We learned that identifying treatment effect modifiers, understanding and managing differences between patient characteristic data sets, and balancing the closeness of trial and target patient populations with effective sample size are key to successfully using this methodology and potentially mitigating some of this uncertainty around local decision making.