Reweighting Randomized Controlled Trial Evidence to Better Reflect Real Life - A Case Study of the Innovative Medicines Initiative

Model-based standardization using multiple imputation

BACKGROUND

When studying the association between treatment and a clinical outcome, a parametric multivariable model of the conditional outcome expectation is often used to adjust for covariates. The treatment coefficient of the outcome model targets a conditional treatment effect. Model-based standardization is typically applied to average the model predictions over the target covariate distribution, and generate a covariate-adjusted estimate of the marginal treatment effect.

METHODS

The standard approach to model-based standardization involves maximum-likelihood estimation and use of the non-parametric bootstrap. We introduce a novel, general-purpose, model-based standardization method based on multiple imputation that is easily applicable when the outcome model is a generalized linear model. We term our proposed approach multiple imputation marginalization (MIM). MIM consists of two main stages: the generation of synthetic datasets and their analysis. MIM accommodates a Bayesian statistical framework, which naturally allows for the principled propagation of uncertainty, integrates the analysis into a probabilistic framework, and allows for the incorporation of prior evidence.

RESULTS

We conduct a simulation study to benchmark the finite-sample performance of MIM in conjunction with a parametric outcome model. The simulations provide proof-of-principle in scenarios with binary outcomes, continuous-valued covariates, a logistic outcome model and the marginal log odds ratio as the target effect measure. When parametric modeling assumptions hold, MIM yields unbiased estimation in the target covariate distribution, valid coverage rates, and similar precision and efficiency than the standard approach to model-based standardization.

CONCLUSION

We demonstrate that multiple imputation can be used to marginalize over a target covariate distribution, providing appropriate inference with a correctly specified parametric outcome model and offering statistical performance comparable to that of the standard approach to model-based standardization.

Transporting Comparative Effectiveness Evidence Between Countries: Considerations for Health Technology Assessments

Internal validity is often the primary concern for health technology assessment agencies when assessing comparative effectiveness evidence. However, the increasing use of real-world data from countries other than a health technology assessment agency's target population in effectiveness research has increased concerns over the external validity, or "transportability", of this evidence, and has led to a preference for local data. Methods have been developed to enable a lack of transportability to be addressed, for example by accounting for cross-country differences in disease characteristics, but their consideration in health technology assessments is limited. This may be because of limited knowledge of the methods and/or uncertainties in how best to utilise them within existing health technology assessment frameworks. This article aims to provide an introduction to transportability, including a summary of its assumptions and the methods available for identifying and adjusting for a lack of transportability, before discussing important considerations relating to their use in health technology assessment settings, including guidance on the identification of effect modifiers, guidance on the choice of target population, estimand, study sample and methods, and how evaluations of transportability can be integrated into health technology assessment submission and decision processes.

Generalizability of clinical trial efficacy results to a real-world population: An example in migraine prevention

BACKGROUND

Health care decision makers are often concerned about the external validity of randomized controlled trials (RCTs), as their results may not apply to certain patients in the real world who intend to receive treatment.

OBJECTIVE

To demonstrate a methodology for assessing the generalizability of clinical trial results to a real-world population, before sufficient and appropriate real-world effectiveness data are available, using individual patient-level data from an RCT and aggregated baseline data from a real-world French registry in migraine.

METHODS

The analyses were conducted in 2 steps. First, individual patient-level baseline data from the multinational CONQUER RCT were weighted to match aggregated real-world InovPain registry patient characteristic data. Matched patient characteristics were sex, age, migraine type and duration, number of monthly migraine headache days, and number of monthly headache days at baseline. Second, the weighted CONQUER patient data were used to reanalyze the primary endpoint of CONQUER (least squares mean change from baseline in the number of monthly migraine headache days during the 3-month double-blind treatment phase) using predefined methodology. Sensitivity analyses were conducted to assess the robustness of findings.

RESULTS

A total of 462 patients with migraine were randomized and treated with galcanezumab or placebo in CONQUER; aggregated InovPain data were available from 130 patients with migraine. We identified no important differences in baseline patient characteristics between the 2 prespecified populations, suggesting good external validity for CONQUER. Although this limited the extent of observed differences between the original and matched CONQUER populations, weighting of CONQUER data did help harmonize the 2 datasets and allow the results obtained in CONQUER to be generalized to patients more representative of the real-world French population with migraine. Results of weighted analyses suggested that galcanezumab would be superior to placebo for reducing monthly migraine headache days in a clinical trial in patients with episodic or chronic migraine who reflected the characteristics of patients eligible to receive the drug in France.

CONCLUSIONS

Findings suggest that our methods may be helpful for assessing the generalizability of clinical trial results to a real-world population before the availability of substantial real-world clinical data.

Transportability of two heart failure trials to a disease registry using individual patient data

OBJECTIVES

Randomized controlled trials are the gold-standard for determining therapeutic efficacy, but are often unrepresentative of real-world settings. Statistical transportation methods (hereafter transportation) can partially account for these differences, improving trial applicability without breaking randomization. We transported treatment effects from two heart failure (HF) trials to a HF registry.

STUDY DESIGN AND SETTING

Individual-patient-level data from two trials (Carvedilol or Metoprolol European Trial (COMET), comparing carvedilol and metoprolol, and digitalis investigation group trial (DIG), comparing digoxin and placebo) and a Scottish HF registry were obtained. The primary end point for both trials was all-cause mortality; composite outcomes were all-cause mortality or hospitalization for COMET and HF-related death or hospitalization for DIG. We performed transportation using regression-based and inverse odds of sampling weights (IOSW) approaches.

RESULTS

Registry patients were older, had poorer renal function and received higher-doses of loop-diuretics than trial participants. For each trial, point estimates were similar for the original and IOSW (e.g., DIG composite outcome: OR 0.75 (0.69, 0.82) vs. 0.73 (0.64, 0.83)). Treatment effect estimates were also similar when examining high-risk (0.64 (0.46, 0.89)) and low-risk registry patients (0.73 (0.61, 0.86)). Similar results were obtained using regression-based transportation.

CONCLUSION

Regression-based or IOSW approaches can be used to transport trial effect estimates to patients administrative/registry data, with only moderate reductions in precision.

Reweighting estimators to extend the external validity of clinical trials: methodological considerations

Methods to extend the strong internal validity of randomized controlled trials to reliably estimate treatment effects in target populations are gaining attention. This paper enumerates steps recommended for undertaking such extended inference, discusses currently viable choices for each one, and provides recommendations. We demonstrate a complete extended inference from a clinical trial studying a pharmaceutical treatment for Alzheimer's disease (AD) to a realistic target population of European residents diagnosed with AD. This case study highlights approaches to overcoming practical difficulties and demonstrates limitations of reliably extending inference from a trial to a real-world population.

Testing the "RCT augmentation" methodology: A trial simulation study to guide the broadening of trials eligibility criteria and inform on effectiveness

Background

Exclusion criteria that are treatment effect modifiers (TEM) decrease RCTs results generalisability and the potentials of effectiveness estimation. In "augmented RCTs", a small proportion of otherwise-excluded patients are included to allow for effectiveness estimation. In Hodgkin Lymphoma (HL) RCTs, older age and comorbidity are common exclusion criteria, while also TEM. We simulated HL RCTs augmented with age or comorbidity, and explored in each scenario the impact of augmentation on effectiveness estimation accuracy.

Methods

Simulated data with a population of HL individuals initiating drug A or B was generated. There were drug-age and drug-comorbidity interactions in the simulated data, with a greater magnitude of the former compared to the latter. Multiple augmented RCTs were simulated by randomly selecting patients with increasing proportions of older, or comorbid patients. Treatment effect size was expressed using the between-group Restricted Mean Survival Time (RMST) difference at 3 years. For each augmentation proportion, a model estimating the "real-world" treatment effect (effectiveness) was fitted and the estimation error measured (Root Mean Square Error, RMSE).

Results

In simulated RCTs including none (0%), or the real-world proportion (30%) of older patients, the interquartile range of RMST difference was 0.4-0.5 years and 0.2-0.3 years, respectively, and RMSE were 0.198 years (highest possible error) and 0.056 years (lowest), respectively. Augmenting RCTs with 5% older patients decreased estimation error substantially (RMSE = 0.076 years). Augmentation with comorbid patients proved less useful for effectiveness estimation.

Conclusion

In augmented RCTs aiming to inform the effectiveness of drugs, augmentation should concern in priority those exclusion criteria of suspected important TEM magnitude, so as to minimie the proportion of augmentation necessary for good effectiveness estimations.

A first step towards best practice recommendations for the design and statistical analyses of pragmatic clinical trials: a modified Delphi approach

AIM

Pragmatic clinical trials (PCTs) are randomised trials implemented through routine clinical practice, where design parameters of traditional randomised controlled trials are modified to increase generalizability. However, this may introduce statistical challenges. We aimed to identify these challenges and discuss possible solutions leading to best practice recommendations for the design and analysis of PCTs.

METHODS

A modified Delphi method was used to reach consensus among a panel of 11 experts in clinical trials and statistics. Statistical issues were identified in a focused literature review and aggregated with insights and possible solutions from expert collected through a series of survey iterations. Issues were ranked according to their importance.

RESULTS

27 articles were included and combined with experts' insight to generate a list of issues categorized into: participants; recruiting sites; randomisation, blinding and intervention; outcome (selection and measurement); and data analysis. Consensus was reached about the most important issues: risk of participants' attrition; heterogeneity of "usual care" across sites; absence of blinding; use of a subjective endpoint; and data analysis aligned with the trial estimand. Potential issues should be anticipated and preferably be addressed in the trial protocol. The experts provided solutions regarding data collection and data analysis, which were considered of equal importance.

DISCUSSION

A set of important statistical issues in PCTs was identified and approaches were suggested to anticipate and/or minimize these through data analysis. Any impact of choosing a pragmatic design feature should be gauged in the light of the trial estimand.

Framework for the synthesis of non-randomised studies and randomised controlled trials: a guidance on conducting a systematic review and meta-analysis for healthcare decision making

INTRODUCTION

High-quality randomised controlled trials (RCTs) provide the most reliable evidence on the comparative efficacy of new medicines. However, non-randomised studies (NRS) are increasingly recognised as a source of insights into the real-world performance of novel therapeutic products, particularly when traditional RCTs are impractical or lack generalisability. This means there is a growing need for synthesising evidence from RCTs and NRS in healthcare decision making, particularly given recent developments such as innovative study designs, digital technologies and linked databases across countries. Crucially, however, no formal framework exists to guide the integration of these data types.

OBJECTIVES AND METHODS

To address this gap, we used a mixed methods approach (review of existing guidance, methodological papers, Delphi survey) to develop guidance for researchers and healthcare decision-makers on when and how to best combine evidence from NRS and RCTs to improve transparency and build confidence in the resulting summary effect estimates.

RESULTS

Our framework comprises seven steps on guiding the integration and interpretation of evidence from NRS and RCTs and we offer recommendations on the most appropriate statistical approaches based on three main analytical scenarios in healthcare decision making (specifically, 'high-bar evidence' when RCTs are the preferred source of evidence, 'medium,' and 'low' when NRS is the main source of inference).

CONCLUSION

Our framework augments existing guidance on assessing the quality of NRS and their compatibility with RCTs for evidence synthesis, while also highlighting potential challenges in implementing it. This manuscript received endorsement from the International Society for Pharmacoepidemiology.

Generalizability of ORIENT-11 trial results to a US standard of care cohort with advanced non-small-cell lung cancer

Aim: This retrospective study estimated efficacy and safety of sintilimab + pemetrexed + platinum (SPP) versus placebo + pemetrexed + platinum (PPP) in untreated locally advanced/metastatic, nonsquamous non-small-cell lung cancer (NSCLC), after adjusting each ORIENT-11 trial patient's contribution to ORIENT-11 data based on characteristics of a target US population. Materials & methods: The target US population (n = 557) was selected from a real-world deidentified advanced NSCLC database based on key ORIENT-11 eligibility criteria. Inverse probability weights for ORIENT-11 patients (n = 397) relative to US patients were calculated. Efficacy and safety of SPP versus PPP were adjusted by inverse probability weights. Results: After adjustment, progression-free survival remained superior for SPP. Other efficacy and safety outcomes were consistent. Conclusion: These results provide evidence on how the effects observed with SPP in ORIENT-11 could translate to a US population with untreated locally advanced/metastatic nonsquamous NSCLC.

Health technology assessment for the acute and preventive treatment of migraine: A position statement of the International Headache Society

The Clinical Trials Subcommittee of the International Headache Society presents the first Health Technology Assessment for the Acute Treatment of Migraine Attacks and Prevention of Migraine. Health technology assessments are systematic evaluations of the properties, effects, and consequences of healthcare technologies; this position statement is designed to inform decision makers about access to and reimbursement for medications and devices for the acute and preventive treatment of migraine. This position statement extends beyond the already available guidelines on randomized controlled trials for migraine to incorporate real-world evidence and a synthetic approach for considering multiple data sources and modelling methods when assessing the value of migraine treatments.