The Extrapolation Performance of Survival Models for Data With a Cure Fraction: A Simulation Study

Assessing Long-term Treatment Benefits Using Complementary Statistical Approaches: An In Silico Analysis of the Phase III Keynote-045 and Checkmate-214 Immune Checkpoint Inhibitor Trials

BACKGROUND

The Keynote-045 trial illustrates that the long-term benefit (LTB) of treatment does not always translate to improved progression-free survival (PFS). Milestone survival and flexible parametric survival model with cure (FPCM) have been proposed as complementary statistical approaches to more comprehensively evaluate LTBs of treatments.

OBJECTIVE

The current study compares milestone survival and FPCM analyses to evaluate treatment effects of immune checkpoint inhibitor (ICI) phase III trials.

DESIGN, SETTING, AND PARTICIPANTS

Individual patient data, from initial and follow-up analyses of Keynote-045 (urothelial cancer) and Checkmate-214 (advanced renal cell carcinoma), were reconstructed for PFS.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

Each trial was reanalyzed using the Cox proportional hazard regression and two complementary methods (milestone survival and FPCM) to estimate treatment impact on the LTB.

RESULTS AND LIMITATIONS

For each trial, there was evidence of nonproportional hazards. For the long-term analysis of the Keynote-045 trial, FPCM identified a time-dependent effect on PFS, but the Cox model found no statistical difference in PFS (hazard ratio, 0.90; 95% confidence interval, 0.75-1.08). Milestone survival and FPCM identified improvements in the LTB fractions. This was consistent with the results from the reanalysis of Keynote-045, based on the shorter follow-up, although the LTB fraction was not retained. The increase in PFS in Checkmate-214 was identified by both Cox model and FPCM. Experimental treatment-dependent improvement in the LTB fraction was demonstrated using milestone survival and FPCM. The LTB fraction estimated with FPCM was consistent with the results from the reanalysis of the shorter follow-up period.

CONCLUSIONS

Although ICIs show substantial shifts toward LTBs in terms of PFS, based on a conventional Kaplan-Meier or Cox model analysis, our approach provides an alternative assessment of benefit-risk ratios for new therapeutics and facilitates communicating risk to patients. Kidney patients treated with ICIs can be counseled that they are potentially cured, but future work will need to definitively validate this conclusion.

PATIENT SUMMARY

Although immune checkpoint inhibitor treatments show substantial shifts toward long-term benefits in terms of progression-free survival, a more rigorous attempt to quantify this shift, rather than simply using a Kaplan-Meier estimate or comparing progression-free survival curves using the classic Cox model, is warranted. Our results suggest that advanced renal cell carcinoma patients who had not received a previous treatment are functionally cured by nivolumab and ipilimumab, which is not the case for second-line urothelial carcinoma.

Survival Extrapolation Incorporating General Population Mortality Using Excess Hazard and Cure Models: A Tutorial

BACKGROUND

Different parametric survival models can lead to widely discordant extrapolations and decision uncertainty in cost-effectiveness analyses. The use of excess hazard (EH) methods, which incorporate general population mortality data, has the potential to reduce model uncertainty. This review highlights key practical considerations of EH methods for estimating long-term survival.

METHODS

Demonstration of methods used a case study of 686 patients from the German Breast Cancer Study Group, followed for a maximum of 7.3 y and divided into low (1/2) and high (3) grade cancers. Seven standard parametric survival models were fit to each group separately. The same 7 distributions were then used in an EH framework, which incorporated general population mortality rates, and fitted both with and without a cure parameter. Survival extrapolations, restricted mean survival time (RMST), and difference in RMST between high and low grades were compared up to 30 years along with Akaike information criterion goodness-of-fit and cure fraction estimates. The sensitivity of the EH models to lifetable misspecification was investigated.

RESULTS

In our case study, variability in survival extrapolations was extensive across the standard models, with 30-y RMST ranging from 7.5 to 14.3 y. Incorporation of general population mortality rates using EH cure methods substantially reduced model uncertainty, whereas EH models without cure had less of an effect. Long-term treatment effects approached the null for most models but at varying rates. Lifetable misspecification had minimal effect on RMST differences.

CONCLUSIONS

EH methods may be useful for survival extrapolation, and in cancer, EHs may decrease over time and be easier to extrapolate than all-cause hazards. EH cure models may be helpful when cure is plausible and likely to result in less extrapolation variability.

HIGHLIGHTS

In health economic modeling, to help anchor long-term survival extrapolation, it has been recommended that survival models incorporate background mortality rates using excess hazard (EH) methods.We present a thorough description of EH methods with and without the assumption of cure and demonstrate user-friendly software to aid researchers wishing to use these methods.EH models are applied to a case study, and we demonstrate that EHs are easier to extrapolate and that the use of the EH cure model, when cure is plausible, can reduce extrapolation variability.EH methods are relatively robust to lifetable misspecification.

A semi-Markov multistate cure model for estimating intervention effects in stepped wedge design trials

Multistate models are useful for studying exposures that affect transitions among a set of health states. However, they can be challenging to apply when exposures are time-varying. We develop a multistate model and a method of likelihood construction that allows application of the model to data in which interventions or other exposures can be time-varying and an individual may to be exposed to multiple intervention conditions while progressing through states. The model includes cure proportions, reflecting the possibility that some individuals will never leave certain states. We apply the approach to analyze patient vaccination data from a stepped wedge design trial evaluating two interventions to increase uptake of human papillomavirus vaccination. The states are defined as the number of vaccine doses the patient has received. We model state transitions as a semi-Markov process and include cure proportions to account for individuals who will never leave a given state (e.g. never receive their next dose). Multistate models typically quantify intervention effects as hazard ratios contrasting the intensities of transitions between states in intervention versus control conditions. For multistate processes, another clinically meaningful outcome is the change in the percentage of the study population that has achieved a specific state (e.g. completion of all required doses) by a specific point in time due to an intervention. We present a method for quantifying intervention effects in this manner. We apply the model to both simulated and real-world data and also explore some conditions under which such models may give biased results.

Impact of Extrapolation Model Choices on the Structural Uncertainty in Economic Evaluations for Cancer Immunotherapy: A Case Study of Checkmate 067

OBJECTIVES

The aim of this study was to compare the performance of different extrapolation modeling techniques and analyze their impact on structural uncertainties in the economic evaluations of cancer immunotherapy.

METHODS

The individual patient data was reconstructed through published Checkmate 067 Kaplan Meier curves. Standard parametric models and six flexible techniques were tested, including fractional polynomial, restricted cubic splines, Royston-Parmar models, generalized additive models, parametric mixture models, and mixture cure models. Mean square errors (MSE) and bias from raw survival plots were used to test the model fitness and extrapolation performance. Variability of estimated incremental cost-effectiveness ratios (ICERs) from different models was used to inform the structural uncertainty in economic evaluations. All indicators were analyzed and compared under cut-offs of 3 years and 6.5 years, respectively, to further discuss model impact under different data maturity. R Codes for reproducing this study can be found on GitHub.

RESULTS

The flexible techniques in general performed better than standard parametric models with smaller MSE irrespective of the data maturity. Survival outcomes projected by long-term extrapolation using immature data differed from those with mature data. Although a best-performing model was not found because several models had very similar MSE in this case, the variability of modeled ICERs significantly increased when prolonging simulation cycles.

CONCLUSIONS

Flexible techniques show better performance in the case of Checkmate 067, regardless of data maturity. Model choices affect ICERs of cancer immunotherapy, especially when dealing with immature survival data. When researchers lack evidence to identify the 'right' model, we recommend identifying and revealing the model impacts on structural uncertainty.

Blended Survival Curves: A New Approach to Extrapolation for Time-to-Event Outcomes from Clinical Trials in Health Technology Assessment

BACKGROUND

Survival extrapolation is essential in cost-effectiveness analysis to quantify the lifetime survival benefit associated with a new intervention, due to the restricted duration of randomized controlled trials (RCTs). Current approaches of extrapolation often assume that the treatment effect observed in the trial can continue indefinitely, which is unrealistic and may have a huge impact on decisions for resource allocation.

OBJECTIVE

We introduce a novel methodology as a possible solution to alleviate the problem of survival extrapolation with heavily censored data from clinical trials.

METHOD

The main idea is to mix a flexible model (e.g., Cox semiparametric) to fit as well as possible the observed data and a parametric model encoding assumptions on the expected behavior of underlying long-term survival. The two are "blended" into a single survival curve that is identical with the Cox model over the range of observed times and gradually approaching the parametric model over the extrapolation period based on a weight function. The weight function regulates the way two survival curves are blended, determining how the internal and external sources contribute to the estimated survival over time.

RESULTS

A 4-y follow-up RCT of rituximab in combination with fludarabine and cyclophosphamide versus fludarabine and cyclophosphamide alone for the first-line treatment of chronic lymphocytic leukemia is used to illustrate the method.

CONCLUSION

Long-term extrapolation from immature trial data may lead to significantly different estimates with various modelling assumptions. The blending approach provides sufficient flexibility, allowing a wide range of plausible scenarios to be considered as well as the inclusion of external information, based, for example, on hard data or expert opinion. Both internal and external validity can be carefully examined.

HIGHLIGHTS

Interim analyses of trials with limited follow-up are often subject to high degrees of administrative censoring, which may result in implausible long-term extrapolations using standard approaches.In this article, we present an innovative methodology based on "blending" survival curves to relax the traditional proportional hazard assumption and simultaneously incorporate external information to guide the extrapolation.The blended method provides a simple and powerful framework to allow a careful consideration of a wide range of plausible scenarios, accounting for model fit to the short-term data as well as the plausibility of long-term extrapolations.

A Guide to Selecting Flexible Survival Models to Inform Economic Evaluations of Cancer Immunotherapies

OBJECTIVES

Parametric models are routinely used to estimate the benefit of cancer drugs beyond trial follow-up. The advent of immune checkpoint inhibitors has challenged this paradigm, and emerging evidence suggests that more flexible survival models, which can better capture the shapes of complex hazard functions, might be needed for these interventions. Nevertheless, there is a need for an algorithm to help analysts decide whether flexible models are required and, if so, which should be chosen for testing. This position article has been produced to bridge this gap.

METHODS

A virtual advisory board comprising 7 international experts with in-depth knowledge of survival analysis and health technology assessment was held in summer 2021. The experts discussed 24 questions across 6 topics: the current survival model selection procedure, data maturity, heterogeneity of treatment effect, cure and mortality, external evidence, and additions to existing guidelines. Their responses culminated in an algorithm to inform selection of flexible survival models.

RESULTS

The algorithm consists of 8 steps and 4 questions. Key elements include the systematic identification of relevant external data, using clinical expert input at multiple points in the selection process, considering the future and the observed hazard functions, assessing the potential for long-term survivorship, and presenting results from all plausible models.

CONCLUSIONS

This algorithm provides a systematic, evidence-based approach to justify the selection of survival extrapolation models for cancer immunotherapies. If followed, it should reduce the risk of selecting inappropriate models, partially addressing a key area of uncertainty in the economic evaluation of these agents.

Dynamic and Flexible Survival Models for Extrapolation of Relative Survival: A Case Study and Simulation Study

BACKGROUND

Extrapolation of survival data is a key task in health technology assessments (HTAs), which may be improved by incorporating general population mortality data via relative survival models. Dynamic survival models are a promising method for extrapolation that may be expanded to dynamic relative survival models (DRSMs), a novel development presented here. There are currently neither examples of dynamic models in HTA nor comparisons of DRSMs with other relative survival models when used for survival extrapolation.

METHODS

An existing appraisal, for which there had been disagreement over the approach to survival extrapolation, was chosen and the health economic model recreated. The sensitivity of estimates of cost-effectiveness to different model choices (standard survival models, DSMs, and DRSMs) and specifications was examined. The appraisal informed a simulation study to evaluate DRSMs with relative survival models based on both standard and spline-based (flexible) models.

RESULTS

Dynamic models provided insight into the behavior of the trend in the hazard function and how it may vary during the extrapolated phase. DRSMs led to extrapolations with improved plausibility for which model choice may be based on clinical input. In the simulation study, the flexible and dynamic relative survival models performed similarly and provided highly variable extrapolations.

LIMITATIONS

Further experience with these models is required to identify settings when they are most useful, and they provide sufficiently accurate extrapolations.

CONCLUSIONS

Dynamic models provide a flexible and attractive method for extrapolating survival data and facilitate the use of clinical input for model choice. Flexible and dynamic relative survival models make few structural assumptions and can improve extrapolation plausibility, but further research is required into methods for reducing the variability in extrapolations.

Methods for Extrapolating Survival Analyses for the Economic Evaluation of Advanced Therapy Medicinal Products

There are two significant challenges for analysts conducting economic evaluations of advanced therapy medicinal products (ATMPs): (i) estimating long-term treatment effects in the absence of mature clinical data, and (ii) capturing potentially complex hazard functions. This review identifies and critiques a variety of methods that can be used to overcome these challenges. The narrative review is informed by a rapid literature review of methods used for the extrapolation of survival analyses in the economic evaluation of ATMPs. There are several methods that are more suitable than traditional parametric survival modelling approaches for capturing complex hazard functions, including, cure-mixture models and restricted cubic spline models. In the absence of mature clinical data, analysts may augment clinical trial data with data from other sources to aid extrapolation, however, the relative merits of employing methods for including data from different sources is not well understood. Given the high and potentially irrecoverable costs of making incorrect decisions concerning the reimbursement or commissioning of ATMPs, it is important that economic evaluations are correctly specified, and that both parameter and structural uncertainty associated with survival extrapolations are considered. Value of information analyses allow for this uncertainty to be expressed explicitly, and in monetary terms.

Incorporating external trial data to improve survival extrapolations: a pilot study of the COU-AA-301 trial

Survival extrapolation plays a key role within cost effectiveness analysis and is often subject to substantial uncertainty. Use of external data to improve extrapolations has been identified as a key research priority. We present findings from a pilot study using data from the COU-AA-301 trial of abiraterone acetate for metastatic castration-resistant prostate cancer, to explore how external trial data may be incorporated into survival extrapolations. External trial data were identified via a targeted search of technology assessment reports. Four methods using external data were compared to simple parametric models (SPMs): informal reference to external data to select appropriate SPMs, piecewise models with, and without, hazard ratio adjustment, and Bayesian models fitted with a prior on the shape parameter(s). Survival and hazard plots were compared, and summary metrics (point estimate accuracy and restricted mean survival time) were calculated. Without consideration of external data, several SPMs may have been selected as the ‘best-fitting’ model. The range of survival probability estimates was generally reduced when external data were included in model estimation, and external hazard plots aided model selection. Different methods yielded varied results, even with the same data source, highlighting potential issues when integrating external trial data within model estimation. By using external trial data, the most (in)appropriate models may be more easily identified. However, benefits of using external data are contingent upon their applicability to the research question, and the choice of method can have a large impact on extrapolations.

Assessment of Treatment Effects and Long-term Benefits in Immune Checkpoint Inhibitor Trials Using the Flexible Parametric Cure Model: A Systematic Review

IMPORTANCE

Compared with standard cytotoxic therapies, randomized immune checkpoint inhibitor (ICI) phase 3 trials reveal delayed benefits in terms of patient survival and/or long-term response. Such outcomes generally violate the assumption of proportional hazards, and the classical Cox proportional hazards regression model is therefore unsuitable for these types of analyses.

OBJECTIVE

To evaluate the ability of the flexible parametric cure model (FPCM) to estimate treatment effects and long-term responder fractions (LRFs) independently of prespecified time points.

EVIDENCE REVIEW

This systematic review used reconstructed individual patient data from ICI advanced or metastatic melanoma and lung cancer phase 3 trials extracted from the literature. Trials published between January 1, 2010, and October 1, 2019, with long-term follow-up periods (maximum follow-up, ≥36 months in first line and ≥30 months otherwise) were selected to identify LRFs. Individual patient data for progression-free survival were reconstructed from the published randomized ICI phase 3 trial results. The FPCM was applied to estimate treatment effects on the overall population and on the following components of the population: LRF and progression-free survival in non-long-term responders. Results obtained were compared with treatment effects estimated using the Cox proportional hazards regression model.

FINDINGS

In this systematic review, among the 23 comparisons studied using the FPCM, a statistically significant association between the time-to-event component and experimental treatment was observed in the main analyses and confirmed in the sensitivity analyses of 18 comparisons. Results were discordant for 4 comparisons that were not significant by the Cox proportional hazards regression model. The LRFs varied from 1.5% to 12.7% for the control arms and from 4.6% to 38.8% for the experimental arms. Differences in LRFs varied from 2% to 29% and were significantly increased in the experimental compared with the control arms, except for 4 comparisons.

CONCLUSIONS AND RELEVANCE

This systematic review of reconstructed individual patient data found that the FPCM was a complementary approach that provided a comprehensive and pertinent evaluation of benefit and risk by assessing whether ICI treatment was associated with an increased probability of patients being long-term responders or with an improved progression-free survival in patients who were not long-term responders.

Comparing current and emerging practice models for the extrapolation of survival data: a simulation study and case-study

BACKGROUND

Estimates of future survival can be a key evidence source when deciding if a medical treatment should be funded. Current practice is to use standard parametric models for generating extrapolations. Several emerging, more flexible, survival models are available which can provide improved within-sample fit. This study aimed to assess if these emerging practice models also provided improved extrapolations.

METHODS

Both a simulation study and a case-study were used to assess the goodness of fit of five classes of survival model. These were: current practice models, Royston Parmar models (RPMs), Fractional polynomials (FPs), Generalised additive models (GAMs), and Dynamic survival models (DSMs). The simulation study used a mixture-Weibull model as the data-generating mechanism with varying lengths of follow-up and sample sizes. The case-study was long-term follow-up of a prostate cancer trial. For both studies, models were fit to an early data-cut of the data, and extrapolations compared to the known long-term follow-up.

RESULTS

The emerging practice models provided better within-sample fit than current practice models. For data-rich simulation scenarios (large sample sizes or long follow-up), the GAMs and DSMs provided improved extrapolations compared with current practice. Extrapolations from FPs were always very poor whilst those from RPMs were similar to current practice. With short follow-up all the models struggled to provide useful extrapolations. In the case-study all the models provided very similar estimates, but extrapolations were all poor as no model was able to capture a turning-point during the extrapolated period.

CONCLUSIONS

Good within-sample fit does not guarantee good extrapolation performance. Both GAMs and DSMs may be considered as candidate extrapolation models in addition to current practice. Further research into when these flexible models are most useful, and the role of external evidence to improve extrapolations is required.