Exploring structural uncertainty in model-based economic evaluations

A systematic review of the current application status of decision-analytical models in the pharmacoeconomic evaluation of targeted therapies for pulmonary arterial hypertension

BACKGROUND

The implementation of targeted drug therapy results in a significant improvement in both survival rates and quality of life among patients diagnosed with pulmonary arterial hypertension (PAH), concurrently imposing a greater financial burden on them. The use of pharmacoeconomic evaluation based on decision-analytical models is extensively employed in the rational allocation of healthcare resources.

OBJECTIVES

The present study conducted a systematic review of the literature on the pharmacoeconomic evaluation of drugs for treating PAH, with a focus on summarizing the composition and sources of parameters in decision-analytical models. This study aims to provide methodological guidance for future economic research.

METHODS

The review was conducted across six databases (PubMed, Embase, the Cochrane Library, CNKI, VIP, WanFang Data) and two health technology assessment agency websites (NHS EED, INAHTA). The characteristics of each study and the compositional details of the decision-analytical models are extracted.

RESULTS

In total, 13 published studies were included. The pharmacoeconomic evaluation methods employed in the studies included cost-effectiveness analysis (CEA) and cost-utility analysis (CUA). The decision analysis models employed in all 13 studies were Markov models. The models were all constructed on the basis of the World Health Organization (WHO) functional class, with variations in parameter settings and sources.

CONCLUSIONS

All 13 Markov models provided useful insight into PAH modeling. Future research in this field can employ these research methods according to diverse research objectives. The utility values were derived from a single source; therefore, future studies should evaluate the quality of life in patients with PAH across varying disease severities.

Unravelling the Association Between Uncertainties in Model-based Economic Analysis and Funding Recommendations of Medicines in Australia

OBJECTIVE

Health technology assessment is used extensively by the Pharmaceutical Benefits Advisory Committee (PBAC) to inform medicine funding recommendations in Australia. The PBAC often does not recommend medicines due to uncertainties in economic modelling that result in delaying access to medicines for patients. The systematic identification of which uncertainties can be reduced with alternative evidence or the collection of additional data can help inform recommendations. This study aims to characterise different types of uncertainty in economic models and empirically assess their association with the PBAC recommendations.

METHODS

A framework was developed to characterise four types of uncertainties: methodological, structural, generalisability and parameter uncertainty. The first two types were further subcategorised into parameterisable and unparameterisable uncertainty. Data on uncertainty and other factors were extracted from PBAC's Public Summary Documents of first submissions for 193 medicine (vaccine)-indication pairs including economic modelling between 2014 and 2021. Logistic regression was used to estimate the average marginal effect of each type of uncertainty on the probability of a positive recommendation.

RESULTS

The PBAC more often raised issues regarding parameter uncertainty (95%) and parameterisable structural uncertainty (83%) than generalisability uncertainty (48%) and unparameterisable methodological uncertainty (56%). The logistic regression results suggested that the PBAC was more likely to recommend a medicine without unparameterisable methodological, generalisability, and parameterisable structural uncertainty by 15.0%, 10.2 %, and 17.6%, respectively. Parameterisable methodological, unparameterisable structural and parameter uncertainty were not significantly associated with the PBAC recommendations.

CONCLUSIONS

This study identified the uncertainties that had significant associations with PBAC recommendations based on the first submission. This may help improve model quality and reduce resubmissions in the future, thus improving patients' access to medicines.

Bring Out Your Dead: A Review of the Cost Minimisation Approach in Health Technology Assessment Submissions to the Australian Pharmaceutical Benefits Advisory Committee

OBJECTIVES

Published literature has levied criticism against the cost-minimisation analysis (CMA) approach to economic evaluation over the past two decades, with multiple papers declaring its 'death'. However, since introducing the requirements for economic evaluations as part of health technology (HTA) decision-making in 1992, the cost-minimisation analysis (CMA) approach has been widely used to inform recommendations about the public subsidy of medicines in Australia. This research aimed to highlight the breadth of use of CMA in Australia and assess the influence of preconditions for the approach on subsidy recommendations METHODS: Relevant information was extracted from Public Summary Documents of Pharmaceutical Benefits Advisory Committee (PBAC) meetings in Australia considering submissions for the subsidy of medicines that included a CMA and were assessed between July 2005 and December 2022. A generalised linear model was used to explore the relationship between whether medicines were recommended and variables that reflected the primary preconditions for using CMA set out in the published PBAC Methodology Guidelines. Other control variables were selected through the Bolasso Method. Subgroup analysis was undertaken which replicated this modelling process.

RESULTS

While the potential for inferior safety or efficacy reduced the likelihood of recommendation (p < 0.01), the effect sizes suggest that the requirements for CMA were not requisite for recommendation.

CONCLUSION

The Australian practice of CMA does not strictly align with the PBAC Methodology Guidelines and the theoretically appropriate application of CMA. However, within the confines of a deliberative HTA decision-making process that balances values and judgement with available evidence, this may be considered acceptable, particularly if stakeholders consider the current approach delivers sufficient clarity of process and enables patients to access medicines at an affordable cost.

Examining the Impact of Structural Uncertainty Across 10 Type 2 Diabetes Models: Results From the 2022 Mount Hood Challenge

OBJECTIVES

The Mount Hood Diabetes Challenge Network aimed to examine the impact of model structural uncertainty on the estimated cost-effectiveness of interventions for type 2 diabetes.

METHODS

Ten independent modeling groups completed a blinded simulation exercise to estimate the cost-effectiveness of 3 interventions in 2 type 2 diabetes populations. Modeling groups were provided with a common baseline population, cost and utility values associated with different model health states, and instructions regarding time horizon and discounting. We collated the results to identify variation in predictions of net monetary benefit (NMB) and the drivers of those differences.

RESULTS

Overall, modeling groups agreed which interventions had a positive NMB (ie, were cost-effective), Although estimates of NMB varied substantially-by up to £23 696 for 1 intervention. Variation was mainly driven through differences in risk equations for complications of diabetes and their implementation between models. The number of modeled health states was also a significant predictor of NMB.

CONCLUSIONS

This exercise demonstrates that structural uncertainty between different health economic models affects cost-effectiveness estimates. Although it is reassuring that a decision maker would likely reach similar conclusions on which interventions were cost-effective using most models, the range in numerical estimates generated across different models would nevertheless be important for price-setting negotiations with intervention developers. Minimizing the impact of structural uncertainty on healthcare decision making therefore remains an important priority. Model registries, which record and compare the impact of structural assumptions, offer one potential avenue to improve confidence in the robustness of health economic modeling.

Impact of Structural Differences on the Modeled Cost-Effectiveness of Noninvasive Prenatal Testing

BACKGROUND

Noninvasive prenatal testing (NIPT) was developed to improve the accuracy of prenatal screening to detect chromosomal abnormalities. Published economic analyses have yielded different incremental cost-effective ratios (ICERs), leading to conclusions of NIPT being dominant, cost-effective, and cost-ineffective. These analyses have used different model structures, and the extent to which these structural variations have contributed to differences in ICERs is unclear.

AIM

To assess the impact of different model structures on the cost-effectiveness of NIPT for the detection of trisomy 21 (T21; Down syndrome).

METHODS

A systematic review identified economic models comparing NIPT to conventional screening. The key variations in identified model structures were the number of health states and modeling approach. New models with different structures were developed in TreeAge and populated with consistent parameters to enable a comparison of the impact of selected structural variations on results.

RESULTS

The review identified 34 economic models. Based on these findings, demonstration models were developed: 1) a decision tree with 3 health states, 2) a decision tree with 5 health states, 3) a microsimulation with 3 health states, and 4) a microsimulation with 5 health states. The base-case ICER from each model was 1) USD$34,474 (2023)/quality-adjusted life-year (QALY), 2) USD$14,990 (2023)/QALY, (3) USD$54,983 (2023)/QALY, and (4) NIPT was dominated.

CONCLUSION

Model-structuring choices can have a large impact on the ICER and conclusions regarding cost-effectiveness, which may inadvertently affect policy decisions to support or not support funding for NIPT. The use of reference models could improve international consistency in health policy decision making for prenatal screening.

HIGHLIGHTS

NIPT is a clinical area in which a variety of modeling approaches have been published, with wide variation in reported cost-effectiveness.This study shows that when broader contextual factors are held constant, varying the model structure yields results that range from NIPT being less effective and more expensive than conventional screening (i.e., NIPT was dominated) through to NIPT being more effective and more expensive than conventional screening with an ICER of USD$54,983 (2023)/QALY.Model-structuring choices may inadvertently affect policy decisions to support or not support funding of NIPT. Reference models could improve international consistency in health policy decision making for prenatal screening.

A Systematic Review of Methodologies Used in Models of the Treatment of Diabetes Mellitus

BACKGROUND

Diabetes mellitus is a chronic and complex disease, increasing in prevalence and consequent health expenditure. Cost-effectiveness models with long time horizons are commonly used to perform economic evaluations of diabetes' treatments. As such, prediction accuracy and structural uncertainty are important features in cost-effectiveness models of chronic conditions.

OBJECTIVES

The aim of this systematic review is to identify and review published cost-effectiveness models of diabetes treatments developed between 2011 and 2022 regarding their methodological characteristics. Further, it also appraises the quality of the methods used, and discusses opportunities for further methodological research.

METHODS

A systematic literature review was conducted in MEDLINE and Embase to identify peer-reviewed papers reporting cost-effectiveness models of diabetes treatments, with time horizons of more than 5 years, published in English between 1 January 2011 and 31 of December 2022. Screening, full-text inclusion, data extraction, quality assessment and data synthesis using narrative synthesis were performed. The Philips checklist was used for quality assessment of the included studies. The study was registered in PROSPERO (CRD42021248999).

RESULTS

The literature search identified 30 studies presenting 29 unique cost-effectiveness models of type 1 and/or type 2 diabetes treatments. The review identified 26 type 2 diabetes mellitus (T2DM) models, 3 type 1 DM (T1DM) models and one model for both types of diabetes. Fifteen models were patient-level models, whereas 14 were at cohort level. Parameter uncertainty was assessed thoroughly in most of the models, whereas structural uncertainty was seldom addressed. All the models where validation was conducted performed well. The methodological quality of the models with respect to structure was high, whereas with respect to data modelling it was moderate.

CONCLUSIONS

Models developed in the past 12 years for health economic evaluations of diabetes treatments are of high-quality and make use of advanced methods. However, further developments are needed to improve the statistical modelling component of cost-effectiveness models and to provide better assessment of structural uncertainty.

How Sensitive is Sensitivity Analysis?: Evaluation of Pharmacoeconomic Submissions in Korea

Purpose: We aimed to describe the types of uncertainties examined in the economic evaluations submitted for reimbursement in Korea and their impact on the incremental cost-effectiveness ratio (ICER).

Method: Fifty dossiers were submitted by pharmaceutical companies to the economic subcommittee of the Pharmaceutical Benefit Coverage Advisory Committee (PBCAC) from January 2014 to December 2018. The types of uncertainties were categorized as structural and parametric, and the frequencies of the sensitivity analysis per variables were analyzed. The impact of uncertainties was measured by the percent variance of the ICER relative to that of the base case analysis.

Results: Of the 50 submissions, varying discount rate (44 submissions), followed by time horizon (38 submissions) and model assumptions (29 submissions), were most frequently used to examine structural uncertainty, while utility (42 submissions), resource use (41 submissions), and relative effectiveness (26 submissions) were used to examine parametric uncertainty. A total of 1,236 scenarios (a scenario corresponds to a case where a single variable is varied by a single range) were presented in the one-way sensitivity analyses, where parametric and structural sensitivity analyses comprised 679 and 557 scenarios, respectively. Varying drug prices had the highest impact on ICER (median variance 19.9%), followed by discount rate (12.2%), model assumptions (11.9%), extrapolation (11.8%), and time horizon (10.0%).

Conclusions: Variables related to long-term assumptions, such as model assumptions, time horizon, extrapolation, and discounting rate, were related to a high level of uncertainty. Caution should be exercised when using immature data.

Calibrating Natural History of Cancer Models in the Presence of Data Incompatibility: Problems and Solutions

The calibration of cancer natural history models is often challenged by a lack of representative calibration targets, forcing modellers to rely on potentially incompatible datasets. Using a microsimulation colorectal cancer model as an example, the purposes of this paper are to (1) highlight the reasons for uncertainty in calibration targets, (2) illustrate practical and generalisable approaches for dealing with incompatibility in calibration targets, and (3) discuss the importance of future research in the area of incorporating uncertainty in calibration. The low quality of data and differences in populations, outcome definitions, and healthcare systems may result in incompatibility between the model and the data. Acknowledging reasons for data incompatibility allows assessment of the risk of incompatibility before calibrating the model. Only a few approaches are available to address data incompatibility, for instance addressing biases in calibration targets and their adjustment, relaxing the goodness-of-fit metric, and validation of the calibration targets to the data not used in the calibration. However, these approaches lack explicit comparison and validation, and so more research is needed to describe the nature and causes of indirect uncertainty (i.e. uncertainty that cannot be expressed in absolute quantitative forms) and identify methods for managing this uncertainty in healthcare modelling.

A Review of Methodological Considerations for Economic Evaluations of Gene Therapies and Their Application in Literature

OBJECTIVES

To identify methodological considerations discussed in literature addressing economic evaluations (EEs) of gene therapies (GTs). Additionally, we assessed if these considerations are applied in published GT EEs to increase understanding and explore impact.

METHODS

First a peer-reviewed literature review was performed to identify research addressing methodological considerations of GT EEs until August 2019. Identified considerations were grouped in themes using thematic content analysis. A second literature search was conducted in which we identified published evaluations. The EE quality of reporting was assessed using Consolidated Health Economic Evaluation Reporting Standards.

RESULTS

The first literature search yielded 13 articles discussing methodological considerations. The second search provided 12 EEs. Considerations identified were payment models, definition of perspectives, addressing uncertainty, data extrapolation, discount rates, novel value elements, and use of indirect and surrogate endpoints. All EEs scored satisfactory to good according to Consolidated Health Economic Evaluation Reporting Standards. Regarding methodological application, we found 1 methodological element (payment models) was applied in 2 base cases. Scenarios explored alternative perspectives, survival assumptions, and extrapolation methods in 10 EEs.

CONCLUSIONS

Although EE quality of reporting was considered good, their informativeness for health technology assessment and decision makers seemed limited owing to many uncertainties. We suggest accepted EE methods can broadly be applied to GTs, but few elements may need adjustment. Further research and multi-stakeholder consensus is needed to determine appropriateness and application of individual methodological considerations. For now, we recommend including scenario analyses to explore impact of methodological choices and (clinical) uncertainties. This study contributes to better understanding of perceived appropriate evaluation of GTs and informs best modeling practices.

Accurately Reflecting Uncertainty When Using Patient-Level Simulation Models to Extrapolate Clinical Trial Data

Introduction. Patient-level simulation models facilitate extrapolation of clinical trial data while allowing for heterogeneity, prior history, and nonlinearity. However, combining different types of uncertainty around within-trial and extrapolated results remains challenging. Methods. We tested 4 methods to combine parameter uncertainty (around the regression coefficients used to predict future events) with sampling uncertainty (uncertainty around mean risk factors within the finite sample whose outcomes are being predicted and the effect of treatment on these risk factors). We compared these 4 methods using a simulation study based on an economic evaluation extrapolating the AFORRD randomized controlled trial using the UK Prospective Diabetes Study Outcomes Model version 2. This established type 2 diabetes model predicts patient-level health outcomes and costs. Results. The 95% confidence intervals around life years gained gave 25% coverage when sampling uncertainty was excluded (i.e., 25% of 95% confidence intervals contained the “true” value). Allowing for sampling uncertainty as well as parameter uncertainty widened confidence intervals by 6.3-fold and gave 96.3% coverage. Methods adjusting for baseline risk factors that combine sampling and parameter uncertainty overcame the bias that can result from between-group baseline imbalance and gave confidence intervals around 50% wider than those just considering parameter uncertainty, with 99.8% coverage. Conclusions. Analyses extrapolating data for individual trial participants should include both sampling uncertainty and parameter uncertainty and should adjust for any imbalance in baseline covariates.

Multivariable and Structural Uncertainty Analyses for Cost-Effectiveness Estimates: Back to the Future

BACKGROUND

In this commentary, celebrating the 20th anniversary of the journal Value in Health, I present a brief overview and illustration of the evolution over the past 20 years of the methodological literature providing guidelines for multivariable and structural uncertainty analysis for cost-effectiveness estimates.

METHODS

To illustrate the impact of the guidelines for uncertainty analyses, I show how the inclusion of multivariable and structural uncertainty analyses in cost-effectiveness analyses published in Value in Health changed over the past 20 years using publications from 1999/2000, 2007 and 2017.

RESULTS

The commentary is organized in three sections: past, focusing on the development and use of methods for multivariable uncertainty analysis; present, focusing on the growing awareness of the need for structural uncertainty analysis, suggested frameworks for structural uncertainty analysis and how it is currently implemented; and future, considering different methods for combining multivariable and structural uncertainty analyses over the next decades.

CONCLUSIONS

I conclude by suggesting how the continued evolution of uncertainty analyses in published studies and health technology assessment submissions can best take into account an important goal of cost-effectiveness analyses: to provide useful information to decision makers.

Model Structuring for Economic Evaluations of New Health Technologies

In countries such as Australia, the UK and Canada, decisions on whether to fund new health technologies are commonly informed by decision analytic models. While the impact of making inappropriate structural choices/assumptions on model predictions is well noted, there is a lack of clarity about the definition of key structural aspects, the process of developing model structure (including the development of conceptual models) and uncertainty associated with the structuring process (structural uncertainty) in guidelines developed by national funding bodies. This forms the focus of this article. Building on the reports of good modelling practice, and recognising the fundamental role of model structuring within the model development process, we specified key structural choices and provided ideas about model structuring for the future direction. This will help to further standardise guidelines developed by national funding bodies, with potential impact on transparency, comprehensiveness and consistency of model structuring. We argue that the process of model structuring and structural sensitivity analysis should be documented in a more systematic and transparent way in submissions to national funding bodies. Within the decision-making process, the development of conceptual models and presentation of all key structural choices would mean that national funding bodies could be more confident of maximising value for money when making public funding decisions.

The Importance of Model Structure in the Cost-Effectiveness Analysis of Primary Care Interventions for the Management of Hypertension

BACKGROUND

Management of hypertension can lead to significant reductions in blood pressure, thereby reducing the risk of cardiovascular disease. Modeling the course of cardiovascular disease is not without complications, and uncertainty surrounding the structure of a model will almost always arise once a choice of a model structure is defined.

OBJECTIVES

To provide a practical illustration of the impact on the results of cost-effectiveness of changing or adapting model structures in a previously published cost-utility analysis of a primary care intervention for the management of hypertension Targets and Self-Management for the Control of Blood Pressure in Stroke and at Risk Groups (TASMIN-SR).

METHODS

The case study assessed the structural uncertainty arising from model structure and from the exclusion of secondary events. Four alternative model structures were implemented. Long-term cost-effectiveness was estimated and the results compared with those from the TASMIN-SR model.

RESULTS

The main cost-effectiveness results obtained in the TASMIN-SR study did not change with the implementation of alternative model structures. Choice of model type was limited to a cohort Markov model, and because of the lack of epidemiological data, only model 4 captured structural uncertainty arising from the exclusion of secondary events in the case study model.

CONCLUSIONS

The results of this study indicate that the main conclusions drawn from the TASMIN-SR model of cost-effectiveness were robust to changes in model structure and the inclusion of secondary events. Even though one of the models produced results that were different to those of TASMIN-SR, the fact that the main conclusions were identical suggests that a more parsimonious model may have sufficed.

Oncology Modeling for Fun and Profit! Key Steps for Busy Analysts in Health Technology Assessment

In evaluating new oncology medicines, two common modeling approaches are state transition (e.g., Markov and semi-Markov) and partitioned survival. Partitioned survival models have become more prominent in oncology health technology assessment processes in recent years. Our experience in conducting and evaluating models for economic evaluation has highlighted many important and practical pitfalls. As there is little guidance available on best practices for those who wish to conduct them, we provide guidance in the form of 'Key steps for busy analysts,' who may have very little time and require highly favorable results. Our guidance highlights the continued need for rigorous conduct and transparent reporting of economic evaluations regardless of the modeling approach taken, and the importance of modeling that better reflects reality, which includes better approaches to considering plausibility, estimating relative treatment effects, dealing with post-progression effects, and appropriate characterization of the uncertainty from modeling itself.

Exploring Uncertainty in Economic Evaluations of Drugs and Medical Devices: Lessons from the First Review of Manufacturers' Submissions to the French National Authority for Health

OBJECTIVES

The objective of this paper was to evaluate how uncertainty has been accounted for in the cost-effectiveness analyses (CEAs) submitted by manufacturers to the French National Authority for Health (HAS) and to identify recurring concerns in these submissions.

METHODS

We used a cross-sectional design to evaluate manufacturers' submissions from the beginning of the evaluation process in October 2013 to the end of May 2015 (n = 28). The sources of uncertainty attached to these CEAs were categorized and assessed. Relevant data were extracted independently by two assessors.

RESULTS

Adherence to the HAS reference case was generally considered to be acceptable. Methodological uncertainty and parameter uncertainty were the sources of uncertainty that were most frequently explored by manufacturers. The quality of reporting of deterministic sensitivity analysis and probabilistic sensitivity analysis varied substantially across submissions, with a frequent lack of justification of the plausible range of parameter point estimates in 12 submissions (43 %). Structural uncertainty was explored much less frequently. Concerns related to omission of either important clinical events or relevant health states or extrapolation of the effects of the technology beyond the time horizon of the clinical trials were identified in 16 submissions (57 %).

CONCLUSIONS

This study presented a characterization of the treatment of uncertainty for the first 28 manufacturers' submissions to the HAS. This work identified important concerns regarding the exploration of sources of uncertainty. The findings may help manufacturers to improve the quality of their submissions and may provide useful insights for extending guidelines on uncertainty analysis in CEAs submitted to the HAS.

Value of Information Analysis Applied to the Economic Evaluation of Interventions Aimed at Reducing Juvenile Delinquency: An Illustration

Objectives

To investigate whether a value of information analysis, commonly applied in health care evaluations, is feasible and meaningful in the field of crime prevention.

Methods

Interventions aimed at reducing juvenile delinquency are increasingly being evaluated according to their cost-effectiveness. Results of cost-effectiveness models are subject to uncertainty in their cost and effect estimates. Further research can reduce that parameter uncertainty. The value of such further research can be estimated using a value of information analysis, as illustrated in the current study. We built upon an earlier published cost-effectiveness model that demonstrated the comparison of two interventions aimed at reducing juvenile delinquency. Outcomes were presented as costs per criminal activity free year.

Results

At a societal willingness-to-pay of €71,700 per criminal activity free year, further research to eliminate parameter uncertainty was valued at €176 million. Therefore, in this illustrative analysis, the value of information analysis determined that society should be willing to spend a maximum of €176 million in reducing decision uncertainty in the cost-effectiveness of the two interventions. Moreover, the results suggest that reducing uncertainty in some specific model parameters might be more valuable than in others.

Conclusions

Using a value of information framework to assess the value of conducting further research in the field of crime prevention proved to be feasible. The results were meaningful and can be interpreted according to health care evaluation studies. This analysis can be helpful in justifying additional research funds to further inform the reimbursement decision in regard to interventions for juvenile delinquents.