Decision-analytic models: current methodological challenges

A Systematic Review of Modeling Approaches to Evaluate Treatments for Relapsed Refractory Multiple Myeloma: Critical Review and Considerations for Future Health Economic Models

BACKGROUND AND OBJECTIVE

Multiple myeloma is a rare incurable hematological cancer in which most patients relapse or become refractory to treatment. This systematic literature review aimed to critically review the existing economic models used in economic evaluations of systemic treatments for relapsed/refractory multiple myeloma and to summarize how the models addressed differences in the line of therapy and exposure to prior treatment.

METHODS

Following a pre-approved protocol, literature searches were conducted on 17 February, 2023, in relevant databases for models published since 2014. Additionally, key health technology assessment agency websites were manually searched for models published as part of submission dossiers since 2018. Reported information related to model conceptualization, structure, uncertainty, validation, and transparency were extracted into a pre-defined extraction sheet.

RESULTS

In total, 49 models assessing a wide range of interventions across multiple lines of therapy were included. Only five models specific to heavily pre-treated patients and/or those who were refractory to multiple treatment classes were identified. Most models followed a conventional simple methodology, such as partitioned survival (n = 28) or Markov models (n = 9). All included models evaluated specific interventions rather than the whole treatment sequence. Where subsequent therapies were included in the model, these were generally only considered from a cost and resource use perspective. The models generally used overall and progression-free survival as model inputs, although data were often immature. Sensitivity analyses were frequently reported (n = 41) whereas validation was only considered in less than half (n = 19) of the models.

CONCLUSIONS

Published economic models in relapsed/refractory multiple myeloma rarely followed an individual patient approach, mainly owing to the higher need for complex data assumptions compared with simpler modeling approaches. As many patients experience disease progression on multiple treatment lines, there is a growing need for modeling complex treatment strategies, leading to more sophisticated approaches in the future. Maintaining transparency, high reporting standards, and thorough analyses of uncertainty are crucial to support these advancements.

Critical Examination of Modeling Approaches Used in Economic Evaluations of First-Line Treatments for Locally Advanced or Metastatic Non-Small Cell Lung Cancer Harboring Epidermal Growth Factor Receptor Mutations: A Systematic Literature Review

BACKGROUND

Non-small cell lung cancer (NSCLC) is the most common type of lung cancer, with up to 32% of patients with NSCLC harboring an epidermal growth factor receptor (EGFR) mutation. NSCLC harboring an EGFR mutation has a dedicated treatment pathway, with EGFR tyrosine kinase inhibitors and platinum-based chemotherapy often being the therapy of choice.

OBJECTIVE

The aim of this study was to systemically review and summarize economic models of first-line treatments used for locally advanced or metastatic NSCLC harboring EGFR mutations, as well as to identify areas for improvement for future models.

METHODS

Literature searches were conducted via Ovid in PubMed, MEDLINE, MEDLINE In-Process, Embase, Evidence-Based Medicine Reviews: Health Technology Assessment, Evidence-Based Medicine Reviews: National Health Service Economic Evaluation Database, and EconLit. An initial search was conducted on 19 December 2022 and updated on 11 April 2023. Studies were selected according to predefined criteria using the Population, Intervention, Comparator, Outcome and Study design (PICOS) framework.

RESULTS

Sixty-seven articles were included in the review, representing 59 unique studies. The majority of included models were cost-utility analyses (n = 52), with the remaining studies being cost-effectiveness analyses (n = 4) and a cost-minimization analysis (n = 1). Two studies incorporated both a cost-utility and cost-minimization analysis. Although the model structure across studies was consistently reported, justification for this choice was often lacking.

CONCLUSIONS

Although the reporting of economic models in NSCLC harboring EGFR mutations is generally good, many of these studies lacked sufficient reporting of justification for structural choices, performing extensive sensitivity analyses and validation in economic evaluations. In resolving such gaps, the validity of future models can be increased to guide healthcare decision making in rare indications.

De Novo Cost-Effectiveness Model Framework for Nonalcoholic Steatohepatitis-Modeling Approach and Validation

BACKGROUND

Nonalcoholic steatohepatitis (NASH) is a chronic liver disease associated with hepatic morbidity and mortality and extra-hepatic comorbidities. Published NASH cost-effectiveness models (CEMs) are heterogeneous and consistently omit comorbid conditions that frequently co-exist alongside NASH. We aimed to develop a de novo CEM framework that incorporates extra-hepatic disease states and outcomes alongside hepatic components to enable future estimation of the cost-effectiveness of NASH interventions.

METHODS

Patient-level simulation and cohort-level Markov models were implemented in the same framework. Model inputs included fibrosis progression; late-stage liver disease outcomes; comorbidity outcomes for cardiovascular disease, type 2 diabetes, and obesity; mortality; health-related quality of life; and direct medical costs. The prototype analysis assessed the cost-effectiveness of obeticholic acid versus standard of care from a US payer perspective over a lifetime horizon with costs and effects discounted at 3% per annum. However, the CEM was designed for easy adaptation to other countries, time horizons, and other considerations. Efficacy and adverse event parameters were obtained from the 18-month interim analysis of the REGENERATE trial. Outputs include total and incremental costs, total life years, and quality-adjusted life years.

RESULTS

In this model, total costs, total life years, and quality-adjusted life years were all higher with obeticholic acid compared with standard of care. Cross-validation of this model with the 2016 and 2020 Institute for Clinical and Economic Review models revealed marked differences, mainly driven by mortality inputs, transition probability estimates, and incorporation of the effect of treatment and comorbidities.

CONCLUSION

This is the first CEM in NASH to incorporate the clinical consequences of several comorbidities. The flexible yet standardized framework permits estimation of the cost-effectiveness of NASH interventions in a variety of settings. The model currently includes several assumptions and will be further developed as more relevant data become available.

Technical/Algorithm, Stakeholder, and Society (TASS) barriers to the application of artificial intelligence in medicine: A systematic review

INTRODUCTION

The use of artificial intelligence (AI), particularly machine learning and predictive analytics, has shown great promise in health care. Despite its strong potential, there has been limited use in health care settings. In this systematic review, we aim to determine the main barriers to successful implementation of AI in healthcare and discuss potential ways to overcome these challenges.

METHODS

We conducted a literature search in PubMed (1/1/2001-1/1/2023). The search was restricted to publications in the English language, and human study subjects. We excluded articles that did not discuss AI, machine learning, predictive analytics, and barriers to the use of these techniques in health care. Using grounded theory methodology, we abstracted concepts to identify major barriers to AI use in medicine.

RESULTS

We identified a total of 2,382 articles. After reviewing the 306 included papers, we developed 19 major themes, which we categorized into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). These themes included: Lack of Explainability, Need for Validation Protocols, Need for Standards for Interoperability, Need for Reporting Guidelines, Need for Standardization of Performance Metrics, Lack of Plan for Updating Algorithm, Job Loss, Skills Loss, Workflow Challenges, Loss of Patient Autonomy and Consent, Disturbing the Patient-Clinician Relationship, Lack of Trust in AI, Logistical Challenges, Lack of strategic plan, Lack of Cost-effectiveness Analysis and Proof of Efficacy, Privacy, Liability, Bias and Social Justice, and Education.

CONCLUSION

We identified 19 major barriers to the use of AI in healthcare and categorized them into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). Future studies should expand on barriers in pediatric care and focus on developing clearly defined protocols to overcome these barriers.

Implementing competing risks in discrete event simulation: the event-specific probabilities and distributions approach

Background: Although several strategies for modelling competing events in discrete event simulation (DES) exist, a methodological gap for the event-specific probabilities and distributions (ESPD) approach when dealing with censored data remains. This study defines and illustrates the ESPD strategy for censored data. Methods: The ESPD approach assumes that events are generated through a two-step process. First, the type of event is selected according to some (unknown) mixture proportions. Next, the times of occurrence of the events are sampled from a corresponding survival distribution. Both of these steps can be modelled based on covariates. Performance was evaluated through a simulation study, considering sample size and levels of censoring. Additionally, an oncology-related case study was conducted to assess the ability to produce realistic results, and to demonstrate its implementation using both frequentist and Bayesian frameworks in R. Results: The simulation study showed good performance of the ESPD approach, with accuracy decreasing as sample sizes decreased and censoring levels increased. The average relative absolute error of the event probability (95%-confidence interval) ranged from 0.04 (0.00; 0.10) to 0.23 (0.01; 0.66) for 60% censoring and sample size 50, showing that increased censoring and decreased sample size resulted in lower accuracy. The approach yielded realistic results in the case study. Discussion: The ESPD approach can be used to model competing events in DES based on censored data. Further research is warranted to compare the approach to other modelling approaches for DES, and to evaluate its usefulness in estimating cumulative event incidences in a broader context.

Comparing Modeling Approaches for Discrete Event Simulations With Competing Risks Based on Censored Individual Patient Data: A Simulation Study and Illustration in Colorectal Cancer

OBJECTIVES

This study aimed to provide detailed guidance on modeling approaches for implementing competing events in discrete event simulations based on censored individual patient data (IPD).

METHODS

The event-specific distributions (ESDs) approach sampled times from event-specific time-to-event distributions and simulated the first event to occur. The unimodal distribution and regression approach sampled a time from a combined unimodal time-to-event distribution, representing all events, and used a (multinomial) logistic regression model to select the event to be simulated. A simulation study assessed performance in terms of relative absolute event incidence difference and relative entropy of time-to-event distributions for different types and levels of right censoring, numbers of events, distribution overlap, and sample sizes. Differences in cost-effectiveness estimates were illustrated in a colorectal cancer case study.

RESULTS

Increased levels of censoring negatively affected the modeling approaches' performance. A lower number of competing events and higher overlap of distributions improved performance. When IPD were censored at random times, ESD performed best. When censoring occurred owing to a maximum follow-up time for 2 events, ESD performed better for a low level of censoring (ie, 10%). For 3 or 4 competing events, ESD better represented the probabilities of events, whereas unimodal distribution and regression better represented the time to events. Differences in cost-effectiveness estimates, both compared with no censoring and between approaches, increased with increasing censoring levels.

CONCLUSIONS

Modelers should be aware of the different modeling approaches available and that selection between approaches may be informed by data characteristics. Performing and reporting extensive validation efforts remains essential to ensure IPD are appropriately represented.

Application of Discrete-Event Simulation in Health Technology Assessment: A Cost-Effectiveness Analysis of Alzheimer's Disease Treatment Using Real-World Evidence in Thailand

OBJECTIVES

Decision-analytic models for Alzheimer's disease (AD) have been advanced to a discrete-event simulation (DES), in which individual-level modeling of disease progression across continuous severity spectra become feasible. This study aimed to apply DES to perform cost-effectiveness analysis of AD treatment in Thailand.

METHODS

A data set of Thai AD patients, representing unique demographic and clinical characteristics, was bootstrapped to generate a baseline cohort of 50 000 patients. Each patient was cloned and assigned to donepezil, galantamine, rivastigmine, memantine, or no treatment. Correlated changes in cognitive and behavioral status over time were developed using patient-level data. Treatment effects were obtained from the most recent network meta-analysis. Treatment persistence; mortality; and predictive equations for functional status, costs (Thai baht in 2017), and quality-adjusted life-year (QALY) were derived from country-specific real-world data.

RESULTS

From a societal perspective, only the prescription of donepezil to AD patients with all disease-severity levels was found to be cost-effective (incremental cost-effectiveness ratio): 138 524 Thai baht/QALY ($4062/QALY)]. Regardless of whether the treatment-stopping rule when the mini-mental state examination score <10 was introduced, providing early treatment with donepezil to mild AD patients further reduced the incremental cost-effectiveness ratio. Extensive sensitivity analyses indicated robust simulation findings.

CONCLUSIONS

Discrete-event simulation greatly enhances the real-world representativeness of decision-analytic models for AD. Donepezil is the most cost-effective treatment option for AD in Thailand and is worth being considered for universal financial coverage. Application of DES in heath technology assessment should be encouraged, especially when the validity of the model is questionable with classical modeling methods.

Comparing Strategies for Modeling Competing Risks in Discrete-Event Simulations: A Simulation Study and Illustration in Colorectal Cancer

BACKGROUND

Different strategies toward implementing competing risks in discrete-event simulation (DES) models are available. This study aims to provide recommendations regarding modeling approaches that can be defined based on these strategies by performing a quantitative comparison of alternative modeling approaches.

METHODS

Four modeling approaches were defined: 1) event-specific distribution (ESD), 2) event-specific probability and distribution (ESPD), 3) unimodal joint distribution and regression model (UDR), and 4) multimodal joint distribution and regression model (MDR). Each modeling approach was applied to uncensored individual patient data in a simulation study and a case study in colorectal cancer. Their performance was assessed in terms of relative event incidence difference, relative absolute event incidence difference, and relative entropy of time-to-event distributions. Differences in health economic outcomes were also illustrated for the case study.

RESULTS

In the simulation study, the ESPD and MDR approaches outperformed the ESD and UDR approaches, in terms of both event incidence differences and relative entropy. Disease pathway and data characteristics, such as the number of competing risks and overlap between competing time-to-event distributions, substantially affected the approaches' performance. Although no considerable differences in health economic outcomes were observed, the case study showed that the ESPD approach was most sensitive to low event rates, which negatively affected performance.

CONCLUSIONS

Based on overall performance, the recommended modeling approach for implementing competing risks in DES models is the MDR approach, which is defined according to the general strategy of selecting the time-to-event first and the corresponding event second. The ESPD approach is a less complex and equally performing alternative if sufficient observations are available for each competing event (i.e., the internal validity shows appropriate data representation).

The value and consequences of using public health technology assessments for private payer decision-making in Canada: one size does not fit all

BACKGROUND

Both public and private insurers provide drug coverage in Canada. All payers are under pressure to contain costs. It has recently been proposed that private plans leverage the public health technology assessment (HTA) evaluation process in their decision-making.

OBJECTIVES

The objectives of the current study were to examine use of public health technology assessments (HTAs) for private payer decision-making in the literature, to gather the perspectives of experts from both public and private insurers on this practice, and to summarize which value parameters of public evaluations can be used for private payer decision-making.

METHODS

A targeted literature review was conducted to identify publications on the use of public HTA or cost-effectiveness data for private payer decision-making on pharmaceutical reimbursement. Concurrently, a roundtable meeting was organized with invited panelists, including private payer representatives and health economic consultants (total n = 9). The findings from both were synthesized and expressed in qualitative terms using the PICO framework.

RESULTS

The targeted review identified 20 studies meeting the inclusion criteria, primarily originating from the US and Canada. The panelists felt that, despite some similarities, there were substantial differences between both systems. The PICO framework highlighted the issues with transferability between the two systems. Most of the value parameters were either not applicable, needed to be added, needed to be adjusted, or their applicability to private payer systems needed to be confirmed.

CONCLUSION

Some components of public HTA may be relevant for private payers, however there are reservations that still exist on whether the HTA process in Canada, designed for a public system, can address the informational needs of private payers. Private insurers need to use caution in assessing which value parameters from public HTAs can be used and which need to be confirmed, ignored, enhanced, or adjusted. One size HTA does not fit all applications.

Trusting the Results of Model-Based Economic Analyses: Is there a Pragmatic Validation Solution?

Models have become a nearly essential component of health technology assessment. This is because the efficacy and safety data available from clinical trials are insufficient to provide the required estimates of impact of new interventions over long periods of time and for other populations and subgroups. Despite more than five decades of use of these decision-analytic models, decision makers are still often presented with poorly validated models and thus trust in their results is impaired. Among the reasons for this vexing situation are the artificial nature of the models, impairing their validation against observable data, the complexity in their formulation and implementation, the lack of data against which to validate the model results, and the challenges of short timelines and insufficient resources. This article addresses this crucial problem of achieving models that produce results that can be trusted and the resulting requirements for validation and transparency, areas where our field is currently deficient. Based on their differing perspectives and experiences, the authors characterize the situation and outline the requirements for improvement and pragmatic solutions to the problem of inadequate validation.

The Importance of Model Structure in the Cost-Effectiveness Analysis of Primary Care Interventions for the Management of Hypertension

BACKGROUND

Management of hypertension can lead to significant reductions in blood pressure, thereby reducing the risk of cardiovascular disease. Modeling the course of cardiovascular disease is not without complications, and uncertainty surrounding the structure of a model will almost always arise once a choice of a model structure is defined.

OBJECTIVES

To provide a practical illustration of the impact on the results of cost-effectiveness of changing or adapting model structures in a previously published cost-utility analysis of a primary care intervention for the management of hypertension Targets and Self-Management for the Control of Blood Pressure in Stroke and at Risk Groups (TASMIN-SR).

METHODS

The case study assessed the structural uncertainty arising from model structure and from the exclusion of secondary events. Four alternative model structures were implemented. Long-term cost-effectiveness was estimated and the results compared with those from the TASMIN-SR model.

RESULTS

The main cost-effectiveness results obtained in the TASMIN-SR study did not change with the implementation of alternative model structures. Choice of model type was limited to a cohort Markov model, and because of the lack of epidemiological data, only model 4 captured structural uncertainty arising from the exclusion of secondary events in the case study model.

CONCLUSIONS

The results of this study indicate that the main conclusions drawn from the TASMIN-SR model of cost-effectiveness were robust to changes in model structure and the inclusion of secondary events. Even though one of the models produced results that were different to those of TASMIN-SR, the fact that the main conclusions were identical suggests that a more parsimonious model may have sufficed.

Assessing the Quality of Economic Evaluations of FDA Novel Drug Approvals: A Systematic Review

OBJECTIVE

To systematically review and assess the quality of the novel drugs' economic evaluation literature in print during the drugs' early commercial availability following US regulatory approval.

DATA SOURCES

MEDLINE and the United Kingdom National Health Service Economic Evaluation Database were searched from 1946 through December 2011 for economic evaluations of the 50 novel drugs approved by the FDA in 2008 and 2009.

STUDY SELECTION AND DATA EXTRACTION

The inclusion criteria were English-language, peer-reviewed, original economic evaluations (cost-utility, cost-effectiveness, cost-minimization, and cost-benefit analyses). We extracted and analyzed data from 36 articles considering 19 of the 50 drugs. Two reviewers assessed each publication's quality using the Quality of Health Economic Studies (QHES) instrument and summarized study quality on a 100-point scale.

DATA SYNTHESIS

Study quality had a mean of 70.0 ± 16.2 QHES points. The only study characteristics associated with QHES score (with P < 0.05) were having used modeling or advanced statistics, 75.1 versus 61.9 without; using quality-adjusted life years as an outcome, 75.9 versus 64.7 without; and cost-utility versus cost-minimization analysis, 75.9 versus 58.7. Studies most often satisfied quality aspects about stating study design choices and least often satisfied aspects about justifying design choices.

CONCLUSION

The reviewed literature considered a minority of the 2008-2009 novel drugs and had mixed study quality. Cost-effectiveness stakeholders might benefit from efforts to improve the quality and quantity of literature examining novel drugs. Editors and reviewers may support quality improvement by stringently imposing economic evaluation guidelines about justifying study design choices.

Do Economic Evaluations in Primary Care Prevention and the Management of Hypertension Conform to Good Practice Guidelines? A Systematic Review

Background: Results of previous research have identified the need for further investigation into the compliance with good practice guidelines for current decision-analytic modeling (DAM). Objective: To identify the extent to which recent model-based economic evaluations of interventions focused on lowering the blood pressure (BP) of patients with hypertension conform to published guidelines for DAM in health care using a five-dimension framework developed to assess compliance to DAM guidelines. Methods: A systematic review of English language articles was undertaken to identify published model-based economic evaluations that examined interventions aimed at lowering BP. The review covered the period January 2000 to March 2015 and included the following electronic bibliographic databases: EMBASE and Medline via Ovid interface and the Centre for Reviews and Dissemination’s (CRD) NHS-EED. Data were extracted based on different components of good practice across five dimensions utilizing a framework to assess compliance to DAM guidelines. Results: Thirteen articles were included in this review. The review found limited compliance to good practice DAM guidelines, which was most frequently justified by the lack of data. Conclusions: The assessment of structural uncertainty cannot yet be considered common practice in primary prevention and management of hypertension, and researchers seem to face difficulties with identifying sources of structural uncertainty and then handling them correctly. Additional guidelines are needed to aid researchers in identifying and managing sources of potential structural uncertainty. Adherence to guidelines is not always possible and it does pose challenges, in particular when there are limitations due to data availability that restrict, for example, a validation process.

Systematic Review of Model-Based Economic Evaluations of Treatments for Alzheimer's Disease

BACKGROUND

Numerous economic evaluations using decision-analytic models have assessed the cost effectiveness of treatments for Alzheimer's disease (AD) in the last two decades. It is important to understand the methods used in the existing models of AD and how they could impact results, as they could inform new model-based economic evaluations of treatments for AD.

OBJECTIVE

The aim of this systematic review was to provide a detailed description on the relevant aspects and components of existing decision-analytic models of AD, identifying areas for improvement and future development, and to conduct a quality assessment of the included studies.

METHODS

We performed a systematic and comprehensive review of cost-effectiveness studies of pharmacological treatments for AD published in the last decade (January 2005 to February 2015) that used decision-analytic models, also including studies considering patients with mild cognitive impairment (MCI). The background information of the included studies and specific information on the decision-analytic models, including their approach and components, assumptions, data sources, analyses, and results, were obtained from each study. A description of how the modeling approaches and assumptions differ across studies, identifying areas for improvement and future development, is provided. At the end, we present our own view of the potential future directions of decision-analytic models of AD and the challenges they might face.

RESULTS

The included studies present a variety of different approaches, assumptions, and scope of decision-analytic models used in the economic evaluation of pharmacological treatments of AD. The major areas for improvement in future models of AD are to include domains of cognition, function, and behavior, rather than cognition alone; include a detailed description of how data used to model the natural course of disease progression were derived; state and justify the economic model selected and structural assumptions and limitations; provide a detailed (rather than high-level) description of the cost components included in the model; and report on the face-, internal-, and cross-validity of the model to strengthen the credibility and confidence in model results. The quality scores of most studies were rated as fair to good (average 87.5, range 69.5-100, in a scale of 0-100).

CONCLUSION

Despite the advancements in decision-analytic models of AD, there remain several areas of improvement that are necessary to more appropriately and realistically capture the broad nature of AD and the potential benefits of treatments in future models of AD.

Exploring Uncertainty in Economic Evaluations of Drugs and Medical Devices: Lessons from the First Review of Manufacturers' Submissions to the French National Authority for Health

OBJECTIVES

The objective of this paper was to evaluate how uncertainty has been accounted for in the cost-effectiveness analyses (CEAs) submitted by manufacturers to the French National Authority for Health (HAS) and to identify recurring concerns in these submissions.

METHODS

We used a cross-sectional design to evaluate manufacturers' submissions from the beginning of the evaluation process in October 2013 to the end of May 2015 (n = 28). The sources of uncertainty attached to these CEAs were categorized and assessed. Relevant data were extracted independently by two assessors.

RESULTS

Adherence to the HAS reference case was generally considered to be acceptable. Methodological uncertainty and parameter uncertainty were the sources of uncertainty that were most frequently explored by manufacturers. The quality of reporting of deterministic sensitivity analysis and probabilistic sensitivity analysis varied substantially across submissions, with a frequent lack of justification of the plausible range of parameter point estimates in 12 submissions (43 %). Structural uncertainty was explored much less frequently. Concerns related to omission of either important clinical events or relevant health states or extrapolation of the effects of the technology beyond the time horizon of the clinical trials were identified in 16 submissions (57 %).

CONCLUSIONS

This study presented a characterization of the treatment of uncertainty for the first 28 manufacturers' submissions to the HAS. This work identified important concerns regarding the exploration of sources of uncertainty. The findings may help manufacturers to improve the quality of their submissions and may provide useful insights for extending guidelines on uncertainty analysis in CEAs submitted to the HAS.

Linked Sensitivity Analysis, Calibration, and Uncertainty Analysis Using a System Dynamics Model for Stroke Comparative Effectiveness Research

BACKGROUND

As health services researchers and decision makers tackle more difficult problems using simulation models, the number of parameters and the corresponding degree of uncertainty have increased. This often results in reduced confidence in such complex models to guide decision making.

OBJECTIVE

To demonstrate a systematic approach of linked sensitivity analysis, calibration, and uncertainty analysis to improve confidence in complex models.

METHODS

Four techniques were integrated and applied to a System Dynamics stroke model of US veterans, which was developed to inform systemwide intervention and research planning: Morris method (sensitivity analysis), multistart Powell hill-climbing algorithm and generalized likelihood uncertainty estimation (calibration), and Monte Carlo simulation (uncertainty analysis).

RESULTS

Of 60 uncertain parameters, sensitivity analysis identified 29 needing calibration, 7 that did not need calibration but significantly influenced key stroke outcomes, and 24 not influential to calibration or stroke outcomes that were fixed at their best guess values. One thousand alternative well-calibrated baselines were obtained to reflect calibration uncertainty and brought into uncertainty analysis. The initial stroke incidence rate among veterans was identified as the most influential uncertain parameter, for which further data should be collected. That said, accounting for current uncertainty, the analysis of 15 distinct prevention and treatment interventions provided a robust conclusion that hypertension control for all veterans would yield the largest gain in quality-adjusted life years.

CONCLUSIONS

For complex health care models, a mixed approach was applied to examine the uncertainty surrounding key stroke outcomes and the robustness of conclusions. We demonstrate that this rigorous approach can be practical and advocate for such analysis to promote understanding of the limits of certainty in applying models to current decisions and to guide future data collection.

Characterizing Heterogeneity Bias in Cohort-Based Models

PURPOSE

Previous research using numerical methods suggested that use of a cohort-based model instead of an individual-based model can result in significant heterogeneity bias. However, the direction of the bias is not known a priori. We characterized mathematically the conditions that lead to upward or downward bias.

METHOD

We used a standard three-state disease progression model to evaluate the cost effectiveness of a hypothetical intervention. We solved the model analytically and derived expressions for life expectancy, discounted quality-adjusted life years (QALYs), discounted lifetime costs and incremental net monetary benefits (INMB). An outcome was calculated using the mean of the input under the cohort-based approach and the whole input distribution for all persons under the individual-based approach. We investigated the impact of heterogeneity on outcomes by varying one parameter at a time while keeping all others constant. We evaluated the curvature of outcome functions and used Jensen's inequality to determine the direction of the bias.

RESULTS

Both life expectancy and QALYs were underestimated by the cohort-based approach. If there was heterogeneity only in disease progression, total costs were overestimated, whereas QALYs gained, incremental costs and INMB were under- or overestimated, depending on the progression rate. INMB was underestimated when only efficacy was heterogeneous. Both approaches yielded the same outcome when the heterogeneity was only in cost or utilities.

CONCLUSION

A cohort-based approach that does not adjust for heterogeneity underestimates life expectancy and may underestimate or overestimate other outcomes. Characterizing the bias is useful for comparative assessment of models and informing decision making.

Exploring structural uncertainty in model-based economic evaluations

Given the inherent uncertainty in estimates produced by decision analytic models, the assessment of uncertainty in model-based evaluations is an essential part of the decision-making process. Although the impact of uncertainty around the choice of model structure and making incorrect structural assumptions on model predictions is noted, relatively little attention has been paid to characterising this type of uncertainty in guidelines developed by national funding bodies such as the Australian Pharmaceutical Benefits Advisory Committee (PBAC). The absence of a detailed description and evaluation of structural uncertainty can add further uncertainty to the decision-making process, with potential impact on the quality of funding decisions. This paper provides a summary of key elements of structural uncertainty describing why it matters and how it could be characterised. Five alternative approaches to characterising structural uncertainty are discussed, including scenario analysis, model selection, model averaging, parameterization and discrepancy. We argue that the potential effect of structural uncertainty on model predictions should be considered in submissions to national funding bodies; however, the characterisation of structural uncertainty is not well defined within the guidelines of these bodies. There has been little consideration of the forms of structural sensitivity analysis that might best inform applied decision-making processes, and empirical research in this area is required.

A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions

Multi-criteria decision analysis (MCDA) is increasingly used to support decisions in healthcare involving multiple and conflicting criteria. Although uncertainty is usually carefully addressed in health economic evaluations, whether and how the different sources of uncertainty are dealt with and with what methods in MCDA is less known. The objective of this study is to review how uncertainty can be explicitly taken into account in MCDA and to discuss which approach may be appropriate for healthcare decision makers. A literature review was conducted in the Scopus and PubMed databases. Two reviewers independently categorized studies according to research areas, the type of MCDA used, and the approach used to quantify uncertainty. Selected full text articles were read for methodological details. The search strategy identified 569 studies. The five approaches most identified were fuzzy set theory (45% of studies), probabilistic sensitivity analysis (15%), deterministic sensitivity analysis (31%), Bayesian framework (6%), and grey theory (3%). A large number of papers considered the analytic hierarchy process in combination with fuzzy set theory (31%). Only 3% of studies were published in healthcare-related journals. In conclusion, our review identified five different approaches to take uncertainty into account in MCDA. The deterministic approach is most likely sufficient for most healthcare policy decisions because of its low complexity and straightforward implementation. However, more complex approaches may be needed when multiple sources of uncertainty must be considered simultaneously.