The Challenge of Transparency and Validation in Health Economic Decision Modelling: A View from Mount Hood

A structured process for the validation of a decision-analytic model: application to a cost-effectiveness model for risk-stratified national breast screening

BACKGROUND

Decision-makers require knowledge of the strengths and weaknesses of decision-analytic models used to evaluate healthcare interventions to be able to confidently use the results of such models to inform policy. A number of aspects of model validity have previously been described, but no systematic approach to assessing the validity of a model has been proposed. This study aimed to consolidate the different aspects of model validity into a step-by-step approach to assessing the strengths and weaknesses of a decision-analytic model.

METHODS

A pre-defined set of steps were used to conduct the validation process of an exemplar early decision-analytic-model-based cost-effectiveness analysis of a risk-stratified national breast cancer screening programme [UK healthcare perspective; lifetime horizon; costs (£; 2021)]. Internal validation was assessed in terms of descriptive validity, technical validity and face validity. External validation was assessed in terms of operational validation, convergent validity (or corroboration) and predictive validity.

RESULTS

The results outline the findings of each step of internal and external validation of the early decision-analytic-model and present the validated model (called 'MANC-RISK-SCREEN'). The positive aspects in terms of meeting internal validation requirements are shown together with the remaining limitations of MANC-RISK-SCREEN.

CONCLUSION

Following a transparent and structured validation process, MANC-RISK-SCREEN has been shown to have satisfactory internal and external validity for use in informing resource allocation decision-making. We suggest that MANC-RISK-SCREEN can be used to assess the cost-effectiveness of exemplars of risk-stratified national breast cancer screening programmes (NBSP) from the UK perspective.

IMPLICATIONS

A step-by-step process for conducting the validation of a decision-analytic model was developed for future use by health economists. Using this approach may help researchers to fully demonstrate the strengths and limitations of their model to decision-makers.

Estimating the cost-effectiveness of intermittently scanned continuous glucose monitoring in adults with type 1 diabetes in England

OBJECTIVE

We previously showed that intermittently scanned continuous glucose monitoring (isCGM) reduces HbA1c at 24 weeks compared with self-monitoring of blood glucose with finger pricking (SMBG) in adults with type 1 diabetes and high HbA1c levels (58-97 mmol/mol [7.5%-11%]). We aim to assess the economic impact of isCGM compared with SMBG.

METHODS

Participant-level baseline and follow-up health status (EQ-5D-5L) and within-trial healthcare resource-use data were collected. Quality-adjusted life-years (QALYs) were derived at 24 weeks, adjusting for baseline EQ-5D-5L. Participant-level costs were generated. Using the IQVIA CORE Diabetes Model, economic analysis was performed from the National Health Service perspective over a lifetime horizon, discounted at 3.5%.

RESULTS

Within-trial EQ-5D-5L showed non-significant adjusted incremental QALY gain of 0.006 (95% CI: -0.007 to 0.019) for isCGM compared with SMBG and an adjusted cost increase of £548 (95% CI: 381-714) per participant. The lifetime projected incremental cost (95% CI) of isCGM was £1954 (-5108 to 8904) with an incremental QALY (95% CI) gain of 0.436 (0.195-0.652) resulting in an incremental cost-per-QALY of £4477. In all subgroups, isCGM had an incremental cost-per-QALY better than £20,000 compared with SMBG; for people with baseline HbA1c >75 mmol/mol (9.0%), it was cost-saving. Sensitivity analysis suggested that isCGM remains cost-effective if its effectiveness lasts for at least 7 years.

CONCLUSION

While isCGM is associated with increased short-term costs, compared with SMBG, its benefits in lowering HbA1c will lead to sufficient long-term health-gains and cost-savings to justify costs, so long as the effect lasts into the medium term.

Projected health and economic impacts of sugar-sweetened beverage taxation in Germany: A cross-validation modelling study

BACKGROUND

Taxes on sugar-sweetened beverages (SSBs) have been implemented globally to reduce the burden of cardiometabolic diseases by disincentivizing consumption through increased prices (e.g., 1 peso/litre tax in Mexico) or incentivizing industry reformulation to reduce SSB sugar content (e.g., tiered structure of the United Kingdom [UK] Soft Drinks Industry Levy [SDIL]). In Germany, where no tax on SSBs is enacted, the health and economic impact of SSB taxation using the experience from internationally implemented tax designs has not been evaluated. The objective of this study was to estimate the health and economic impact of national SSBs taxation scenarios in Germany.

METHODS AND FINDINGS

In this modelling study, we evaluated a 20% ad valorem SSB tax with/without taxation of fruit juice (based on implemented SSB taxes and recommendations) and a tiered tax (based on the UK SDIL) in the German adult population aged 30 to 90 years from 2023 to 2043. We developed a microsimulation model (IMPACTNCD Germany) that captures the demographics, risk factor profile and epidemiology of type 2 diabetes, coronary heart disease (CHD) and stroke in the German population using the best available evidence and national data. For each scenario, we estimated changes in sugar consumption and associated weight change. Resulting cases of cardiometabolic disease prevented/postponed and related quality-adjusted life years (QALYs) and economic impacts from healthcare (medical costs) and societal (medical, patient time, and productivity costs) perspectives were estimated using national cost and health utility data. Additionally, we assessed structural uncertainty regarding direct, body mass index (BMI)-independent cardiometabolic effects of SSBs and cross-validated results with an independently developed cohort model (PRIMEtime). We found that SSB taxation could reduce sugar intake in the German adult population by 1 g/day (95%-uncertainty interval [0.05, 1.65]) for a 20% ad valorem tax on SSBs leading to reduced consumption through increased prices (pass-through of 82%) and 2.34 g/day (95%-UI [2.32, 2.36]) for a tiered tax on SSBs leading to 30% reduction in SSB sugar content via reformulation. Through reductions in obesity, type 2 diabetes, and cardiovascular disease (CVD), 106,000 (95%-UI [57,200, 153,200]) QALYs could be gained with a 20% ad valorem tax and 192,300 (95%-UI [130,100, 254,200]) QALYs with a tiered tax. Respectively, €9.6 billion (95%-UI [4.7, 15.3]) and €16.0 billion (95%-UI [8.1, 25.5]) costs could be saved from a societal perspective over 20 years. Impacts of the 20% ad valorem tax were larger when additionally taxing fruit juice (252,400 QALYs gained, 95%-UI [176,700, 325,800]; €11.8 billion costs saved, 95%-UI [€6.7, €17.9]), but impacts of all scenarios were reduced when excluding direct health effects of SSBs. Cross-validation with PRIMEtime showed similar results. Limitations include remaining uncertainties in the economic and epidemiological evidence and a lack of product-level data.

CONCLUSIONS

In this study, we found that SSB taxation in Germany could help to reduce the national burden of noncommunicable diseases and save a substantial amount of societal costs. A tiered tax designed to incentivize reformulation of SSBs towards less sugar might have a larger population-level health and economic impact than an ad valorem tax that incentivizes consumer behaviour change only through increased prices.

Prediction of complications in health economic models of type 2 diabetes: a review of methods used

AIM

Diabetes health economic (HE) models play important roles in decision making. For most HE models of diabetes 2 diabetes (T2D), the core model concerns the prediction of complications. However, reviews of HE models pay little attention to the incorporation of prediction models. The objective of the current review is to investigate how prediction models have been incorporated into HE models of T2D and to identify challenges and possible solutions.

METHODS

PubMed, Web of Science, Embase, and Cochrane were searched from January 1, 1997, to November 15, 2022, to identify published HE models for T2D. All models that participated in The Mount Hood Diabetes Simulation Modeling Database or previous challenges were manually searched. Data extraction was performed by two independent authors. Characteristics of HE models, their underlying prediction models, and methods of incorporating prediction models were investigated.

RESULTS

The scoping review identified 34 HE models, including a continuous-time object-oriented model (n = 1), discrete-time state transition models (n = 18), and discrete-time discrete event simulation models (n = 15). Published prediction models were often applied to simulate complication risks, such as the UKPDS (n = 20), Framingham (n = 7), BRAVO (n = 2), NDR (n = 2), and RECODe (n = 2). Four methods were identified to combine interdependent prediction models for different complications, including random order evaluation (n = 12), simultaneous evaluation (n = 4), the 'sunflower method' (n = 3), and pre-defined order (n = 1). The remaining studies did not consider interdependency or reported unclearly.

CONCLUSIONS

The methodology of integrating prediction models in HE models requires further attention, especially regarding how prediction models are selected, adjusted, and ordered.

Cost-Effectiveness of Venetoclax Plus Obinutuzumab Versus Chlorambucil Plus Obinutuzumab for the First-Line Treatment of Adult Patients With Chronic Lymphocytic Leukemia: An Extended Societal View

OBJECTIVES

Efficacy of venetoclax plus obinutuzumab (VenO) compared with chlorambucil plus obinutuzumab (ClbO) for treatment-naïve adult patients with chronic lymphocytic leukemia (CLL) with coexisting medical conditions was investigated in CLL14 (NCT02242942). Our aim was to evaluate the cost-effectiveness of VenO versus ClbO for these patients from a Dutch societal perspective.

METHODS

A 3-state partitioned survival model was constructed to evaluate the cost-effectiveness of VenO. The outcome of the analysis was the incremental cost-effectiveness ratio (ICER) with effectiveness measured in quality-adjusted life-years (QALYs) gained. Uncertainty was explored through deterministic and probabilistic sensitivity analyses, scenario analyses, and value of information analysis (VOI).

RESULTS

The base case resulted in a discounted ICER -49 928 EUR/QALY gained (with incremental negative costs and positive effects). None of the ICERs resulted from deterministic sensitivity and scenario analyses exceeded the chosen willingness-to-pay threshold of 20 000 EUR/QALY, and > 99% of the iterations in the probabilistic sensitivity analysis were cost-effective. VOI analyses showed a maximum expected value of eliminating all model parameter uncertainty of 183 591 EUR.

CONCLUSIONS

Our study demonstrated VenO being dominant over ClbO in treatment-naïve adult patients with CLL assuming a Dutch societal perspective. We concluded that our results are robust as tested through sensitivity and scenario analyses. Additionally, the VOI analyses confirmed that our current evidence base is strong enough to generate reliable results for our study. Nevertheless, further research based on real-world data or longer follow-up period could further contribute to the robustness of the current study's conclusions.

Interpreting decision-analytic modelling-based economic evaluations in orthopaedics

Economic evaluation provides a framework for assessing the costs and consequences of alternative programmes or interventions. One common vehicle for economic evaluations in the healthcare context is the decision-analytic model, which synthesizes information on parameter inputs (for example, probabilities or costs of clinical events or health states) from multiple sources and requires application of mathematical techniques, usually within a software program. A plethora of decision-analytic modelling-based economic evaluations of orthopaedic interventions have been published in recent years. This annotation outlines a number of issues that can help readers, reviewers, and decision-makers interpret evidence from decision-analytic modelling-based economic evaluations of orthopaedic interventions.Cite this article: Bone Joint J 2023;105-B(1):17-20.

Costs of major complications in people with and without diabetes in Tasmania, Australia

Objective We set out to estimate healthcare costs of diabetes complications in the year of first occurrence and the second year, and to quantify the incremental costs of diabetes versus non-diabetes related to each complication. Methods In this cohort study, people with diabetes (n  = 45 378) and their age/sex propensity score matched controls (n  = 90 756) were identified from a linked dataset in Tasmania, Australia between 2004 and 2017. Direct costs (including hospital, emergency room visits and pathology costs) were calculated from the healthcare system perspective and expressed in 2020 Australian dollars. The average-per-patient costs and the incremental costs in people with diabetes were calculated for each complication. Results First-year costs when the complications occurred were: dialysis $78 152 (95% CI 71 095, 85 858), lower extremity amputations $63 575 (58 290, 68 688), kidney transplant $48 487 (33 862, 68 283), non-fatal myocardial infarction $30 827 (29 558, 32 197), foot ulcer/gangrene $29 803 (27 183, 32 675), ischaemic heart disease $29 160 (26 962, 31 457), non-fatal stroke $27 782 (26 285, 29 354), heart failure $27 379 (25 968, 28 966), kidney failure $24 904 (19 799, 32 557), angina pectoris $18 430 (17 147, 19 791), neuropathy $15 637 (14 265, 17 108), nephropathy $15 133 (12 285, 18 595), retinopathy $14 775 (11 798, 19 199), transient ischaemic attack $13 905 (12 529, 15 536), vitreous hemorrhage $13 405 (10 241, 17 321), and blindness/low vision $12 941 (8164, 19 080). The second-year costs ranged from 16% (ischaemic heart disease) to 74% (dialysis) of first-year costs. Complication costs were 109-275% higher than in people without diabetes. Conclusions Diabetes complications are costly, and the costs are higher in people with diabetes than without diabetes. Our results can be used to populate diabetes simulation models and will support policy analyses to reduce the burden of diabetes.

Estimating the impact of nutrition and physical activity policies with quasi-experimental methods and simulation modelling: an integrative review of methods, challenges and synergies

BACKGROUND

The promotion of healthy lifestyles has high priority on the global public health agenda. Evidence on the real-world (cost-)effectiveness of policies addressing nutrition and physical activity is needed. To estimate short-term policy impacts, quasi-experimental methods using observational data are useful, while simulation models can estimate long-term impacts. We review the methods, challenges and potential synergies of both approaches for the evaluation of nutrition and physical activity policies.

METHODS

We performed an integrative review applying purposive literature sampling techniques to synthesize original articles, systematic reviews and lessons learned from public international workshops conducted within the European Union Policy Evaluation Network.

RESULTS

We highlight data requirements for policy evaluations, discuss the distinct assumptions of instrumental variable, difference-in-difference, and regression discontinuity designs and describe the necessary robustness and falsification analyses to test them. Further, we summarize the specific assumptions of comparative risk assessment and Markov state-transition simulation models, including their extension to microsimulation. We describe the advantages and limitations of these modelling approaches and discuss future directions, such as the adequate consideration of heterogeneous policy responses. Finally, we highlight how quasi-experimental and simulation modelling methods can be integrated into an evidence cycle for policy evaluation.

CONCLUSIONS

Assumptions of quasi-experimental and simulation modelling methods in policy evaluations should be credible, rigorously tested and transparently communicated. Both approaches can be applied synergistically within a coherent framework to compare policy implementation scenarios and improve the estimation of nutrition and physical activity policy impacts, including their distribution across population sub-groups.

A meta-epidemiological assessment of transparency indicators of infectious disease models

Mathematical models have become very influential, especially during the COVID-19 pandemic. Data and code sharing are indispensable for reproducing them, protocol registration may be useful sometimes, and declarations of conflicts of interest (COIs) and of funding are quintessential for transparency. Here, we evaluated these features in publications of infectious disease-related models and assessed whether there were differences before and during the COVID-19 pandemic and for COVID-19 models versus models for other diseases. We analysed all PubMed Central open access publications of infectious disease models published in 2019 and 2021 using previously validated text mining algorithms of transparency indicators. We evaluated 1338 articles: 216 from 2019 and 1122 from 2021 (of which 818 were on COVID-19); almost a six-fold increase in publications within the field. 511 (39.2%) were compartmental models, 337 (25.2%) were time series, 279 (20.9%) were spatiotemporal, 186 (13.9%) were agent-based and 25 (1.9%) contained multiple model types. 288 (21.5%) articles shared code, 332 (24.8%) shared data, 6 (0.4%) were registered, and 1197 (89.5%) and 1109 (82.9%) contained COI and funding statements, respectively. There was no major changes in transparency indicators between 2019 and 2021. COVID-19 articles were less likely to have funding statements and more likely to share code. Further validation was performed by manual assessment of 10% of the articles identified by text mining as fulfilling transparency indicators and of 10% of the articles lacking them. Correcting estimates for validation performance, 26.0% of papers shared code and 41.1% shared data. On manual assessment, 5/6 articles identified as registered had indeed been registered. Of articles containing COI and funding statements, 95.8% disclosed no conflict and 11.7% reported no funding. Transparency in infectious disease modelling is relatively low, especially for data and code sharing. This is concerning, considering the nature of this research and the heightened influence it has acquired.

Exploring Structural Uncertainty and Impact of Health State Utility Values on Lifetime Outcomes in Diabetes Economic Simulation Models: Findings from the Ninth Mount Hood Diabetes Quality-of-Life Challenge

BACKGROUND

Structural uncertainty can affect model-based economic simulation estimates and study conclusions. Unfortunately, unlike parameter uncertainty, relatively little is known about its magnitude of impact on life-years (LYs) and quality-adjusted life-years (QALYs) in modeling of diabetes. We leveraged the Mount Hood Diabetes Challenge Network, a biennial conference attended by international diabetes modeling groups, to assess structural uncertainty in simulating QALYs in type 2 diabetes simulation models.

METHODS

Eleven type 2 diabetes simulation modeling groups participated in the 9th Mount Hood Diabetes Challenge. Modeling groups simulated 5 diabetes-related intervention profiles using predefined baseline characteristics and a standard utility value set for diabetes-related complications. LYs and QALYs were reported. Simulations were repeated using lower and upper limits of the 95% confidence intervals of utility inputs. Changes in LYs and QALYs from tested interventions were compared across models. Additional analyses were conducted postchallenge to investigate drivers of cross-model differences.

RESULTS

Substantial cross-model variability in incremental LYs and QALYs was observed, particularly for HbA1c and body mass index (BMI) intervention profiles. For a 0.5%-point permanent HbA1c reduction, LY gains ranged from 0.050 to 0.750. For a 1-unit permanent BMI reduction, incremental QALYs varied from a small decrease in QALYs (-0.024) to an increase of 0.203. Changes in utility values of health states had a much smaller impact (to the hundredth of a decimal place) on incremental QALYs. Microsimulation models were found to generate a mean of 3.41 more LYs than cohort simulation models (P = 0.049).

CONCLUSIONS

Variations in utility values contribute to a lesser extent than uncertainty captured as structural uncertainty. These findings reinforce the importance of assessing structural uncertainty thoroughly because the choice of model (or models) can influence study results, which can serve as evidence for resource allocation decisions.HighlightsThe findings indicate substantial cross-model variability in QALY predictions for a standardized set of simulation scenarios and is considerably larger than within model variability to alternative health state utility values (e.g., lower and upper limits of the 95% confidence intervals of utility inputs).There is a need to understand and assess structural uncertainty, as the choice of model to inform resource allocation decisions can matter more than the choice of health state utility values.

Simulation Modeling for the Economic Evaluation of Population-Based Dietary Policies: A Systematic Scoping Review

Simulation modeling can be useful to estimate the long-term health and economic impacts of population-based dietary policies. We conducted a systematic scoping review following the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guideline to map and critically appraise economic evaluations of population-based dietary policies using simulation models. We searched Medline, Embase, and EconLit for studies published in English after 2005. Modeling studies were mapped based on model type, dietary policy, and nutritional target, and modeled risk factor-outcome pathways were analyzed. We included 56 studies comprising 136 model applications evaluating dietary policies in 21 countries. The policies most often assessed were reformulation (34/136), taxation (27/136), and labeling (20/136); the most common targets were salt/sodium (60/136), sugar-sweetened beverages (31/136), and fruit and vegetables (15/136). Model types included Markov-type (35/56), microsimulation (11/56), and comparative risk assessment (7/56) models. Overall, the key diet-related risk factors and health outcomes were modeled, but only 1 study included overall diet quality as a risk factor. Information about validation was only reported in 19 of 56 studies and few studies (14/56) analyzed the equity impacts of policies. Commonly included cost components were health sector (52/56) and public sector implementation costs (35/56), as opposed to private sector (18/56), lost productivity (11/56), and informal care costs (3/56). Most dietary policies (103/136) were evaluated as cost-saving independent of the applied costing perspective. An analysis of the main limitations reported by authors revealed that model validity, uncertainty of dietary effect estimates, and long-term intervention assumptions necessitate a careful interpretation of results. In conclusion, simulation modeling is widely applied in the economic evaluation of population-based dietary policies but rarely takes dietary complexity and the equity dimensions of policies into account. To increase relevance for policymakers and support diet-related disease prevention, economic effects beyond the health sector should be considered, and transparent conduct and reporting of model validation should be improved.

Appraisal of patient-level health economic models of severe mental illness: systematic review

BACKGROUND

Healthcare decision makers require accurate long-term economic models to evaluate the cost-effectiveness of new mental health interventions.

AIMS

To assess the suitability of current patient-level economic models to estimate long-term economic outcomes in severe mental illness.

METHOD

We undertook pre-specified systematic searches in MEDLINE, Embase and PsycINFO to identify reviews and stand-alone publications of economic models of interventions for schizophrenia, bipolar disorder and major depressive disorder (PROSPERO: CRD42020158243). We screened paper titles and abstracts to identify unique patient-level economic models. We conducted a structured extraction of identified models, recording the presence of key predefined model features. Model quality and validation were appraised using the 2014 ISPOR and 2016 AdViSHE model checklists.

RESULTS

We identified 15 unique patient-level models for psychosis and major depressive disorder from 1481 non-duplicate records. Models addressed schizophrenia (n = 6), bipolar disorder (n = 2) and major depressive disorder (n = 7). The predominant model type was discrete event simulation (n = 9). Model complexity and incorporation of patient heterogeneity varied considerably, and only five models extrapolated costs and outcomes over a lifetime horizon. Key model parameters were often based on low-quality evidence, and checklist quality assessment revealed weak model verification procedures.

CONCLUSIONS

Existing patient-level economic models of interventions for severe mental illness have considerable limitations. New modelling efforts must be supplemented by the generation of good-quality, contemporary evidence suitable for model building. Combined effort across the research community is required to build and validate economic extrapolation models suitable for accurately assessing the long-term value of new interventions from short-term clinical trial data.

Optimizing diabetes screening frequencies for at-risk groups

There is strong evidence that diabetes is underdiagnosed in the US: the Centers for Disease Control and Prevention (CDC) estimates that approximately 25% of diabetic patients are unaware of their condition. To encourage timely diagnosis of at-risk patients, we develop screening guidelines stratified by body mass index (BMI), age, and prior test history by using a Partially Observed Markov Decision Process (POMDP) framework to provide more personalized screening frequency recommendations. We identify structural results that prove the existence of threshold solutions in our problem and allow us to determine the relative timing and frequency of screening given different risk profiles. We then use nationally representative empirical data to identify a policy that provides the optimal action (screen or wait) every six months from age 45 to 90. We find that the current screening guidelines are suboptimal, and the recommended diabetes screening policy should be stratified by age and by finer BMI thresholds than in the status quo. We identify age ranges and BMI categories for which relatively less or more screening is needed compared to the existing guidelines to help physicians target patients most at risk. Compared to the status quo, we estimate that an optimal screening policy would generate higher net monetary benefits by $3,200-$3,570 and save $120-$1,290 in health expenditures per individual in the US above age 45.

Putting Stakeholder Engagement at the Center of Health Economic Modeling for Health Technology Assessment in the United States

While evidence generated from health economic (HE) models is being used more commonly in health technology assessment (HTA) in the US, it is not consistently adopted by different stakeholder groups or across therapeutic areas. We hypothesize that actively engaging with multiple stakeholder groups throughout the model development process may result in models more widely considered by decision makers. To test this hypothesis, the Innovation and Value Initiative has launched a modeling effort to build an open-source HE model focusing on the disease state 'major depressive disorder'. A 20-member advisory group has been formed with representatives from patients, employers, clinicians, innovators, payers, and researchers to guide the model development process. While this effort is still in the early stages, the ongoing stakeholder engagement effort has yielded valuable insights that inform the model design. We have also identified several challenges to implementing this new approach. Our early findings suggest that the stakeholder engagement approach to HE model development has the potential to improve HTA in the US.

Prediction of mortality and major cardiovascular complications in type 2 diabetes: External validation of UK Prospective Diabetes Study outcomes model version 2 in two European observational cohorts

AIM

To externally validate the UK Prospective Diabetes Study Outcomes Model version 2 (UKPDS-OM2) by comparing the predicted and observed outcomes in two European population-based cohorts of people with type 2 diabetes.

MATERIALS AND METHODS

We used data from the Casale Monferrato Survey (CMS; n = 1931) and a subgroup of the Hoorn Diabetes Care System (DCS) cohort (n = 5188). The following outcomes were analysed: all-cause mortality, myocardial infarction (MI), ischaemic heart disease (IHD), stroke, and congestive heart failure (CHF). Model performance was assessed by comparing predictions with observed cumulative incidences in each cohort during follow-up.

RESULTS

All-cause mortality was overestimated by the UKPDS-OM2 in both the cohorts, with a bias of 0.05 in the CMS and 0.12 in the DCS at 10 years of follow-up. For MI, predictions were consistently higher than observed incidence over the entire follow-up in both cohorts (10 years bias 0.07 for CMS and 0.10 for DCS). The model performed well for stroke and IHD outcomes in both cohorts. CHF incidence was predicted well for the DCS (5 years bias -0.001), but underestimated for the CMS cohort.

CONCLUSIONS

The UKPDS-OM2 consistently overpredicted the risk of mortality and MI in both cohorts during follow-up. Period effects may partially explain the differences. Results indicate that transferability is not satisfactory for all outcomes, and new or adjusted risk equations may be needed before applying the model to the Italian or Dutch settings.

Macrovascular Risk Equations Based on the CANVAS Program

BACKGROUND

Widely used risk equations for cardiovascular outcomes for individuals with type 2 diabetes mellitus (T2DM) have been incapable of predicting cardioprotective effects observed in recent cardiovascular outcomes trials (CVOTs) involving individuals with T2DM at high risk for or with established cardiovascular disease (CVD).

OBJECTIVE

We developed cardiovascular and mortality risk equations using patient-level data from the CANVAS (CANagliflozin cardioVascular Assessment Study) Program to address this shortcoming.

METHODS

Data from 10,142 patients with T2DM at high risk for or with established CVD, randomized to canagliflozin + standard of care (SoC) or SoC alone and followed for a mean duration of 3.6 years in the CANVAS Program were used to derive parametric risk equations for myocardial infarction (MI), stroke, hospitalization for heart failure (HHF), and death. Accumulated knowledge from the widely used UKPDS-OM2 (United Kingdom Prospective Diabetes Study Outcomes Model 2) was leveraged, and any departures in parameterization were limited to those necessary to provide adequate goodness of fit. Candidate explanatory covariates were selected using only the placebo arm to minimize confounding effects. Internal validation was performed separately by study treatment arm.

RESULTS

UKPDS-OM2 predicted CANVAS Program outcomes poorly. Recalibrating UKPDS-OM2 intercepts improved calibration in some cases. Refitting the coefficients but otherwise preserving the UKPDS-OM2 structure improved the fit substantially, which was sufficient for stroke and death. For MI, reselecting UKPDS-OM2 covariates and functional form proved sufficient. For HHF, selection from a broad set of candidate covariates and inclusion of a canagliflozin indicator was required.

CONCLUSION

These risk equations address some of the limitations of widely used risk equations, such as the UKPDS-OM2, for modeling cardioprotective treatments for individuals with T2DM and high cardiovascular risk, including derivation from overly healthy patients treated with agents that lack cardioprotection and have been described as reflecting a different therapeutic era. Future work is needed to examine external validity.

Development of a pharmacoeconomic registry: an example using hormonal contraceptives

BACKGROUND

Disease-specific registries, documenting costs and probabilities from pharmacoeconomic studies along with health state utility values from quality-of-life studies could serve as a resource to guide researchers in evaluating the published literature and in the conduct of future economic evaluations for their own research. Registries cataloging economic evaluations currently exist, however they are restricted by the type of economic evaluations they include. There is a need for intervention-specific registries, that document all types of complete and partial economic evaluations and auxiliary information such as quality of life studies. The objective of this study is to describe the development of a pharmacoeconomic registry and provide best practices using an example of hormonal contraceptives.

METHODS

An expert panel consisting of researchers with expertise in pharmacoeconomics and outcomes research was convened and the clinical focus of the registry was finalized after extensive discussion. A list of key continuous, categorical and descriptive variables was developed to capture all relevant data with each variable defined in a data dictionary. A web-based data collection tool was designed to capture and store the resulting metadata. A keyword based search strategy was developed to retrieve the published sources of literature. Finally, articles were screened for relevancy and data was extracted to populate the registry. Expert opinions were taken from the panel at each stage to arrive at consensus and ensure validity of the registry.

RESULTS

The registry focused on economic evaluation literature of hormonal contraceptives used for contraception. The registry consisted of 65 articles comprising of 22 cost-effectiveness analyses, 9 cost-utility analyses, 7 cost-benefit analyses, 1 cost-minimization, 14 cost analyses, 10 cost of illness studies and 2 quality of life studies. The best practices followed in the development of the registry were summarized as recommendations. The completed registry, data dictionary and associated data files can be accessed in the supplementary information files.

CONCLUSION

This registry is a comprehensive database of economic evaluations, including costs, clinical probabilities and health-state utility estimates. The collated data captured from published information in this registry can be used to identify trends in the literature, conduct systematic reviews and meta-analysis and develop novel pharmacoeconomic models.

External Validation of the Core Obesity Model to Assess the Cost-Effectiveness of Weight Management Interventions

BACKGROUND

For economic models to be considered fit for purpose, it is vital that their outputs can be interpreted with confidence by clinicians, budget holders and other stakeholders. Consequently, thorough validation of models should be carried out to enhance confidence in their predictions. Here, we present results of external dependent and independent validations of the Core Obesity Model (COM), which was developed to assess the cost-effectiveness of weight management interventions.

OBJECTIVE

The aim was to assess the external validity of the COM (version 6.1), in line with best practice guidance from the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making.

METHODS

For validation, suitable sources and outcomes were identified, and used to populate the COM with relevant inputs to allow prediction of study outcomes. Study characteristics were entered into the COM to replicate either the studies used to develop the model (dependent validation) or those not included in the model (independent validation). The concordance between predicted and observed outcomes was then assessed using established statistical methods and generation of mean error estimates.

RESULTS

For most outcomes, the predictions of the COM showed good linear correlation with observed outcomes, as evidenced by the high coefficients of determination (R² values). The independent validation revealed a degree of underestimation in predictions of cardiovascular (CV) disease and mortality, and type 2 diabetes.

CONCLUSION

The predictions generated by the risk equations used in the COM showed good concordance both with the studies used to develop the model and with studies not included in the model. In particular, the concordance observed in the external dependent validation suggests that the COM accurately predicts obesity-related event rates observed in the studies used to develop the model. However, the impact of existing CV risk, as well as mortality, is a key area for future refinement of the COM. Our results should increase confidence in the estimates derived from the COM and reduce uncertainty associated with analyses using this model.

Characterizing the Validity and Real-World Utility of Health Technology Assessments in Healthcare: Future Directions Comment on "Problems and Promises of Health Technologies: The Role of Early Health Economic Modelling"

With their article, Grutters et al raise an important question: What do successful health technology assessments (HTAs) look like, and what is their real-world utility in decision-making? While many HTAs are published in peer-reviewed journals, many are considered proprietary and their attributes remain confidential, limiting researchers’ ability to answer these questions. Models for economic evaluations like cost-effectiveness analyses (CEAs) synthesize a wide range of evidence, are often statistically and mathematically sophisticated, and require untestable assumptions. As such, there is nearly universal agreement among researchers that enhancing transparency is an important issue in health economic modeling. However, the definition of transparency and guidelines for its implementation vary. Model registration combined with a linked database of model-based economic evaluations has been proposed as a solution, whereby registered models and their accompanying technical and nontechnical documentation are sourced into a single publicly-available repository, ideally in a standardized format to ensure consistent and complete representation of features, code, data sources, results, validation exercises, and policy recommendations. When such a repository is ultimately created, modelers will not have to reinvent the wheel for every new drug launched or new treatment pathway. These more open and transparent approaches will have substantial implications for model accuracy, reliability, and validity, improving trust and acceptance by healthcare decision-makers.

A Systematic Review of Economic Models Across the Entire Schizophrenia Pathway

BACKGROUND

Schizophrenia is associated with a high economic burden. Economic models can help to inform resource allocation decisions to maximise benefits to patients.

OBJECTIVES

This systematic review aims to assess the availability, quality and consistency of conclusions of health economic models evaluating the cost effectiveness of interventions for schizophrenia.

METHODS

An electronic search was performed on multiple databases (MEDLINE, EMBASE, PsycINFO, Cochrane database of systematic reviews, NHS Economic Evaluation Database and Health Technology Assessment database) to identify economic models of interventions for schizophrenia published between 2005 and 2020. Two independent reviewers selected studies for inclusion. Study quality was assessed using the National Institute for Health and Care Excellence (NICE) checklist and the Cooper hierarchy. Model characteristics and conclusions were descriptively summarised.

RESULTS

Seventy-three models met inclusion criteria. Seventy-eight percent of existing models assessed antipsychotics; however, due to inconsistent conclusions reported by different studies, no antipsychotic can be considered clearly cost effective compared with the others. A very limited number of models suggest that the following non-pharmacological interventions might be cost effective: psychosocial interventions, stratified tests, employment intervention and intensive intervention to improve liaison between primary and secondary care. The quality of included models is generally low due to use of a short time horizon, omission of adverse events of interventions, poor data quality and potential conflicts of interest.

CONCLUSIONS

This review highlights a lack of models for non-pharmacological interventions, and limitations of the existing models, including low quality and inconsistency in conclusions. Recommendations on future modelling approaches for schizophrenia are provided.

Economic Simulation Modeling in Type 2 Diabetes

PURPOSE OF REVIEW

This paper provides an overview of type 2 diabetes economic simulation modeling and reviews current topics of discussion and major challenges in the field.

RECENT FINDINGS

Important challenges in the field include increasing the generalizability of models and improving transparency in model reporting. To identify and address these issues, modeling groups have organized through the Mount Hood Diabetes Challenge meetings and developed tools (i.e., checklist, impact inventory) to standardize modeling methods and reporting of results. Accordingly, many newer diabetes models have begun utilizing these tools, allowing for improved comparability between diabetes models. In the last two decades, type 2 diabetes simulation models have improved considerably, due to the collaborative work performed through the Mount Hood Diabetes Challenge meetings. To continue to improve diabetes models, future work must focus on clarifying diabetes progression in racial/ethnic minorities and incorporating equity considerations into health economic analysis.

Systematic Literature Review and Critical Appraisal of Health Economic Models Used in Cost-Effectiveness Analyses in Non-Alcoholic Steatohepatitis: Potential for Improvements

BACKGROUND

Non-alcoholic steatohepatitis (NASH) is a severe, typically progressive form of non-alcoholic fatty liver disease (NAFLD). The global prevalence of NASH is increasing, driven partly by the global increase in obesity and type 2 diabetes mellitus (T2DM), such that NASH is now a leading cause of cirrhosis. There is currently an unmet clinical need for efficacious and cost-effective treatments for NASH; no pharmacologic agents have an approved indication for NASH.

OBJECTIVE

Our objective was to summarise and critically appraise published health economic models of NASH, to evaluate their quality and suitability for use in the assessment of novel treatments for NASH, and to identify knowledge gaps, challenges and opportunities for future modelling.

METHODS

A systematic literature review was performed using the MEDLINE, Embase, Cochrane Library and EconLit databases to identify published health economic analyses in patients with NAFLD or NASH. Supplementary hand searches of grey literature were also performed. Articles published up to November 2019 were included in the review. Quality assessment of identified studies was also performed.

RESULTS

A total of 19 articles comprising 16 unique models including either NAFLD as a whole or NASH alone were included in the review. Structurally, most models had a state-transition component; in terms of health states, two different approaches to early disease states were used, modelling either progression through fibrosis stages or NAFLD/NASH-specific health states. Conditions that frequently co-exist with NASH, such as obesity, T2DM and cardiovascular disease were not captured in models identified here. Late-stage complications such as cirrhosis, decompensated cirrhosis and hepatocellular carcinoma were consistently included, but input data (e.g. costs, utilities and transition probabilities) for late-stage complications were frequently sourced from other liver disease areas. The quality of included studies was heterogenous, and only a small proportion of studies reported internal and external validation processes.

CONCLUSION

The health economic models identified in this review are associated with limitations primarily driven by a lack of NASH-specific data. Identified models also largely overlooked the intricate association between NASH and other conditions, including obesity and T2DM, and did not capture the increased risk of cardiovascular events associated with NASH. High-quality, transparent, validated health economic models of NASH will be required to evaluate the cost effectiveness of treatments currently in development, particularly compounds that may target other non-hepatic outcomes.