The Mt. Hood challenge: cross-testing two diabetes simulation models

Cross-model validation of public health microsimulation models; comparing two models on estimated effects of a weight management intervention

BACKGROUND

Health economic modelling indicates that referral to a behavioural weight management programme is cost saving and generates QALY gains compared with a brief intervention. The aim of this study was to conduct a cross-model validation comparing outcomes from this cost-effectiveness analysis to those of a comparator model, to understand how differences in model structure contribute to outcomes.

METHODS

The outcomes produced by two models, the School for Public Health Research diabetes prevention (SPHR) and Health Checks (HC) models, were compared for three weight-management programme strategies; Weight Watchers (WW) for 12 weeks, WW for 52 weeks, and a brief intervention, and a simulated no intervention scenario. Model inputs were standardised, and iterative adjustments were made to each model to identify drivers of differences in key outcomes.

RESULTS

The total QALYs estimated by the HC model were higher in all treatment groups than those estimated by the SPHR model, and there was a large difference in incremental QALYs between the models. SPHR simulated greater QALY gains for 12-week WW and 52-week WW relative to the Brief Intervention. Comparisons across socioeconomic groups found a stronger socioeconomic gradient in the SPHR model. Removing the impact of treatment on HbA1c from the SPHR model, running both models only with the conditions that the models have in common and, to a lesser extent, changing the data used to estimate risk factor trajectories, resulted in more consistent model outcomes.

CONCLUSIONS

The key driver of difference between the models was the inclusion of extra evidence-based detail in SPHR on the impacts of treatments on HbA1c. The conclusions were less sensitive to the dataset used to inform the risk factor trajectories. These findings strengthen the original cost-effectiveness analyses of the weight management interventions and provide an increased understanding of what is structurally important in the models.

Exploring Structural Uncertainty and Impact of Health State Utility Values on Lifetime Outcomes in Diabetes Economic Simulation Models: Findings from the Ninth Mount Hood Diabetes Quality-of-Life Challenge

BACKGROUND

Structural uncertainty can affect model-based economic simulation estimates and study conclusions. Unfortunately, unlike parameter uncertainty, relatively little is known about its magnitude of impact on life-years (LYs) and quality-adjusted life-years (QALYs) in modeling of diabetes. We leveraged the Mount Hood Diabetes Challenge Network, a biennial conference attended by international diabetes modeling groups, to assess structural uncertainty in simulating QALYs in type 2 diabetes simulation models.

METHODS

Eleven type 2 diabetes simulation modeling groups participated in the 9th Mount Hood Diabetes Challenge. Modeling groups simulated 5 diabetes-related intervention profiles using predefined baseline characteristics and a standard utility value set for diabetes-related complications. LYs and QALYs were reported. Simulations were repeated using lower and upper limits of the 95% confidence intervals of utility inputs. Changes in LYs and QALYs from tested interventions were compared across models. Additional analyses were conducted postchallenge to investigate drivers of cross-model differences.

RESULTS

Substantial cross-model variability in incremental LYs and QALYs was observed, particularly for HbA1c and body mass index (BMI) intervention profiles. For a 0.5%-point permanent HbA1c reduction, LY gains ranged from 0.050 to 0.750. For a 1-unit permanent BMI reduction, incremental QALYs varied from a small decrease in QALYs (-0.024) to an increase of 0.203. Changes in utility values of health states had a much smaller impact (to the hundredth of a decimal place) on incremental QALYs. Microsimulation models were found to generate a mean of 3.41 more LYs than cohort simulation models (P = 0.049).

CONCLUSIONS

Variations in utility values contribute to a lesser extent than uncertainty captured as structural uncertainty. These findings reinforce the importance of assessing structural uncertainty thoroughly because the choice of model (or models) can influence study results, which can serve as evidence for resource allocation decisions.HighlightsThe findings indicate substantial cross-model variability in QALY predictions for a standardized set of simulation scenarios and is considerably larger than within model variability to alternative health state utility values (e.g., lower and upper limits of the 95% confidence intervals of utility inputs).There is a need to understand and assess structural uncertainty, as the choice of model to inform resource allocation decisions can matter more than the choice of health state utility values.

Macrovascular Risk Equations Based on the CANVAS Program

BACKGROUND

Widely used risk equations for cardiovascular outcomes for individuals with type 2 diabetes mellitus (T2DM) have been incapable of predicting cardioprotective effects observed in recent cardiovascular outcomes trials (CVOTs) involving individuals with T2DM at high risk for or with established cardiovascular disease (CVD).

OBJECTIVE

We developed cardiovascular and mortality risk equations using patient-level data from the CANVAS (CANagliflozin cardioVascular Assessment Study) Program to address this shortcoming.

METHODS

Data from 10,142 patients with T2DM at high risk for or with established CVD, randomized to canagliflozin + standard of care (SoC) or SoC alone and followed for a mean duration of 3.6 years in the CANVAS Program were used to derive parametric risk equations for myocardial infarction (MI), stroke, hospitalization for heart failure (HHF), and death. Accumulated knowledge from the widely used UKPDS-OM2 (United Kingdom Prospective Diabetes Study Outcomes Model 2) was leveraged, and any departures in parameterization were limited to those necessary to provide adequate goodness of fit. Candidate explanatory covariates were selected using only the placebo arm to minimize confounding effects. Internal validation was performed separately by study treatment arm.

RESULTS

UKPDS-OM2 predicted CANVAS Program outcomes poorly. Recalibrating UKPDS-OM2 intercepts improved calibration in some cases. Refitting the coefficients but otherwise preserving the UKPDS-OM2 structure improved the fit substantially, which was sufficient for stroke and death. For MI, reselecting UKPDS-OM2 covariates and functional form proved sufficient. For HHF, selection from a broad set of candidate covariates and inclusion of a canagliflozin indicator was required.

CONCLUSION

These risk equations address some of the limitations of widely used risk equations, such as the UKPDS-OM2, for modeling cardioprotective treatments for individuals with T2DM and high cardiovascular risk, including derivation from overly healthy patients treated with agents that lack cardioprotection and have been described as reflecting a different therapeutic era. Future work is needed to examine external validity.

Replication of Published Health Economic Obesity Models: Assessment of Facilitators, Hurdles and Reproduction Success

OBJECTIVES

This research aims to (1) replicate published health economic models, (2) compare reproduced results with original results, (3) identify facilitators and hurdles to model replicability and determine reproduction success, and (4) suggest model replication reporting standards to enhance model reproducibility, in the context of health economic obesity models.

METHODS

Four health economic obesity models simulating an adult UK population were identified, selected for replication, and evaluated using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist. Reproduction results were compared to original results, focusing on cost-effectiveness outcomes, and the resulting reproduction success was assessed by published criteria. Replication facilitators and hurdles were identified and transferred into related reporting standards.

RESULTS

All four case studies were state-transition models simulating costs and quality-adjusted life-years (QALYs). Comparing original versus reproduction outcomes, the following deviation ranges were observed: costs - 3.9 to 16.1% (mean over all model simulations 3.78%), QALYs - 3.7 to 2.1% (mean - 0.11%), and average cost-utility ratios - 3.0 to 17.9% (mean 4.28%). Applying different published criteria, an overall reproduction success was observed for three of four models. Key replication facilitators were input data tables and model diagrams, while missing standard deviations and missing formulas for equations were considered as key hurdles.

CONCLUSIONS

This study confirms the feasibility of rebuilding health economic obesity models, but minor to major assumptions were needed to fill reporting gaps. Model replications can help to assess the quality of health economic model documentation and can be used to validate current model reporting practices. Simple changes to actual CHEERS reporting criteria may solve identified replication hurdles.

Evaluating the Ability of Economic Models of Diabetes to Simulate New Cardiovascular Outcomes Trials: A Report on the Ninth Mount Hood Diabetes Challenge

OBJECTIVES

The cardiovascular outcomes challenge examined the predictive accuracy of 10 diabetes models in estimating hard outcomes in 2 recent cardiovascular outcomes trials (CVOTs) and whether recalibration can be used to improve replication.

METHODS

Participating groups were asked to reproduce the results of the Empagliflozin Cardiovascular Outcome Event Trial in Type 2 Diabetes Mellitus Patients (EMPA-REG OUTCOME) and the Canagliflozin Cardiovascular Assessment Study (CANVAS) Program. Calibration was performed and additional analyses assessed model ability to replicate absolute event rates, hazard ratios (HRs), and the generalizability of calibration across CVOTs within a drug class.

RESULTS

Ten groups submitted results. Models underestimated treatment effects (ie, HRs) using uncalibrated models for both trials. Calibration to the placebo arm of EMPA-REG OUTCOME greatly improved the prediction of event rates in the placebo, but less so in the active comparator arm. Calibrating to both arms of EMPA-REG OUTCOME individually enabled replication of the observed outcomes. Using EMPA-REG OUTCOME-calibrated models to predict CANVAS Program outcomes was an improvement over uncalibrated models but failed to capture treatment effects adequately. Applying canagliflozin HRs directly provided the best fit.

CONCLUSIONS

The Ninth Mount Hood Diabetes Challenge demonstrated that commonly used risk equations were generally unable to capture recent CVOT treatment effects but that calibration of the risk equations can improve predictive accuracy. Although calibration serves as a practical approach to improve predictive accuracy for CVOT outcomes, it does not extrapolate generally to other settings, time horizons, and comparators. New methods and/or new risk equations for capturing these CV benefits are needed.

A Patient-Level Model to Estimate Lifetime Health Outcomes of Patients With Type 1 Diabetes

OBJECTIVE

To develop a patient-level simulation model for predicting lifetime health outcomes of patients with type 1 diabetes and as a tool for economic evaluation of type 1 diabetes treatment based on data from a large, longitudinal cohort.

RESEARCH DESIGN AND METHODS

Data for model development were obtained from the Swedish National Diabetes Register. We derived parametric proportional hazards models predicting the absolute risk of diabetes complications and death based on a wide range of clinical variables and history of complications. We used linear regression models to predict risk factor progression. Internal validation was performed, estimates of life expectancies for different age-sex strata were computed, and the impact of key risk factors on life expectancy was assessed.

RESULTS

The study population consisted of 27,841 patients with type 1 diabetes with a mean duration of follow-up of 7 years. Internal validation showed good agreement between the predicted and observed cumulative incidence of death and 10 complications. Simulated life expectancy was ∼13 years lower than that of the sex- and age-matched general population, and patients with type 1 diabetes could expect to live with one or more complications for ∼40% of their remaining life. Sensitivity analysis showed the importance of preventing renal dysfunction, hypoglycemia, and hyperglycemia as well as lowering HbA_1c in reducing the risk of complications and death.

CONCLUSIONS

Our model was able to simulate risk factor progression and event histories that closely match the observed outcomes and to project events occurring over patients' lifetimes. The model can serve as a tool to estimate the impact of changing clinical risk factors on health outcomes to inform economic evaluations of interventions in type 1 diabetes.

Economic Simulation Modeling in Type 2 Diabetes

PURPOSE OF REVIEW

This paper provides an overview of type 2 diabetes economic simulation modeling and reviews current topics of discussion and major challenges in the field.

RECENT FINDINGS

Important challenges in the field include increasing the generalizability of models and improving transparency in model reporting. To identify and address these issues, modeling groups have organized through the Mount Hood Diabetes Challenge meetings and developed tools (i.e., checklist, impact inventory) to standardize modeling methods and reporting of results. Accordingly, many newer diabetes models have begun utilizing these tools, allowing for improved comparability between diabetes models. In the last two decades, type 2 diabetes simulation models have improved considerably, due to the collaborative work performed through the Mount Hood Diabetes Challenge meetings. To continue to improve diabetes models, future work must focus on clarifying diabetes progression in racial/ethnic minorities and incorporating equity considerations into health economic analysis.

Cost-effectiveness of intensive interventions compared to standard care in individuals with type 2 diabetes: A systematic review and critical appraisal of decision-analytic models

AIMS

The objective of this systematic review is to identify and assess the quality of published decision-analytic models evaluating the long-term cost-effectiveness of target-driven intensive interventions for single and multifactorial risk factor control compared to standard care in people with type 2 diabetes.

METHODS

We searched the electronic databases MEDLINE, the National Health Service Economic Evaluation Database, Web of Science and the Cochrane Library from inception to October 31, 2019. Articles were eligible for inclusion if the studies had used a decision-analytic model evaluating both the long-term costs and benefits associated with intensive interventions for risk factor control compared to standard care in people with type 2 diabetes. Data were extracted using a standardised form, while quality was assessed using the decision-analytic model-specific Philips-criteria.

RESULTS

Overall, nine articles (11 models) were identified, four models evaluated intensive glycaemic control, three evaluated intensive blood pressure control, two evaluated intensive lipid control, and two evaluated intensive multifactorial interventions. Six reported using discrete-time simulations modelling approach, whereas five reported using a Markov modelling framework. The majority, seven studies, reported that the intensive interventions were dominant or cost-effective, given the assumptions and analytical perspective taken. The methodological and reporting quality of the studies was generally weak, with only four studies fulfilling more than 50% of their applicable Philips-criteria.

CONCLUSIONS

This is the first systematic review of decision-analytic models of target-driven intensive interventions for single and multifactorial risk factor control in individuals with type 2 diabetes. Identified shortcomings are lack of transparency in data identification and evidence synthesis as well as for the selection of the modelling approaches. Future models should aim to include greater evaluation of the quality of the data sources used and the assessment of uncertainty in the model.

The Challenge of Transparency and Validation in Health Economic Decision Modelling: A View from Mount Hood

Transparency in health economic decision modelling is important for engendering confidence in the models and in the reliability of model-based cost-effectiveness analyses. The Mount Hood Diabetes Challenge Network has taken a lead in promoting transparency through validation with biennial conferences in which diabetes modelling groups meet to compare simulated outcomes of pre-specified scenarios often based on the results of pivotal clinical trials. Model registration is a potential method for promoting transparency, while also reducing the duplication of effort. An important network initiative is the ongoing construction of a diabetes model registry (https://www.mthooddiabeteschallenge.com). Following the 2012 International Society for Pharmacoeconomics and Outcomes Research and the Society of Medical Decision Making (ISPOR-SMDM) guidelines, we recommend that modelling groups provide technical and non-technical documentation sufficient to enable model reproduction, but not necessarily provide the model code. We also request that modelling groups upload documentation on the methods and outcomes of validation efforts, and run reference case simulations so that model outcomes can be compared. In this paper, we discuss conflicting definitions of transparency in health economic modelling, and describe the ongoing development of a registry of economic models for diabetes through the Mount Hood Diabetes Challenge Network, its objectives and potential further developments, and highlight the challenges in its construction and maintenance. The support of key stakeholders such as decision-making bodies and journals is key to ensuring the success of this and other registries. In the absence of public funding, the development of a network of modellers is of huge value in enhancing transparency, whether through registries or other means.

Cost-effectiveness analysis of a cluster-randomized, culturally tailored, community health worker home-visiting diabetes intervention versus standard care in American Samoa

BACKGROUND

Type 2 diabetes mellitus (T2DM) is highly prevalent in American Samoa. Community health worker (CHW) interventions may improve T2DM care and be cost-effective. Current cost-effectiveness analyses (CEA) of CHW interventions have either overlooked important cost considerations or not been based on randomized clinical trials (RCTs). The Diabetes Care in American Samoa (DCAS) intervention which occurred in 2009-2010 was a cluster-randomized, culturally tailored, home-visiting CHW intervention and improved HbA1c levels.

OBJECTIVE

To analyze the cost-effectiveness of the DCAS intervention against standard care using a RCT in a low-resource setting.

METHODS

We collected clinical, utilization, and cost data over 2 years and modeled quality-adjusted life years (QALYs) gained based on the RCT glycated hemoglobin (HbA1c) improvements. We calculated an incremental cost-effectiveness ratio (ICER) from the societal perspective over a 2-year time horizon and reported all costs in 2012 USD ($).

RESULTS

Two hundred sixty-eight American Samoans diagnosed with T2DM were cluster randomized into the CHW (n = 104) or standard care control (n = 164) arms. The CHW arm had a mean reduction of 0.53% in HbA1c, an increase of $594 in cost, and an increase of 0.05 QALYs. The ICER for the CHW arm compared to the control arm was $1121 per percentage point HbA1c reduced and $13 191 per QALY gained.

CONCLUSIONS

Compared to a variety of willingness-to-pay thresholds from $39 000 to $154 353 per QALY gained, this ICER shows that the CHW intervention is highly cost-effective. Future studies of the cost-effectiveness of CHW T2DM interventions in similar settings should model lifetime costs and QALYs gained to better assess long-term cost-effectiveness.

TRIAL REGISTRATION

ClinicalTrials.gov , ID NCT00850824. Registered 9 February 2009, https://clinicaltrials.gov/ct2/show/NCT00850824 .

Computer Modeling of Diabetes and Its Transparency: A Report on the Eighth Mount Hood Challenge

OBJECTIVES

The Eighth Mount Hood Challenge (held in St. Gallen, Switzerland, in September 2016) evaluated the transparency of model input documentation from two published health economics studies and developed guidelines for improving transparency in the reporting of input data underlying model-based economic analyses in diabetes.

METHODS

Participating modeling groups were asked to reproduce the results of two published studies using the input data described in those articles. Gaps in input data were filled with assumptions reported by the modeling groups. Goodness of fit between the results reported in the target studies and the groups' replicated outputs was evaluated using the slope of linear regression line and the coefficient of determination (R²). After a general discussion of the results, a diabetes-specific checklist for the transparency of model input was developed.

RESULTS

Seven groups participated in the transparency challenge. The reporting of key model input parameters in the two studies, including the baseline characteristics of simulated patients, treatment effect and treatment intensification threshold assumptions, treatment effect evolution, prediction of complications and costs data, was inadequately transparent (and often missing altogether). Not surprisingly, goodness of fit was better for the study that reported its input data with more transparency. To improve the transparency in diabetes modeling, the Diabetes Modeling Input Checklist listing the minimal input data required for reproducibility in most diabetes modeling applications was developed.

CONCLUSIONS

Transparency of diabetes model inputs is important to the reproducibility and credibility of simulation results. In the Eighth Mount Hood Challenge, the Diabetes Modeling Input Checklist was developed with the goal of improving the transparency of input data reporting and reproducibility of diabetes simulation model results.

External validation of type 2 diabetes computer simulation models: definitions, approaches, implications and room for improvement-a protocol for a systematic review

BACKGROUND

Type 2 diabetes mellitus (T2DM), a highly prevalent chronic disease, puts a large burden on individual health and health care systems. Computer simulation models, used to evaluate the clinical and economic effectiveness of various interventions to handle T2DM, have become a well-established tool in diabetes research. Despite the broad consensus about the general importance of validation, especially external validation, as a crucial instrument of assessing and controlling for the quality of these models, there are no systematic reviews comparing such validation of diabetes models. As a result, the main objectives of this systematic review are to identify and appraise the different approaches used for the external validation of existing models covering the development and progression of T2DM.

METHODS

We will perform adapted searches by applying respective search strategies to identify suitable studies from 14 electronic databases. Retrieved study records will be included or excluded based on predefined eligibility criteria as defined in this protocol. Among others, a publication filter will exclude studies published before 1995. We will run abstract and full text screenings and then extract data from all selected studies by filling in a predefined data extraction spreadsheet. We will undertake a descriptive, narrative synthesis of findings to address the study objectives. We will pay special attention to aspects of quality of these models in regard to the external validation based upon ISPOR and ADA recommendations as well as Mount Hood Challenge reports. All critical stages within the screening, data extraction and synthesis processes will be conducted by at least two authors. This protocol adheres to PRISMA and PRISMA-P standards.

DISCUSSION

The proposed systematic review will provide a broad overview of the current practice in the external validation of models with respect to T2DM incidence and progression in humans built on simulation techniques.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42017069983 .

Validation of the Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM)

BACKGROUND

The Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM) was developed to address study questions pertaining to the cost-effectiveness of treatment alternatives in the care of patients with type 2 diabetes mellitus (T2DM). Naturally, the usefulness of a model is determined by the accuracy of its predictions. A previous version of ECHO-T2DM was validated against actual trial outcomes and the model predictions were generally accurate. However, there have been recent upgrades to the model, which modify model predictions and necessitate an update of the validation exercises.

OBJECTIVES

The objectives of this study were to extend the methods available for evaluating model validity, to conduct a formal model validation of ECHO-T2DM (version 2.3.0) in accordance with the principles espoused by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision Making (SMDM), and secondarily to evaluate the relative accuracy of four sets of macrovascular risk equations included in ECHO-T2DM.

METHODS

We followed the ISPOR/SMDM guidelines on model validation, evaluating face validity, verification, cross-validation, and external validation. Model verification involved 297 'stress tests', in which specific model inputs were modified systematically to ascertain correct model implementation. Cross-validation consisted of a comparison between ECHO-T2DM predictions and those of the seminal National Institutes of Health model. In external validation, study characteristics were entered into ECHO-T2DM to replicate the clinical results of 12 studies (including 17 patient populations), and model predictions were compared to observed values using established statistical techniques as well as measures of average prediction error, separately for the four sets of macrovascular risk equations supported in ECHO-T2DM. Sub-group analyses were conducted for dependent vs. independent outcomes and for microvascular vs. macrovascular vs. mortality endpoints.

RESULTS

All stress tests were passed. ECHO-T2DM replicated the National Institutes of Health cost-effectiveness application with numerically similar results. In external validation of ECHO-T2DM, model predictions agreed well with observed clinical outcomes. For all sets of macrovascular risk equations, the results were close to the intercept and slope coefficients corresponding to a perfect match, resulting in high R ² and failure to reject concordance using an F test. The results were similar for sub-groups of dependent and independent validation, with some degree of under-prediction of macrovascular events.

CONCLUSION

ECHO-T2DM continues to match health outcomes in clinical trials in T2DM, with prediction accuracy similar to other leading models of T2DM.

How Consistent is the Relationship between Improved Glucose Control and Modelled Health Outcomes for People with Type 2 Diabetes Mellitus? a Systematic Review

BACKGROUND

There are an increasing number of studies using simulation models to conduct cost-effectiveness analyses for type 2 diabetes mellitus.

OBJECTIVE

To evaluate the relationship between improvements in glycosylated haemoglobin (HbA_1c) and simulated health outcomes in type 2 diabetes cost-effectiveness studies.

METHODS

A systematic review was conducted on MEDLINE and EMBASE to collect cost-effectiveness studies using type 2 diabetes simulation models that reported modelled health outcomes of blood glucose-related interventions in terms of quality-adjusted life-years (QALYs) or life expectancy (LE). The data extracted included information used to characterise the study cohort, the intervention's treatment effects on risk factors and model outcomes. Linear regressions were used to test the relationship between the difference in HbA_1c (∆HbA_1c) and incremental QALYs (∆QALYs) or LE (∆LE) of intervention and control groups. The ratio between the ∆QALYs and ∆LE was calculated and a scatterplot between the ratio and ∆HbA_1c was used to explore the relationship between these two.

RESULTS

Seventy-six studies were included in this research, contributing to 124 pair of comparators. The pooled regressions indicated that the marginal effect of a 1% HbA_1c decrease in intervention resulted in an increase in life-time QALYs and LE of 0.371 (95% confidence interval 0.286-0.456) and 0.642 (95% CI 0.494-0.790), respectively. No evidence of heterogeneity between models was found. An inverse exponential relationship was found and fitted between the ratio (∆QALY/∆LE) and ∆HbA_1c.

CONCLUSION

There is a consistent relationship between ∆HbA_1c and ∆QALYs or ∆LE in cost-effectiveness analyses using type 2 diabetes simulation models. This relationship can be used as a diagnostic tool for decision makers.

Integrated sensor-augmented pump therapy systems [the MiniMed® Paradigm™ Veo system and the Vibe™ and G4® PLATINUM CGM (continuous glucose monitoring) system] for managing blood glucose levels in type 1 diabetes: a systematic review and economic evaluation

BACKGROUND

In recent years, meters for continuous monitoring of interstitial fluid glucose have been introduced to help people with type 1 diabetes mellitus (T1DM) to achieve better control of their disease.

OBJECTIVE

The objective of this project was to summarise the evidence on the clinical effectiveness and cost-effectiveness of the MiniMed(®) Paradigm™ Veo system (Medtronic Inc., Northridge, CA, USA) and the Vibe™ (Animas(®) Corporation, West Chester, PA, USA) and G4(®) PLATINUM CGM (continuous glucose monitoring) system (Dexcom Inc., San Diego, CA, USA) in comparison with multiple daily insulin injections (MDIs) or continuous subcutaneous insulin infusion (CSII), both with either self-monitoring of blood glucose (SMBG) or CGM, for the management of T1DM in adults and children.

DATA SOURCES

A systematic review was conducted in accordance with the principles of the Centre for Reviews and Dissemination guidance and the National Institute for Health and Care Excellence Diagnostic Assessment Programme manual. We searched 14 databases, three trial registries and two conference proceedings from study inception up to September 2014. In addition, reference lists of relevant systematic reviews were checked. In the absence of randomised controlled trials directly comparing Veo or an integrated CSII + CGM system, such as Vibe, with comparator interventions, indirect treatment comparisons were performed if possible.

METHODS

A commercially available cost-effectiveness model, the IMS Centre for Outcomes Research and Effectiveness diabetes model version 8.5 (IMS Health, Danbury, CT, USA), was used for this assessment. This model is an internet-based, interactive simulation model that predicts the long-term health outcomes and costs associated with the management of T1DM and type 2 diabetes. The model consists of 15 submodels designed to simulate diabetes-related complications, non-specific mortality and costs over time. As the model simulates individual patients over time, it updates risk factors and complications to account for disease progression.

RESULTS

Fifty-four publications resulting from 19 studies were included in the review. Overall, the evidence suggests that the Veo system reduces hypoglycaemic events more than other treatments, without any differences in other outcomes, including glycated haemoglobin (HbA1c) levels. We also found significant results in favour of the integrated CSII + CGM system over MDIs with SMBG with regard to HbA1c levels and quality of life. However, the evidence base was poor. The quality of the included studies was generally low, often with only one study comparing treatments in a specific population at a specific follow-up time. In particular, there was only one study comparing Veo with an integrated CSII + CGM system and only one study comparing Veo with a CSII + SMBG system in a mixed population. Cost-effectiveness analyses indicated that MDI + SMBG is the option most likely to be cost-effective, given the current threshold of £30,000 per quality-adjusted life-year gained, whereas integrated CSII + CGM systems and Veo are dominated and extendedly dominated, respectively, by stand-alone, non-integrated CSII with CGM. Scenario analyses did not alter these conclusions. No cost-effectiveness modelling was conducted for children or pregnant women.

CONCLUSIONS

The Veo system does appear to be better than the other systems considered at reducing hypoglycaemic events. However, in adults, it is unlikely to be cost-effective. Integrated systems are also generally unlikely to be cost-effective given that stand-alone systems are cheaper and, possibly, no less effective. However, evidence in this regard is generally lacking, in particular for children. Future trials in specific child, adolescent and adult populations should include longer term follow-up and ratings on the European Quality of Life-5 Dimensions scale at various time points with a view to informing improved cost-effectiveness modelling.

STUDY REGISTRATION

PROSPERO Registration Number CRD42014013764.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Validation of the UKPDS 82 risk equations within the Cardiff Diabetes Model

BACKGROUND

For end-users of diabetes models that include UKPDS 82 risk equations, an important question is how well these new equations perform. Consequently, the principal objective of this study was to validate the UKPDS 82 risk equations, embedded within an established type 2 diabetes mellitus (T2DM) model, the Cardiff Diabetes Model, to contemporary T2DM outcomes studies.

METHODS

A total of 100 validation endpoints were simulated across treatment arms of twelve pivotal T2DM outcomes studies, simulation cohorts representing each validation study's cohort profile were generated and intensive and conventional treatment arms were defined in the Cardiff Diabetes Model.

RESULTS

Overall the validation coefficient of determination was similar between both sets of risk equations: UKPDS 68, R(2) = 0.851; UKPDS 82, R(2) = 0.870. Results stratified by internal and external validation studies produced MAPE of 43.77 and 37.82%, respectively, when using UKPDS 82, and MAPE of 40.49 and 53.92%, respectively when using UKPDS 68. Areas of lack of fit, as measured by MAPE were inconsistent between sets of equations with ACCORD demonstrating a noteworthy lack of fit with UKPPDS 68 (MAPE = 170.88%) and the ADDITION study for UKPDS 82 (MAPE = 89.90%).

CONCLUSIONS

This study has demonstrated that the UKPDS 82 equations exhibit similar levels of external validity to the UKPDS 68 equations with the additional benefit of enabling more diabetes related endpoints to be modeled.

Screening for and treatment of osteoporosis: construction and validation of a state-transition microsimulation cost-effectiveness model

This study aimed to document and validate a new cost-effectiveness model of osteoporosis screening and treatment strategies. The state-transition microsimulation model demonstrates strong internal and external validity. It is an important tool for researchers and policy makers to test the cost-effectiveness of osteoporosis screening and treatment strategies.

INTRODUCTION

The objective of this study was to document and validate a new cost-effectiveness model of screening for and treatment of osteoporosis.

METHODS

A state-transition microsimulation model using a lifetime horizon was constructed with seven Markov states (no history of fractures, hip fracture, vertebral fracture, wrist fracture, other fracture, postfracture state, and death) describing the most important clinical outcomes of osteoporotic fractures. Tracker variables were used to record patients' history, such as fracture events, duration of treatment, and time since last screening. The model was validated for Chinese postmenopausal women receiving screening and treatment versus no screening. Goodness-of-fit analyses were performed for internal and external validation. External validity was tested by comparing life expectancy, osteoporosis prevalence rate, and lifetime and 10-year fracture risks with published data not used in the model.

RESULTS

The model represents major clinical facets of osteoporosis-related conditions. Age-specific hip, vertebral, and wrist fracture incidence rates were accurately reproduced (the regression line slope was 0.996, R(2) = 0.99). The changes in costs, effectiveness, and cost-effectiveness were consistent with changes in both one-way and probabilistic sensitivity analysis. The model predicted life expectancy and 10-year any major osteoporotic fracture risk at the age of 65 of 19.01 years and 13.7%, respectively. The lifetime hip, clinical vertebral, and wrist fracture risks at age 50 were 7.9, 29.8, and 18.7% respectively, all consistent with reported data.

CONCLUSIONS

Our model demonstrated good internal and external validity, ensuring it can be confidently applied in economic evaluations of osteoporosis screening and treatment strategies.

Validation of the IMS CORE Diabetes Model

BACKGROUND

The IMS CORE Diabetes Model (CDM) is a widely published and validated simulation model applied in both type 1 diabetes mellitus (T1DM) and type 2 diabetes mellitus (T2DM) analyses. Validation to external studies is an important part of demonstrating model credibility.

OBJECTIVE

Because the CDM is widely used to estimate long-term clinical outcomes in diabetes patients, the objective of this analysis was to validate the CDM to contemporary outcomes studies, including those with long-term follow-up periods.

METHODS

A total of 112 validation simulations were performed, stratified by study follow-up duration. For long-term results (≥15-year follow-up), simulation cohorts representing baseline Diabetes Control and Complications Trial (DCCT) and United Kingdom Prospective Diabetes Study (UKPDS) cohorts were generated and intensive and conventional treatment arms were defined in the CDM. Predicted versus observed macrovascular and microvascular complications and all-cause mortality were assessed using the coefficient of determination (R(2)) goodness-of-fit measure.

RESULTS

Across all validation studies, the CDM simulations produced an R(2) statistic of 0.90. For validation studies with a follow-up duration of less than 15 years, R(2) values of 0.90 and 0.88 were achieved for T1DM and T2DM respectively. In T1DM, validating against 30-year outcomes data (DCCT) resulted in an R(2) of 0.72. In T2DM, validating against 20-year outcomes data (UKPDS) resulted in an R(2) of 0.92.

CONCLUSIONS

This analysis supports the CDM as a credible tool for predicting the absolute number of clinical events in DCCT- and UKPDS-like populations. With increasing incidence of diabetes worldwide, the CDM is particularly important for health care decision makers, for whom the robust evaluation of health care policies is essential.

Validation of economic and health outcomes simulation model of type 2 diabetes mellitus (ECHO-T2DM)

OBJECTIVE

This study constructed the Economic and Health Outcomes Model for type 2 diabetes mellitus (ECHO-T2DM), a long-term stochastic microsimulation model, to predict the costs and health outcomes in patients with T2DM. Naturally, the usefulness of the model depends upon its predictive accuracy. The objective of this work is to present results of a formal validation exercise of ECHO-T2DM.

METHODS

The validity of ECHO-T2DM was assessed using criteria recommended by the International Society for Pharmacoeconomics and Outcomes Research/Society for Medical Decision Making (ISPOR/SMDM). Specifically, the results of a number of clinical trials were predicted and compared with observed study end-points using a scatterplot and regression approach. An F-test of the best-fitting regression was added to assess whether it differs statistically from the identity (45°) line defining perfect predictions. In addition to testing the full model using all of the validation study data, tests were also performed of microvascular, macrovascular, and survival outcomes separately. The validation tests were also performed separately by type of data (used vs not used to construct the model, economic simulations, and treatment effects).

RESULTS

The intercept and slope coefficients of the best-fitting regression line between the predicted outcomes and corresponding trial end-points in the main analysis were -0.0011 and 1.067, respectively, and the R(2) was 0.95. A formal F-test of no difference between the fitted line and the identity line could not be rejected (p = 0.16). The high R(2) confirms that the data points are closely (and linearly) associated with the fitted regression line. Additional analyses identified that disagreement was highest for macrovascular end-points, for which the intercept and slope coefficients were 0.0095 and 1.225, respectively. The R(2) was 0.95 and the estimated intercept and slope coefficients were 0.017 and 1.048, respectively, for mortality, and the F-test was narrowly rejected (p = 0.04). The sub-set of microvascular end-points showed some tendency to over-predict (the slope coefficient was 1.095), although concordance between predictions and observed values could not be rejected (p = 0.16).

LIMITATIONS

Important study limitations include: (1) data availability limited one to tests based on end-of-study outcomes rather than time-varying outcomes during the studies analyzed; (2) complex inclusion and exclusion criteria in two studies were difficult to replicate; (3) some of the studies were older and reflect outdated treatment patterns; and (4) the authors were unable to identify published data on resource use and costs of T2DM suitable for testing the validity of the economic calculations.

CONCLUSIONS

Using conventional methods, ECHO-T2DM simulated the treatment, progression, and patient outcomes observed in important clinical trials with an accuracy consistent with other well-accepted models. Macrovascular outcomes were over-predicted, which is common in health-economic models of diabetes (and may be related to a general over-prediction of event rates in the United Kingdom Prospective Diabetes Study [UKPDS] Outcomes Model). Work is underway in ECHO-T2DM to incorporate new risk equations to improve model prediction.

Computer modeling of diabetes and its complications: a report on the Fifth Mount Hood challenge meeting

OBJECTIVES

The Mount Hood Challenge meetings provide a forum for computer modelers of diabetes to discuss and compare models, to assess predictions against data from clinical trials and other studies, and to identify key future developments in the field. This article reports the proceedings of the Fifth Mount Hood Challenge in 2010.

METHODS

Eight modeling groups participated. Each group was given four modeling challenges to perform (in type 2 diabetes): to simulate a trial of a lipid-lowering intervention (The Atorvastatin Study for Prevention of Coronary Heart Disease Endpoints in Non-Insulin-Dependent Diabetes Mellitus [ASPEN]), to simulate a trial of a blood glucose-lowering intervention (Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation [ADVANCE]), to simulate a trial of a blood pressure-lowering intervention (Cardiovascular Risk in Diabetes [ACCORD]), and (optional) to simulate a second trial of blood glucose-lowering therapy (ACCORD). Model outcomes for each challenge were compared with the published findings of the respective trials.

RESULTS

The results of the models varied from each other and, in some cases, from the published trial data in important ways. In general, the models performed well in terms of predicting the relative benefit of interventions, but performed less well in terms of quantifying the absolute risk of complications in patients with type 2 diabetes. Methodological challenges were highlighted including matching trial end-point definitions, the importance of assumptions concerning the progression of risk factors over time, and accurately matching the patient characteristics from each trial.

CONCLUSIONS

The Fifth Mount Hood Challenge allowed modelers, through systematic comparison and validation exercises, to identify important differences between models, address key methodological challenges, and discuss avenues of research to improve future diabetes models.

Dynamic microsimulation models for health outcomes: a review

BACKGROUND

Microsimulation models (MSMs) for health outcomes simulate individual event histories associated with key components of a disease process; these simulated life histories can be aggregated to estimate population-level effects of treatment on disease outcomes and the comparative effectiveness of treatments. Although MSMs are used to address a wide range of research questions, methodological improvements in MSM approaches have been slowed by the lack of communication among modelers. In addition, there are few resources to guide individuals who may wish to use MSM projections to inform decisions.

METHODS

. This article presents an overview of microsimulation modeling, focusing on the development and application of MSMs for health policy questions. The authors discuss MSM goals, overall components of MSMs, methods for selecting MSM parameters to reproduce observed or expected results (calibration), methods for MSM checking (validation), and issues related to reporting and interpreting MSM findings(sensitivity analyses, reporting of variability, and model transparency).

CONCLUSIONS

. MSMs are increasingly being used to provide information to guide health policy decisions. This increased use brings with it the need for both better understanding of MSMs by policy researchers, and continued improvement in methods for developing and applying MSMs.

The CHD challenge: comparing four cost-effectiveness models

OBJECTIVES

To compare four UK models evaluating the cost-effectiveness of interventions in coronary heart disease (CHD), exploring the relative importance of structure and inputs in accounting for differences, and the scope for consensus on structure and data.

METHODS

We compared published cost-effectiveness results (incremental cost, quality-adjusted life year, and cost-effectiveness ratio) of three models conforming to the National Institute for Health and Clinical Excellence guidelines dealing with three interventions (statins, percutaneous coronary intervention, and clopidogrel) with a model developed in Southampton. Comparisons were made using three separate stages: 1) comparison of published results; 2) comparison of the results using the same data inputs wherever possible; and 3) an in-depth exploration of reasons for differences and the potential for consensus.

RESULTS

Although published results differed by up to 73% (for statins), standardization of inputs (stage 2) narrowed these gaps. Greater understanding of the reasons for differences was achieved, but a consensus on preferred values for all data inputs was not reached.

CONCLUSIONS

We found that published guidance on methods was important to reduce variation in important model inputs. Although the comparison of models did not lead to consensus for all model inputs, it provided a better understanding of the reasons for these differences, and enhanced the transparency and credibility of all models. Similar comparisons would be aided by fuller publication of models, perhaps through detailed web appendices.

Structural differences among cost-effectiveness models of human papillomavirus vaccines

In this article we compare previously published cost-effectiveness studies of human papillomavirus (HPV) vaccines along a defined subset of key model structural assumptions relating to HPV infection and disease, cervical cancer screening and HPV vaccination. For each structural aspect examined, we summarize assumptions from each study, provide a critical review and discuss the impact upon results. Considerable variation was observed across HPV vaccine cost-effectiveness models in a number of influential assumptions. Holding constant factors for which current data are lacking, the combined impact of assumptions made for the remaining parameters examined would appear to tend toward underestimation of the cost-effectiveness of HPV vaccination within existing studies. However, uncertainty concerning parameters, such as the duration of vaccine protection and acquired immunity following HPV infection, and the relationship between age and HPV virulence, complicates precise estimation of the cost-effectiveness of HPV vaccination and rigorous evaluation of the validity of existing modeling results.

Predicted quality-adjusted life years as a composite measure of the clinical value of diabetes risk factor control

BACKGROUND

Control of blood pressure, low-density lipoprotein cholesterol (LDL-c), and A1c can lower the risk for diabetes complications. These quality indicators often are examined separately and weighted equally in performance measurement, potentially discarding important information.

OBJECTIVES

We sought to create a composite indicator of the clinical benefit, or value, of diabetes risk factor control that appropriately weights the clinical importance of A1c, LDL-c, and blood pressure, and to test its usability for quality measurement.

METHODS

The combined value of control for 3 diabetes risk factors, measured by predicted quality-adjusted life years (QALYs), was compared in diabetes patients (n = 129,236 in 2001; n = 185,006 in 2003) in Kaiser Permanente Northern California across 16 medical center populations in 2001 and 2003 using hierarchical linear regression to adjust for case-mix differences. Patient-level QALYs, simulated from risk factor and case-mix variables in a Markov model, was the main outcome variable.

RESULTS

There was significant cross-sectional variability in average case-mix adjusted QALYs for diabetes patients across centers in 2003. QALYs increased from 2001 to 2003 as the result of improved risk factor control; longitudinal improvements in QALYs also showed variation across centers. Regression analyses demonstrated the greater impact of blood pressure versus LDL-c or A1c control on QALYs, and the greater value of risk factor control in those with poor versus near or in-control blood pressure.

CONCLUSION

Using predicted QALYs to measure value holds promise as a sensitive composite indicator for quality measurement. Complex, evidence-based quality indicators such as these can potentially provide accurate and useful information to health plans, providers, and consumers.

Emerging methods in economic modeling of imaging costs and outcomes a short report on discrete event simulation

RATIONALE AND OBJECTIVES

This short report provides a non-technical overview of one emerging modeling technique, discrete event simulation (DES).

METHODS

A selective review of the literature that has applied DES methods to evaluate imaging technologies.

RESULTS

Mathematical models to evaluate the likely costs and outcomes of health technologies have become increasingly accepted. Increasing experience has also brought a mounting awareness of the limitations of conventional modeling techniques such as decision trees and Markov processes. Patient-level simulation, including DES, may provide a more flexible approach to modeling for economic evaluation of health technologies.

CONCLUSIONS

The strengths of DES suggest that it may have an increasingly important role in the future modeling of annual screening programs, diagnosis, and treatment of chronic recurrent disease and modeling the utilization of imaging equipment.

Crossing the evidence chasm: building evidence bridges from process changes to clinical outcomes

OBJECTIVE

Although demand for information about the effectiveness and efficiency of health care information technology grows, large-scale resource-intensive randomized controlled trials of health care information technology remain impractical. New methods are needed to translate more commonly available clinical process measures into potential impact on clinical outcomes.

DESIGN

The authors propose a method for building mathematical models based on published evidence that provides an evidence bridge between process changes and resulting clinical outcomes. This method combines tools from systematic review, influence diagramming, and health care simulations.

MEASUREMENTS

The authors apply this method to create an evidence bridge between retinopathy screening rates and incidence of blindness in diabetic patients.

RESULTS

The resulting model uses changes in eye examination rates and other evidence-based population parameters to generate clinical outcomes and costs in a Markov model.

CONCLUSION

This method may serve as an alternative to more expensive study designs and provide useful estimates of the impact of health care information technology on clinical outcomes through changes in clinical process measures.

A taxonomy of model structures for economic evaluation of health technologies

Models for the economic evaluation of health technologies provide valuable information to decision makers. The choice of model structure is rarely discussed in published studies and can affect the results produced. Many papers describe good modelling practice, but few describe how to choose from the many types of available models. This paper develops a new taxonomy of model structures. The horizontal axis of the taxonomy describes assumptions about the role of expected values, randomness, the heterogeneity of entities, and the degree of non-Markovian structure. Commonly used aggregate models, including decision trees and Markov models require large population numbers, homogeneous sub-groups and linear interactions. Individual models are more flexible, but may require replications with different random numbers to estimate expected values. The vertical axis of the taxonomy describes potential interactions between the individual actors, as well as how the interactions occur through time. Models using interactions, such as system dynamics, some Markov models, and discrete event simulation are fairly uncommon in the health economics but are necessary for modelling infectious diseases and systems with constrained resources. The paper provides guidance for choosing a model, based on key requirements, including output requirements, the population size, and system complexity.

Development and validation of the Economic Assessment of Glycemic Control and Long-Term Effects of diabetes (EAGLE) model

BACKGROUND

The Economic Assessment of Glycemic control and Long-term Effects of diabetes (EAGLE) model was developed to provide a flexible and comprehensive tool for the simulation of the long-term effects of diabetes treatment and related costs in type 1 and type 2 diabetes.

METHODS

EAGLE simulations are based on risk equations, which were developed using published data from several large studies including the Diabetes Control and Complications Trial, the United Kingdom Prospective Diabetes Study, and the Wisconsin Epidemiological Study of Diabetic Retinopathy. Risk equations for the probability of complications (including hypoglycemia, retinopathy, macular edema, end-stage renal disease, neuropathy, diabetic foot syndrome, myocardial infarction, and stroke) were based on regression analyses, using linear, exponential, and quadratic regression formulae. Subsequent cost calculations are made from the simulated event rates. Internal validation of the EAGLE model was completed by comparing simulated event rates with the published event rates used as the basis for the model.

RESULTS

EAGLE provides microsimulations of virtual patient cohorts for type 1 and type 2 diabetes over n years in 1-year cycles. Complications include microvascular and macrovascular events and death, which are calculated over time as cumulative incidences. Glycosylated hemoglobin levels over time are simulated in relation to treatment regimen. Internal validation demonstrated that each mean event rate simulated by EAGLE overlapped with the published mean event (within a range of +/-10%).

CONCLUSIONS

The EAGLE model is an evidence-based, internally valid tool for the assessment of the long-term effects of diabetes treatment and related costs.

The burden of treatment failure in type 2 diabetes

OBJECTIVE

In type 2 diabetes, therapies to maintain blood glucose control usually fail after several years. We estimated the glycemic burden that accumulates from treatment failure and describe the time course and predictors of failure.

RESEARCH DESIGN AND METHODS

A prospective, population-based study using retrospective observational data. We identified all 7208 complete courses of treatment with nondrug therapy, sulfonylurea monotherapy, metformin monotherapy, and combination oral antihyperglycemic therapy between 1994 and 2002, inclusive, among members of the Kaiser Permanente Northwest Region. We calculated mean cumulative glycemic burden, defined as HbA(1c)-months >8.0 or 7.0% for each treatment. We then measured the likelihood that the next HbA(1c) would exceed 8.0 and 7.0% after HbA(1c) exceeded each of ten hypothetical treatment thresholds. Finally, we estimated multivariate logistic regression models to predict when HbA(1c) would continue to deteriorate.

RESULTS

In this well-controlled population, the average patient accumulated nearly 5 HbA(1c)-years of excess glycemic burden >8.0% from diagnosis until starting insulin and about 10 HbA(1c)-years of burden >7.0%. Whenever patients crossed the American Diabetes Association-recommended treatment threshold of 8.0%, their next HbA(1c) result was as likely to be <8.0 as >8.0%. Multivariate prediction models had highly statistically significant coefficients, but predicted <10% of the variation in future HbA(1c) results.

CONCLUSIONS

Clinicians should change glucose-lowering treatments in type 2 diabetes much sooner or use treatments that are less likely to fail. An action point at 7.0% or lower is more likely to prevent additional deterioration than the traditional action point of 8.0%.

Intensive lifestyle changes or metformin in patients with impaired glucose tolerance: Modeling the long-term health economic implications of the diabetes prevention program in Australia, France, Germany, Switzerland, and the United Kingdom

BACKGROUND

In the Diabetes Prevention Program (DPP), interventions with metformin (plus standard lifestyle advice) or intensive lifestyle changes (ILC) reduced the risk of developing type 2 diabetes mellitus (DM) by 31% and 58%, respectively, versus control (standard lifestyle advice only) in patients with impaired glucose tolerance (IGT).

OBJECTIVE

The goal of this study was to establish whether implementing the active treatments used in the DPP would be cost-effective in Australia, France, Germany, Switzerland, and the United Kingdom.

METHODS

A Markov model simulated 3 states-IGT, type 2 DM, and deceased-using probabilities from the DPP and published data. Country-specific direct costs were used throughout.

RESULTS

Assuming only within-trial effects and costs of interventions, both metformin and ILC improved life expectancy versus control. Mean improvements in nondiscounted life expectancy were 0.11 and 0.22 years for metformin and ILC, respectively. Both interventions were associated with cost savings versus control in all countries except the United Kingdom, where a small increase in costs was observed in both intervention arms. When a lifetime effect of interventions was assumed, incremental improvements in life expectancy were 0.35 and 0.90 years for metformin and ILC, respectively. Results were sensitive to probabilities of developing type 2 DM, the projected long-term duration of effect of interventions after the 3-year trial period, the relative risk of mortality for type 2 DM compared with IGT, and the costs of implementing the interventions.

CONCLUSIONS

Based on probabilities from the DPP and published data, in this model analysis, incorporation of the DPP interventions into clinical practice in 5 developed countries was projected to lead to an increase in DM-free years of life, improvements in life expectancy, and either cost savings or minor increases in costs compared with standard lifestyle advice in a population with IGT. Thus, financial constraints should not prevent the implementation of DM prevention programs.

Validation of the archimedes diabetes model

OBJECTIVE

To validate the Archimedes model of diabetes and its complications for a variety of populations, organ systems, treatments, and outcomes.

RESEARCH DESIGN AND METHODS

We simulated a variety of randomized controlled trials by repeating in the model the steps taken for the real trials and comparing the results calculated by the model with the results of the trial. Eighteen trials were chosen by an independent advisory committee. Half the trials had been used to help build the model ("internal" or "dependent" validations); the other half had not. Those trials comprise "external" or "independent" validations.

RESULTS

A total of 74 validation exercises were conducted involving different treatments and outcomes in the 18 trials. For 71 of the 74 exercises there were no statistically significant differences between the results calculated by the model and the results observed in the trial. Considering only the trials that were never used to help build the model-the independent or external validations-the correlation was r = 0.99. Including all of the exercises, the correlation between the outcomes calculated by the model and the outcomes seen in the trials was r = 0.99. When the absolute differences in outcomes between the control and treatment groups were compared, the correlation coefficient was r = 0.97.

CONCLUSIONS

The Archimedes diabetes model is a realistic representation of the anatomy, pathophysiology, treatments, and outcomes pertinent to diabetes and its complications for applications that involve the populations, treatments, outcomes, and health care settings spanned by the trials.

Determining the Value of Disease Management Programs

BACKGROUND

Increasing prevalence, rising costs, and persisting deficiencies in quality of care for chronic diseases pose economic and policy challenges to providers and purchasers. Disease management (DM) programs may address these challenges, but neither purchasers nor providers can assess their value. The potpourri of current quality indicators provides limited insight into the actual clinical benefit achieved. A conference sponsored by the Agency for Healthcare Research and Quality (AHRQ) and held in October 2002 explored new approaches to measuring and reporting the value of DM for diabetes mellitus.

RESULTS

Quantifying the value of DM requires measuring clinical benefit and net impact on health care costs for the entire population with diabetes. If quality is measured with indicators that are clearly linked to outcomes, clinical benefit can be estimated. Natural history models combine the expected benefits of improvements in multiple indicators to yield a single, composite measure, the quality-adjusted life-year. Such metrics could fairly express, in terms of survival and complications prevention, relatively disparate DM programs' benefits. Measuring and comparing health care costs requires data validation and appropriate case-mix adjustment. Comparing value across programs may provide more accurate assessments of performance, enhance quality improvement efforts within systems, and contribute generalizable knowledge on the utility of DM approaches.

CONCLUSIONS

Conference attendees recommended pilot projects to further explore use of natural history models for measuring and reporting the value of DM.

Diabetic retinopathy: contemporary prevalence in a well-controlled population

OBJECTIVE

To measure the extent to which modern intensified risk factor control has lessened the duration-specific prevalence of diabetic retinopathy and, therefore, has decreased the risk of blindness in Americans with type 2 diabetes.

RESEARCH DESIGN AND METHODS

Intensified control of blood glucose and blood pressure has prevented diabetic retinopathy in randomized controlled trials. There is as yet no confirmation that subsequent treatment intensification in the community has had the same result. We identified all 6993 members of a health maintenance organization, Kaiser Permanente Northwest (KPNW), who, in 1997-1998, had dilated retinal examinations and verifiable data of diagnosis of type 2 diabetes. We plotted prevalence by time since diagnosis for background diabetic retinopathy (BDR) and proliferative diabetic retinopathy (PDR) and compared these results to identically derived 1980-1982 results from the Wisconsin Epidemiologic Study of Diabetic Retinopathy (WESDR). We estimated multivariate predictive models.

RESULTS

Mean (+/- SD) HbA(1c) in KPNW was 7.84 +/- 1.26% versus 10.37% (standardized) in the WESDR. KPNW blood pressure averaged 138.6 +/- 13.8/79.5 +/- 7.4 mmHg compared with 147.0/79.0 in the WESDR. BDR was much less prevalent in KPNW, but PDR prevalence appeared unchanged. BDR preceded diagnosis in 20.8% of the WESDR subjects but only 2.0% of KPNW subjects. However, in both populations, the first cases of PDR appeared similarly, soon after diagnosis.

CONCLUSIONS

Earlier diagnosis and more aggressive control of blood glucose and blood pressure decreased the duration-adjusted prevalence of background, but not of sight-threatening proliferative retinopathy. More population-based research is needed to replicate and explain this unexpected finding. Detecting and treating PDR should not be neglected on the assumption that risk-factor control has minimized its prevalence.

Selecting methods for the prediction of future events in cost-effectiveness models: a decision-framework and example from the cardiovascular field

Evidence on the cost-effectiveness of healthcare interventions is increasingly required by decision-makers. Economic models can provide timely information on the long-term impact of new technologies. However, models have been criticised because of the implicit assumptions they make, in particular the methods used to extrapolate data are rarely documented. This paper presents a systematic process for choosing a method of predicting events in economic models. This process is illustrated using a model examining the cost-effectiveness of a new HMG-CoA reductase inhibitor (statin) for primary prevention of cardiovascular disease (CVD). The prediction of future CVD events is a central component of the model, and the choice of method for predicting events was an important issue in the model's development. A literature review identified 11 studies with the information required to predict CVD events. A set of criteria were developed to assess the different methods of risk estimation, covering issues like scientific validity and acceptability to decision-makers. Risk equations derived from the Framingham Heart Study were found to be most suitable for predicting future events in the economic model. The paper illustrates how the development of economic models can be made more transparent, and suggests that the process outlined may be applied to other disease areas where there are several event prediction methods to choose from. In disease areas where published methods for predicting events are not available, the process outlined can make the uncertainty this leads to explicit, and highlight where further research is required. Such transparency can help decision-makers understand the scientific basis underpinning models, and therefore make these models more acceptable and useful for health policy-making.