Measuring decision sensitivity: a combined Monte Carlo-logistic regression approach

Validating Health Economic Models With the Probabilistic Analysis Check dashBOARD

OBJECTIVES

Health economic (HE) models are often considered as "black boxes" because they are not publicly available and lack transparency, which prevents independent scrutiny of HE models. Additionally, validation efforts and validation status of HE models are not systematically reported. Methods to validate HE models in absence of their full underlying code are therefore urgently needed to improve health policy making. This study aimed to develop and test a generic dashboard to systematically explore the workings of HE models and validate their model parameters and outcomes.

METHODS

The Probabilistic Analysis Check dashBOARD (PACBOARD) was developed using insights from literature, health economists, and a data scientist. Functionalities of PACBOARD are (1) exploring and validating model parameters and outcomes using standardized validation tests and interactive plots, (2) visualizing and investigating the relationship between model parameters and outcomes using metamodeling, and (3) predicting HE outcomes using the fitted metamodel. To test PACBOARD, 2 mock HE models were developed, and errors were introduced in these models, eg, negative costs inputs, utility values exceeding 1. PACBOARD metamodeling predictions of incremental net monetary benefit were validated against the original model's outcomes.

RESULTS

PACBOARD automatically identified all errors introduced in the erroneous HE models. Metamodel predictions were accurate compared with the original model outcomes.

CONCLUSIONS

PACBOARD is a unique dashboard aiming at improving the feasibility and transparency of validation efforts of HE models. PACBOARD allows users to explore the working of HE models using metamodeling based on HE models' parameters and outcomes.

Human behavior in the time of COVID-19: Learning from big data

Since the World Health Organization (WHO) characterized COVID-19 as a pandemic in March 2020, there have been over 600 million confirmed cases of COVID-19 and more than six million deaths as of October 2022. The relationship between the COVID-19 pandemic and human behavior is complicated. On one hand, human behavior is found to shape the spread of the disease. On the other hand, the pandemic has impacted and even changed human behavior in almost every aspect. To provide a holistic understanding of the complex interplay between human behavior and the COVID-19 pandemic, researchers have been employing big data techniques such as natural language processing, computer vision, audio signal processing, frequent pattern mining, and machine learning. In this study, we present an overview of the existing studies on using big data techniques to study human behavior in the time of the COVID-19 pandemic. In particular, we categorize these studies into three groups-using big data to measure, model, and leverage human behavior, respectively. The related tasks, data, and methods are summarized accordingly. To provide more insights into how to fight the COVID-19 pandemic and future global catastrophes, we further discuss challenges and potential opportunities.

Metamodeling for Policy Simulations with Multivariate Outcomes

PURPOSE

Metamodels are simplified approximations of more complex models that can be used as surrogates for the original models. Challenges in using metamodels for policy analysis arise when there are multiple correlated outputs of interest. We develop a framework for metamodeling with policy simulations to accommodate multivariate outcomes.

METHODS

We combine 2 algorithm adaptation methods-multitarget stacking and regression chain with maximum correlation-with different base learners including linear regression (LR), elastic net (EE) with second-order terms, Gaussian process regression (GPR), random forests (RFs), and neural networks. We optimize integrated models using variable selection and hyperparameter tuning. We compare the accuracy, efficiency, and interpretability of different approaches. As an example application, we develop metamodels to emulate a microsimulation model of testing and treatment strategies for hepatitis C in correctional settings.

RESULTS

Output variables from the simulation model were correlated (average ρ = 0.58). Without multioutput algorithm adaptation methods, in-sample fit (measured by R2) ranged from 0.881 for LR to 0.987 for GPR. The multioutput algorithm adaptation method increased R2 by an average 0.002 across base learners. Variable selection and hyperparameter tuning increased R2 by 0.009. Simpler models such as LR, EE, and RF required minimal training and prediction time. LR and EE had advantages in model interpretability, and we considered methods for improving the interpretability of other models.

CONCLUSIONS

In our example application, the choice of base learner had the largest impact on R2; multioutput algorithm adaptation and variable selection and hyperparameter tuning had a modest impact. Although advantages and disadvantages of specific learning algorithms may vary across different modeling applications, our framework for metamodeling in policy analyses with multivariate outcomes has broad applicability to decision analysis in health and medicine.

Personalization of Medical Treatment Decisions: Simplifying Complex Models while Maintaining Patient Health Outcomes

BACKGROUND

Personalizing medical treatments based on patient-specific risks and preferences can improve patient health. However, models to support personalized treatment decisions are often complex and difficult to interpret, limiting their clinical application.

METHODS

We present a new method, using machine learning to create meta-models, for simplifying complex models for personalizing medical treatment decisions. We consider simple interpretable models, interpretable ensemble models, and noninterpretable ensemble models. We use variable selection with a penalty for patient-specific risks and/or preferences that are difficult, risky, or costly to obtain. We interpret the meta-models to the extent permitted by their model architectures. We illustrate our method by applying it to simplify a previously developed model for personalized selection of antipsychotic drugs for patients with schizophrenia.

RESULTS

The best simplified interpretable, interpretable ensemble, and noninterpretable ensemble models contained at most half the number of patient-specific risks and preferences compared with the original model. The simplified models achieved 60.5% (95% credible interval [crI]: 55.2-65.4), 60.8% (95% crI: 55.5-65.7), and 83.8% (95% crI: 80.8-86.6), respectively, of the net health benefit of the original model (quality-adjusted life-years gained). Important variables in all models were similar and made intuitive sense. Computation time for the meta-models was orders of magnitude less than for the original model.

LIMITATIONS

The simplified models share the limitations of the original model (e.g., potential biases).

CONCLUSIONS

Our meta-modeling method is disease- and model- agnostic and can be used to simplify complex models for personalization, allowing for variable selection in addition to improved model interpretability and computational performance. Simplified models may be more likely to be adopted in clinical settings and can help improve equity in patient outcomes.

Introduction to Metamodeling for Reducing Computational Burden of Advanced Analyses with Health Economic Models: A Structured Overview of Metamodeling Methods in a 6-Step Application Process

Metamodels can be used to reduce the computational burden associated with computationally demanding analyses of simulation models, although applications within health economics are still scarce. Besides a lack of awareness of their potential within health economics, the absence of guidance on the conceivably complex and time-consuming process of developing and validating metamodels may contribute to their limited uptake. To address these issues, this article introduces metamodeling to the wider health economic audience and presents a process for applying metamodeling in this context, including suitable methods and directions for their selection and use. General (i.e., non-health economic specific) metamodeling literature, clinical prediction modeling literature, and a previously published literature review were exploited to consolidate a process and to identify candidate metamodeling methods. Methods were considered applicable to health economics if they are able to account for mixed (i.e., continuous and discrete) input parameters and continuous outcomes. Six steps were identified as relevant for applying metamodeling methods within health economics: 1) the identification of a suitable metamodeling technique, 2) simulation of data sets according to a design of experiments, 3) fitting of the metamodel, 4) assessment of metamodel performance, 5) conducting the required analysis using the metamodel, and 6) verification of the results. Different methods are discussed to support each step, including their characteristics, directions for use, key references, and relevant R and Python packages. To address challenges regarding metamodeling methods selection, a first guide was developed toward using metamodels to reduce the computational burden of analyses of health economic models. This guidance may increase applications of metamodeling in health economics, enabling increased use of state-of-the-art analyses (e.g., value of information analysis) with computationally burdensome simulation models.

Comparing Strategies to Prevent Stroke and Ischemic Heart Disease in the Tunisian Population: Markov Modeling Approach Using a Comprehensive Sensitivity Analysis Algorithm

Background

Mathematical models offer the potential to analyze and compare the effectiveness of very different interventions to prevent future cardiovascular disease. We developed a comprehensive Markov model to assess the impact of three interventions to reduce ischemic heart diseases (IHD) and stroke deaths: (i) improved medical treatments in acute phase, (ii) secondary prevention by increasing the uptake of statins, (iii) primary prevention using health promotion to reduce dietary salt consumption.

Methods

We developed and validated a Markov model for the Tunisian population aged 35–94 years old over a 20-year time horizon. We compared the impact of specific treatments for stroke, lifestyle, and primary prevention on both IHD and stroke deaths. We then undertook extensive sensitivity analyses using both a probabilistic multivariate approach and simple linear regression (metamodeling).

Results

The model forecast a dramatic mortality rise, with 111,134 IHD and stroke deaths (95% CI 106567 to 115048) predicted in 2025 in Tunisia. The salt reduction offered the potentially most powerful preventive intervention that might reduce IHD and stroke deaths by 27% (−30240 [−30580 to −29900]) compared with 1% for medical strategies and 3% for secondary prevention. The metamodeling highlighted that the initial development of a minor stroke substantially increased the subsequent probability of a fatal stroke or IHD death.

Conclusions

The primary prevention of cardiovascular disease via a reduction in dietary salt consumption appeared much more effective than secondary or tertiary prevention approaches. Our simple but comprehensive model offers a potentially attractive methodological approach that might now be extended and replicated in other contexts and populations.

Linear regression metamodeling as a tool to summarize and present simulation model results

BACKGROUND / OBJECTIVE

Modelers lack a tool to systematically and clearly present complex model results, including those from sensitivity analyses. The objective was to propose linear regression metamodeling as a tool to increase transparency of decision analytic models and better communicate their results.

METHODS

We used a simplified cancer cure model to demonstrate our approach. The model computed the lifetime cost and benefit of 3 treatment options for cancer patients. We simulated 10,000 cohorts in a probabilistic sensitivity analysis (PSA) and regressed the model outcomes on the standardized input parameter values in a set of regression analyses. We used the regression coefficients to describe measures of sensitivity analyses, including threshold and parameter sensitivity analyses. We also compared the results of the PSA to deterministic full-factorial and one-factor-at-a-time designs.

RESULTS

The regression intercept represented the estimated base-case outcome, and the other coefficients described the relative parameter uncertainty in the model. We defined simple relationships that compute the average and incremental net benefit of each intervention. Metamodeling produced outputs similar to traditional deterministic 1-way or 2-way sensitivity analyses but was more reliable since it used all parameter values.

CONCLUSIONS

Linear regression metamodeling is a simple, yet powerful, tool that can assist modelers in communicating model characteristics and sensitivity analyses.

MedView: an instrument for clinical research and education in oral medicine

The etiology for many of the mucosal lesions we encounter in clinical practice is frequently uncertain or unknown and there is reason to believe that multicausality plays an important role. To detect multicausal relationships, the analysis must include multiple variables and large amounts of data. A traditional retrospective analysis is often based on a limited number of variables and frequently entails methodological errors where vital information may be missing. Prospective studies may be hampered by the fact that the prevalences of many conditions are relatively low. The search for new knowledge in oral medicine should therefore be facilitated by prospective use of formalized information gathered in multicenter studies. MedView is a computer program that is based on formalized input and registration of all clinical information. The output applications are focused on visualization and statistical analysis. MedView is aimed at clinical research and is well suited for multicenter studies. It also contains applications for education and distant consultations.

Vascular catheter site care: the clinical and economic benefits of chlorhexidine gluconate compared with povidone iodine

The use of chlorhexidine gluconate solution for vascular catheter insertion site care reduces the risk of catheter-related bloodstream infection by one-half, compared with povidone iodine. Our objective was to evaluate the cost-effectiveness of chlorhexidine gluconate versus povidone iodine. We used data from randomized, controlled trials, meta-analyses, and epidemiologic studies to construct a decision analysis model. We estimated that use of chlorhexidine, rather than povidone, for central catheter site care resulted in a 1.6% decrease in the incidence of catheter-related bloodstream infection, a 0.23% decrease in the incidence of death, and savings of 113 dollars per catheter used. For peripheral catheter site care, the results were similar, although the differences were smaller. The results were found to be robust on multivariate sensitivity analyses. Use of chlorhexidine gluconate in place of the current standard solution for vascular catheter site care is a simple and cost-effective method of improving patient safety in the hospital setting.

Correlations in uncertainty analysis for medical decision making: an application to heart-valve replacement

A Monte Carlo uncertainty analysis with correlations between parameters is applied to a Markov-chain model that is used to support the choice of a replacement heart-valve. The objective is to quantify the effects of uncertainty in and of correlations between probabilities of valve-related events on the life expectancies of four valve types. The uncertainty in the logit- and log-transformed parameters-mostly representing probabilities and durations-is modeled as a multivariate normal distribution. The univariate distributions are obtained through values for the median and the 0.975 quantile of each parameter. Correlations between parameters are difficult to quantify. A sensitivity analysis is suggested to study their influences on the uncertainty in valve preference prior to further elicitation efforts. The results of the uncertainty analysis strengthen the conclusions from a preceding study, which did not include uncertainty in the model parameters, where the homograft turned out to be the best choice. It is concluded that the influence of correlations is limited in most cases. Preference statements become more certain when the correlation between valve types increases.

Probabilistic sensitivity analysis incorporating the bootstrap: an example comparing treatments for the eradication of Helicobacter pylori

Decision-analytic models are frequently used to evaluate the relative costs and benefits of alternative therapeutic strategies for health care. Various types of sensitivity analysis are used to evaluate the uncertainty inherent in the models. Although probabilistic sensitivity analysis is more difficult theoretically and computationally, the results can be much more powerful and useful than deterministic sensitivity analysis. The authors show how a Monte Carlo simulation can be implemented using standard software to perform a probabilistic sensitivity analysis incorporating the bootstrap. The method is applied to a decision-analytic model evaluating the cost-effectiveness of Helicobacter pylori eradication. The necessary steps are straightforward and are described in detail. The use of the bootstrap avoids certain difficulties encountered with theoretical distributions. The probabilistic sensitivity analysis provided insights into the decision-analytic model beyond the traditional base-case and deterministic sensitivity analyses and should become the standard method for assessing sensitivity.

A Bayesian approach to sensitivity analysis

Sensitivity analysis has traditionally been applied to decision models to quantify the stability of a preferred alternative to parametric variation. In the health literature, sensitivity measures have traditionally been based upon distance metrics, payoff variations, and probability measures. We advocate a new approach based on information value and argue that such an approach is better suited to address the decision-maker's real concerns. We provide an example comparing conventional sensitivity analysis to one based on information value. This article is a US government work and is in the public domain in the United States.

Computational methods for probabilistic decision trees

Decision tree models may be more realistic if branching probabilities (and possibly utilities) are represented as distributions rather than point estimates. However, numerical analysis of such "probabilistic" trees is more difficult. This study employed the Mathematica computer algebra system to implement and verify previously described probabilistic methods. Both algebraic approximations and Monte Carlo simulation methods were used; in particular, simulations with beta, logistic-normal, and triangular distributions for branching probabilities were compared. Algebraic and simulation methods of sensitivity analysis were also implemented and compared. Computation required minimal programming and was reasonably fast using Mathematica on a standard personal computer. This study verified previously published results, including methods of sensitivity analysis. Changing the input distributional form had little effect. Computation is no longer a significant barrier to the use of probabilistic methods for analysis of decision trees.

Rapid approximation of confidence intervals for Markov process decision models: applications in decision support systems

OBJECTIVE

Develop the methodological foundation for interactive use of Markov process decision models by patients and physicians at the bedside.

DESIGN

Monte Carlo simulation studies of a decision model comparing two treatments for benign prostatic hypertrophy: watchful waiting (WW) and transurethral prostatectomy (TUR).

MEASUREMENTS

The 95% confidence interval (CI) for the mean of the Markov model; the correlation of a linear approximation with the full Markov model; the predictive performance of the approximation; the information index of specific utilities in the model.

RESULTS

The 95% CI for the gain in utility with initial TUR was -1.4 to 19.0 quality-adjusted life-months. A multivariate linear model had an excellent fit to the predictions of the Markov model (R2 = 0.966). In an independent data set, the linear model also had a high correlation with the full Markov model (R2 = 0.967); its predictions were unbiased (p = 0.597, paired t-test); and, in 96.4% of simulated cases, its treatment recommendation was the same.

CONCLUSION

Using the linear model, it was possible to efficiently compute which health state had the largest contribution to the variance of the decision model. This is the most informative utility value to elicit next. The most informative utility at any point in a sequence changed depending on utilities previously entered into the model. A linear model can be used to approximate the predictions of a Markov process decision model.