Data-Driven Markov Decision Process Approximations for Personalized Hypertension Treatment Planning

Examining chronic kidney disease screening frequency among diabetics: a POMDP approach

Forty percent of diabetics will develop chronic kidney disease (CKD) in their lifetimes. However, as many as 50% of these CKD cases may go undiagnosed. We developed screening recommendations stratified by age and previous test history for individuals with diagnosed diabetes and unknown proteinuria status by race and gender groups. To do this, we used a Partially Observed Markov Decision Process (POMDP) to identify whether a patient should be screened at every three-month interval from ages 30-85. Model inputs were drawn from nationally-representative datasets, the medical literature, and a microsimulation that integrates this information into group-specific disease progression rates. We implement the POMDP solution policy in the microsimulation to understand how this policy may impact health outcomes and generate an easily-implementable, non-belief-based approximate policy for easier clinical interpretability. We found that the status quo policy, which is to screen annually for all ages and races, is suboptimal for maximizing expected discounted future net monetary benefits (NMB). The POMDP policy suggests more frequent screening after age 40 in all race and gender groups, with screenings 2-4 times a year for ages 61-70. Black individuals are recommended for screening more frequently than their White counterparts. This policy would increase NMB from the status quo policy between $1,000 to  $8,000 per diabetic patient at a willingness-to-pay of $150,000 per quality-adjusted life year (QALY).

Monitoring policy in the context of preventive treatment of cardiovascular disease

Preventing chronic diseases is an essential aspect of medical care. To prevent chronic diseases, physicians focus on monitoring their risk factors and prescribing the necessary medication. The optimal monitoring policy depends on the patient's risk factors and demographics. Monitoring too frequently may be unnecessary and costly; on the other hand, monitoring the patient infrequently means the patient may forgo needed treatment and experience adverse events related to the disease. We propose a finite horizon and finite-state Markov decision process to define monitoring policies. To build our Markov decision process, we estimate stochastic models based on longitudinal observational data from electronic health records for a large cohort of patients seen in the national U.S. Veterans Affairs health system. We use our model to study policies for whether or when to assess the need for cholesterol-lowering medications. We further use our model to investigate the role of gender and race on optimal monitoring policies.

A Promising Approach to Optimizing Sequential Treatment Decisions for Depression: Markov Decision Process

The most appropriate next step in depression treatment after the initial treatment fails is unclear. This study explores the suitability of the Markov decision process for optimizing sequential treatment decisions for depression. We conducted a formal comparison of a Markov decision process approach and mainstream state-transition models as used in health economic decision analysis to clarify differences in the model structure. We performed two reviews: the first to identify existing applications of the Markov decision process in the field of healthcare and the second to identify existing health economic models for depression. We then illustrated the application of a Markov decision process by reformulating an existing health economic model. This provided input for discussing the suitability of a Markov decision process for solving sequential treatment decisions in depression. The Markov decision process and state-transition models differed in terms of flexibility in modeling actions and rewards. In all, 23 applications of a Markov decision process within the context of somatic disease were included, 16 of which concerned sequential treatment decisions. Most existing health economic models relating to depression have a state-transition structure. The example application replicated the health economic model and enabled additional capacity to make dynamic comparisons of more interventions over time than was possible with traditional state-transition models. Markov decision processes have been successfully applied to address sequential treatment-decision problems, although the results have been published mostly in economics journals that are not related to healthcare. One advantage of a Markov decision process compared with state-transition models is that it allows extended action space: the possibility of making dynamic comparisons of different treatments over time. Within the context of depression, although existing state-transition models are too basic to evaluate sequential treatment decisions, the assumptions of a Markov decision process could be satisfied. The Markov decision process could therefore serve as a powerful model for optimizing sequential treatment in depression. This would require a sufficiently elaborate state-transition model at the cohort or patient level.

Optimal cholesterol treatment plans and genetic testing strategies for cardiovascular diseases

Atherosclerotic cardiovascular disease (ASCVD) is among the leading causes of death in the US. Although research has shown that ASCVD has genetic elements, the understanding of how genetic testing influences its prevention and treatment has been limited. To this end, we model the health trajectory of patients stochastically and determine treatment and testing decisions simultaneously. Since the cholesterol level of patients is one controllable risk factor for ASCVD events, we model cholesterol treatment plans as Markov decision processes. We determine whether and when patients should receive a genetic test using value of information analysis. By simulating the health trajectory of over 64 million adult patients, we find that 6.73 million patients undergo genetic testing. The optimal treatment plans informed with clinical and genetic information save 5,487 more quality-adjusted life-years while costing $1.18 billion less than the optimal treatment plans informed with clinical information only. As precision medicine becomes increasingly important, understanding the impact of genetic information becomes essential.

Personalizing Second-Line Type 2 Diabetes Treatment Selection: Combining Network Meta-analysis, Individualized Risk, and Patient Preferences for Unified Decision Support

BACKGROUND

Personalizing medical treatment often requires practitioners to compare multiple treatment options, assess a patient's unique risk and benefit from each option, and elicit a patient's preferences around treatment. We integrated these 3 considerations into a decision-modeling framework for the selection of second-line glycemic therapy for type 2 diabetes.

METHODS

Based on multicriteria decision analysis, we developed a unified treatment decision support tool accounting for 3 factors: patient preferences, disease outcomes, and medication efficacy and safety profiles. By standardizing and multiplying these 3 factors, we calculated the ranking score for each medication. This approach was applied to determining second-line glycemic therapy by integrating 1) treatment efficacy and side-effect data from a network meta-analysis of 301 randomized trials ( N = 219,277), 2) validated risk equations for type 2 diabetes complications, and 3) patient preferences around treatment (e.g., to avoid daily glucose testing). Data from participants with type 2 diabetes in the U.S. National Health and Nutrition Examination Survey (NHANES 2003-2014, N = 1107) were used to explore variations in treatment recommendations and associated quality-adjusted life-years given different patient features.

RESULTS

Patients at the highest microvascular disease risk had glucagon-like peptide 1 agonists or basal insulin recommended as top choices, whereas those wanting to avoid an injected medication or daily glucose testing had sodium-glucose linked transporter 2 or dipeptidyl peptidase 4 inhibitors commonly recommended, and those with major cost concerns had sulfonylureas commonly recommended. By converting from the most common sulfonylurea treatment to the model-recommended treatment, NHANES participants were expected to save an average of 0.036 quality-adjusted life-years per person (about a half month) from 10 years of treatment.

CONCLUSIONS

Models can help integrate meta-analytic treatment effect estimates with individualized risk calculations and preferences, to aid personalized treatment selection.

Dynamic treatment selection and modification for personalised blood pressure therapy using a Markov decision process model: a cost-effectiveness analysis

OBJECTIVE

Personalised medicine seeks to select and modify treatments based on individual patient characteristics and preferences. We sought to develop an automated strategy to select and modify blood pressure treatments, incorporating the likelihood that patients with different characteristics would benefit from different types of medications and dosages and the potential severity and impact of different side effects among patients with different characteristics.

DESIGN, SETTING AND PARTICIPANTS

We developed a Markov decision process (MDP) model to incorporate meta-analytic data and estimate the optimal treatment for maximising discounted lifetime quality-adjusted life-years (QALYs) based on individual patient characteristics, incorporating medication adjustment choices when a patient incurs side effects. We compared the MDP to current US blood pressure treatment guidelines (the Eighth Joint National Committee, JNC8) and a variant of current guidelines that incorporates results of a major recent trial of intensive treatment (Intensive JNC8). We used a microsimulation model of patient demographics, cardiovascular disease risk factors and side effect probabilities, sampling from the National Health and Nutrition Examination Survey (2003-2014), to compare the expected population outcomes from adopting the MDP versus guideline-based strategies.

MAIN OUTCOME MEASURES

Costs and QALYs for the MDP-based treatment (MDPT), JNC8 and Intensive JNC8 strategies.

RESULTS

Compared with the JNC8 guideline, the MDPT strategy would be cost-saving from a societal perspective with discounted savings of US$1187 per capita (95% CI 1178 to 1209) and an estimated discounted gain of 0.06 QALYs per capita (95% CI 0.04 to 0.08) among the US adult population. QALY gains would largely accrue from reductions in severe side effects associated with higher treatment doses later in life. The Intensive JNC8 strategy was dominated by the MDPT strategy.

CONCLUSIONS

An MDP-based approach can aid decision-making by incorporating meta-analytic evidence to personalise blood pressure treatment and improve overall population health compared with current blood pressure treatment guidelines.