Robust Q-learning

Reinforcement learning using neural networks in estimating an optimal dynamic treatment regime in patients with sepsis

OBJECTIVE

Early fluid resuscitation is crucial in the treatment of sepsis, yet the optimal dosage remains debated. This study aims to determine the optimal multi-stage fluid resuscitation dosage for sepsis patients.

METHODS

We propose a reinforcement learning algorithm with neural networks (RL-NN), utilizing the flexibility of deep learning architectures to mitigate model misspecification. We use cross-validation and random search for hyperparameter tuning to further enhance model robustness and generalization.

RESULTS

Simulation results demonstrate that our method outperforms existing methods in terms of both the percentage of correctly classified optimal treatments and the predicted counterfactual mean outcome. Applying this method to the sepsis cohort from the Medical Information Mart for Intensive Care III (MIMIC-III), we recommend that all sepsis patients receive adequate fluid resuscitation (≥ 30 mL/kg) within the first 3 h of admission to the MICU. Our approach is expected to significantly reduce the mean SOFA score by 23.71%, enhancing patient outcomes.

CONCLUSION

Our RL-NN method offers an accurate, real-time approach to optimizing sepsis treatment and aligns with the 'Surviving Sepsis Campaign' guidelines. It also has the potential to be integrated with existing electronic health record (EHR) systems, guiding clinical decision-making and thereby improving patient prognosis.

Machine Learning Approaches for Neuroblastoma Risk Prediction and Stratification

Machine learning (ML) holds great promise in advancing risk prediction and stratification for neuroblastoma, a highly heterogeneous pediatric cancer. By utilizing large-scale biological and clinical data, ML models can detect complex patterns that traditional approaches often overlook, enabling more personalized treatments and better patient outcomes. Various ML techniques, such as support vector machines, random forests, and deep learning, have shown superior performance in predicting survival, relapse, and treatment responses in neuroblastoma patients compared to conventional methods. However, challenges like limited data size, model interpretability, data variability, and difficulties in clinical integration hinder broader adoption. Additionally, ethical concerns related to bias and privacy must be addressed. Future work should focus on improving data quality, enhancing model transparency, and conducting thorough clinical validation. With these advancements, ML has the potential to revolutionize neuroblastoma care by refining early diagnosis, risk assessment, and therapeutic decision-making.

Dynamic Treatment Regimes on Dyadic Networks

Identifying interventions that are optimally tailored to each individual is of significant interest in various fields, in particular precision medicine. Dynamic treatment regimes (DTRs) employ sequences of decision rules that utilize individual patient information to recommend treatments. However, the assumption that an individual's treatment does not impact the outcomes of others, known as the no interference assumption, is often challenged in practical settings. For example, in infectious disease studies, the vaccine status of individuals in close proximity can influence the likelihood of infection. Imposing this assumption when it, in fact, does not hold, may lead to biased results and impact the validity of the resulting DTR optimization. We extend the estimation method of dynamic weighted ordinary least squares (dWOLS), a doubly robust and easily implemented approach for estimating optimal DTRs, to incorporate the presence of interference within dyads (i.e., pairs of individuals). We formalize an appropriate outcome model and describe the estimation of an optimal decision rule in the dyadic-network context. Through comprehensive simulations and analysis of the Population Assessment of Tobacco and Health (PATH) data, we demonstrate the improved performance of the proposed joint optimization strategy compared to the current state-of-the-art conditional optimization methods in estimating the optimal treatment assignments when within-dyad interference exists.

Bayesian Empirical Likelihood Regression for Semiparametric Estimation of Optimal Dynamic Treatment Regimes

We propose a semiparametric approach to Bayesian modeling of dynamic treatment regimes that is built on a Bayesian likelihood-based regression estimation framework. Methods based on this framework exhibit a probabilistic coherence property that leads to accurate estimation of the optimal dynamic treatment regime. Unlike most Bayesian estimation methods, our proposed method avoids strong distributional assumptions for the intermediate and final outcomes by utilizing empirical likelihoods. Our proposed method allows for either linear, or more flexible forms of mean functions for the stagewise outcomes. A variational Bayes approximation is used for computation to avoid common pitfalls associated with Markov Chain Monte Carlo approaches coupled with empirical likelihood. Through simulations and analysis of the STAR*D sequential randomized trial data, our proposed method demonstrates superior accuracy over Q-learning and parametric Bayesian likelihood-based regression estimation, particularly when the parametric assumptions of regression error distributions may be potentially violated.

On "Reflections on the concept of optimality of single decision point treatment regimes"

This is a discussion of "Reflections on the concept of optimality of single decision point treatment regimes" by Trung Dung Tran, Ariel Alonso Abad, Geert Verbeke, Geert Molenberghs, and Iven Van Mechelen. The authors propose a thoughtful consideration of optimization targets and the implications of such targets for the resulting optimal treatment rule. However, we contest the assertation that targets of optimization have been overlooked and suggest additional considerations that researchers must contemplate as part of a complete framework for learning about optimal treatment regimes.

Modified interactive Q-learning for attenuating the impact of model misspecification with treatment effect heterogeneity

A sequential multiple assignment randomized trial, which incorporates multiple stages of randomization, is a popular approach for collecting data to inform personalized and adaptive treatments. There is an extensive literature on statistical methods to analyze data collected in sequential multiple assignment randomized trials and estimate the optimal dynamic treatment regime. Q-learning with linear regression is widely used for this purpose due to its ease of implementation. However, model misspecification is a common problem with this approach, and little attention has been given to the impact of model misspecification when treatment effects are heterogeneous across subjects. This article describes the integrative impact of two possible types of model misspecification related to treatment effect heterogeneity: omitted early-stage treatment effects in late-stage main effect model, and violated linearity assumption between pseudo-outcomes and predictors despite non-linearity arising from the optimization operation. The proposed method, aiming to deal with both types of misspecification concomitantly, builds interactive models into modified parametric Q-learning with Murphy's regret function. Simulations show that the proposed method is robust to both sources of model misspecification. The proposed method is applied to a two-stage sequential multiple assignment randomized trial with embedded tailoring aimed at reducing binge drinking in first-year college students.

Relative sparsity for medical decision problems

Existing statistical methods can estimate a policy, or a mapping from covariates to decisions, which can then instruct decision makers (eg, whether to administer hypotension treatment based on covariates blood pressure and heart rate). There is great interest in using such data-driven policies in healthcare. However, it is often important to explain to the healthcare provider, and to the patient, how a new policy differs from the current standard of care. This end is facilitated if one can pinpoint the aspects of the policy (ie, the parameters for blood pressure and heart rate) that change when moving from the standard of care to the new, suggested policy. To this end, we adapt ideas from Trust Region Policy Optimization (TRPO). In our work, however, unlike in TRPO, the difference between the suggested policy and standard of care is required to be sparse, aiding with interpretability. This yields "relative sparsity," where, as a function of a tuning parameter,λ $$ \lambda $$ , we can approximately control the number of parameters in our suggested policy that differ from their counterparts in the standard of care (eg, heart rate only). We propose a criterion for selectingλ $$ \lambda $$ , perform simulations, and illustrate our method with a real, observational healthcare dataset, deriving a policy that is easy to explain in the context of the current standard of care. Our work promotes the adoption of data-driven decision aids, which have great potential to improve health outcomes.

Penalized robust learning for optimal treatment regimes with heterogeneous individualized treatment effects

The growing popularity of personalized medicine motivates people to explore individualized treatment regimes according to heterogeneous characteristics of the patients. For the large-scale data analysis, however, the data are collected at different times and different locations, i.e. subjects are usually from a heterogeneous population, which causes that the optimal treatment regimes also vary for patients across different subgroups. In this paper, we mainly focus on the estimation of optimal treatment regimes for subjects come from a heterogeneous population with high-dimensional data. We first remove the main effects of the covariates for each subgroup to eliminate non-ignorable residual confounding. Based on the centralized outcome, we propose a penalized robust learning that estimates the coefficient matrix of the interactions between covariates and treatment by penalizing pairwise differences of the coefficients of any two subgroups for the same covariate, which can automatically identify the latent complex structure of the coefficient matrix with heterogeneous and homogeneous columns. At the same time, the penalized robust learning can also select the important variables that truly contribute to the individualized treatment decisions with commonly used sparsity structure penalty. Extensive simulation studies show that our proposed method outperforms current popular methods, and it is further illustrated in the real analysis of the Tamoxifen breast cancer data.

Double robust estimation of optimal partially adaptive treatment strategies: An application to breast cancer treatment using hormonal therapy

Precision medicine aims to tailor treatment decisions according to patients' characteristics. G-estimation and dynamic weighted ordinary least squares are double robust methods to identify optimal adaptive treatment strategies. It is underappreciated that they require modeling all existing treatment-confounder interactions to be consistent. Identifying optimal partially adaptive treatment strategies that tailor treatments according to only a few covariates, ignoring some interactions, may be preferable in practice. Building on G-estimation and dWOLS, we propose estimators of such partially adaptive strategies and demonstrate their double robustness. We investigate these estimators in a simulation study. Using data maintained by the Centre des Maladies du Sein, we estimate a partially adaptive treatment strategy for tailoring hormonal therapy use in breast cancer patients. R software implementing our estimators is provided.