Stochastic Tree Search for Estimating Optimal Dynamic Treatment Regimes

A general framework for subgroup detection via one-step value difference estimation

Recent statistical methodology for precision medicine has focused on either identification of subgroups with enhanced treatment effects or estimating optimal treatment decision rules so that treatment is allocated in a way that maximizes, on average, predefined patient outcomes. Less attention has been given to subgroup testing, which involves evaluation of whether at least a subgroup of the population benefits from an investigative treatment, compared to some control or standard of care. In this work, we propose a general framework for testing for the existence of a subgroup with enhanced treatment effects based on the difference of the estimated value functions under an estimated optimal treatment regime and a fixed regime that assigns everyone to the same treatment. Our proposed test does not require specification of the parametric form of the subgroup and allows heterogeneous treatment effects within the subgroup. The test applies to cases when the outcome of interest is either a time-to-event or a (uncensored) scalar, and is valid at the exceptional law. To demonstrate the empirical performance of the proposed test, we study the type I error and power of the test statistics in simulations and also apply our test to data from a Phase III trial in patients with hematological malignancies.

Estimating tree-based dynamic treatment regimes using observational data with restricted treatment sequences

A dynamic treatment regime (DTR) is a sequence of decision rules that provide guidance on how to treat individuals based on their static and time-varying status. Existing observational data are often used to generate hypotheses about effective DTRs. A common challenge with observational data, however, is the need for analysts to consider "restrictions" on the treatment sequences. Such restrictions may be necessary for settings where (1) one or more treatment sequences that were offered to individuals when the data were collected are no longer considered viable in practice, (2) specific treatment sequences are no longer available, or (3) the scientific focus of the analysis concerns a specific type of treatment sequences (eg, "stepped-up" treatments). To address this challenge, we propose a restricted tree-based reinforcement learning (RT-RL) method that searches for an interpretable DTR with the maximum expected outcome, given a (set of) user-specified restriction(s), which specifies treatment options (at each stage) that ought not to be considered as part of the estimated tree-based DTR. In simulations, we evaluate the performance of RT-RL versus the standard approach of ignoring the partial data for individuals not following the (set of) restriction(s). The method is illustrated using an observational data set to estimate a two-stage stepped-up DTR for guiding the level of care placement for adolescents with substance use disorder.

Penalized Spline-Involved Tree-based (PenSIT) Learning for estimating an optimal dynamic treatment regime using observational data

Dynamic treatment regimes are a set of time-adaptive decision rules that can be used to personalize treatment across multiple stages of care. Grounded in causal inference methods, dynamic treatment regimes identify variables that differentiate the treatment effect and may be used to tailor treatments across individuals based on the patient's own characteristics - thereby representing an important step toward personalized medicine. In this manuscript we introduce Penalized Spline-Involved Tree-based Learning, which seeks to improve upon existing tree-based approaches to estimating an optimal dynamic treatment regime. Instead of using weights determined from the estimated propensity scores, which may result in unstable estimates when weights are highly variable, we predict missing counterfactual outcomes using regression models that incorporate a penalized spline of the propensity score and other covariates predictive of the outcome. We further develop a novel purity measure applied within a decision tree framework to produce a flexible yet interpretable method for estimating an optimal multi-stage multi-treatment dynamic treatment regime. In simulation experiments we demonstrate good performance of Penalized Spline-Involved Tree-based Learning relative to competing methods and, in particular, we show that Penalized Spline-Involved Tree-based Learning may be advantageous when the sample size is small and/or when the level of confounding of the outcome is high. We apply Penalized Spline-Involved Tree-based Learning to the retrospectively-collected Medical Information Mart for Intensive Care dataset to identify variables that may be used to tailor early fluid resuscitation strategies in septic patients.

Estimation in regret-regression using quadratic inference functions with ridge estimator

In this paper, we propose a new estimation method in estimating optimal dynamic treatment regimes. The quadratic inference functions in myopic regret-regression (QIF-MRr) can be used to estimate the parameters of the mean response at each visit, conditional on previous states and actions. Singularity issues may arise during computation when estimating the parameters in ODTR using QIF-MRr due to multicollinearity. Hence, the ridge penalty was introduced in rQIF-MRr to tackle the issues. A simulation study and an application to anticoagulation dataset were conducted to investigate the model’s performance in parameter estimation. The results show that estimations using rQIF-MRr are more efficient than the QIF-MRr.

Survival Augmented Patient Preference Incorporated Reinforcement Learning to Evaluate Tailoring Variables for Personalized Healthcare

In this paper, we consider personalized treatment decision strategies in the management of chronic diseases, such as chronic kidney disease, which typically consists of sequential and adaptive treatment decision making. We investigate a two-stage treatment setting with a survival outcome that could be right censored. This can be formulated through a dynamic treatment regime (DTR) framework, where the goal is to tailor treatment to each individual based on their own medical history in order to maximize a desirable health outcome. We develop a new method, Survival Augmented Patient Preference incorporated reinforcement Q-Learning (SAPP-Q-Learning) to decide between quality of life and survival restricted at maximal follow-up. Our method incorporates the latent patient preference into a weighted utility function that balances between quality of life and survival time, in a Q-learning model framework. We further propose a corresponding m-out-of-n Bootstrap procedure to accurately make statistical inferences and construct confidence intervals on the effects of tailoring variables, whose values can guide personalized treatment strategies.