1
|
Cho H, She J, De Marchi D, El-Zaatari H, Barnes EL, Kahkoska AR, Kosorok MR, Virkud AV. Machine Learning and Health Science Research: Tutorial. J Med Internet Res 2024; 26:e50890. [PMID: 38289657 PMCID: PMC10865203 DOI: 10.2196/50890] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2023] [Revised: 11/30/2023] [Accepted: 12/21/2023] [Indexed: 02/01/2024] Open
Abstract
Machine learning (ML) has seen impressive growth in health science research due to its capacity for handling complex data to perform a range of tasks, including unsupervised learning, supervised learning, and reinforcement learning. To aid health science researchers in understanding the strengths and limitations of ML and to facilitate its integration into their studies, we present here a guideline for integrating ML into an analysis through a structured framework, covering steps from framing a research question to study design and analysis techniques for specialized data types.
Collapse
Affiliation(s)
- Hunyong Cho
- Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Jane She
- Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Daniel De Marchi
- Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Helal El-Zaatari
- Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Edward L Barnes
- Division of Gastroenterology and Hepatology, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
- Center for Gastrointestinal Biology and Diseases, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Anna R Kahkoska
- Department of Nutrition, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
- Division of Endocrinology and Metabolism, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
- Center for Aging and Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Michael R Kosorok
- Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Arti V Virkud
- Kidney Center, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| |
Collapse
|
2
|
Shi J, Wu Z, Dempsey W. ASSESSING TIME-VARYING CAUSAL EFFECT MODERATION IN THE PRESENCE OF CLUSTER-LEVEL TREATMENT EFFECT HETEROGENEITY AND INTERFERENCE. Biometrika 2023; 110:645-662. [PMID: 37711671 PMCID: PMC10501736 DOI: 10.1093/biomet/asac065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/16/2023] Open
Abstract
The micro-randomized trial (MRT) is a sequential randomized experimental design to empirically evaluate the effectiveness of mobile health (mHealth) intervention components that may be delivered at hundreds or thousands of decision points. MRTs have motivated a new class of causal estimands, termed "causal excursion effects", for which semiparametric inference can be conducted via a weighted, centered least squares criterion (Boruvka et al., 2018). Existing methods assume between-subject independence and non-interference. Deviations from these assumptions often occur. In this paper, causal excursion effects are revisited under potential cluster-level treatment effect heterogeneity and interference, where the treatment effect of interest may depend on cluster-level moderators. Utility of the proposed methods is shown by analyzing data from a multi-institution cohort of first year medical residents in the United States.
Collapse
Affiliation(s)
- Jieru Shi
- Department of Biostatistics, University of Michigan, 1415 Washington Heights, Ann Arbor, MI 48109, USA
| | - Zhenke Wu
- Department of Biostatistics, University of Michigan, 1415 Washington Heights, Ann Arbor, MI 48109, USA
| | - Walter Dempsey
- Department of Biostatistics, University of Michigan, 1415 Washington Heights, Ann Arbor, MI 48109, USA
| |
Collapse
|
3
|
Weisenthal SJ, Thurston SW, Ertefaie A. Relative sparsity for medical decision problems. Stat Med 2023; 42:3067-3092. [PMID: 37315949 PMCID: PMC10524900 DOI: 10.1002/sim.9755] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 03/24/2023] [Accepted: 04/02/2023] [Indexed: 06/16/2023]
Abstract
Existing statistical methods can estimate a policy, or a mapping from covariates to decisions, which can then instruct decision makers (eg, whether to administer hypotension treatment based on covariates blood pressure and heart rate). There is great interest in using such data-driven policies in healthcare. However, it is often important to explain to the healthcare provider, and to the patient, how a new policy differs from the current standard of care. This end is facilitated if one can pinpoint the aspects of the policy (ie, the parameters for blood pressure and heart rate) that change when moving from the standard of care to the new, suggested policy. To this end, we adapt ideas from Trust Region Policy Optimization (TRPO). In our work, however, unlike in TRPO, the difference between the suggested policy and standard of care is required to be sparse, aiding with interpretability. This yields "relative sparsity," where, as a function of a tuning parameter,λ $$ \lambda $$ , we can approximately control the number of parameters in our suggested policy that differ from their counterparts in the standard of care (eg, heart rate only). We propose a criterion for selectingλ $$ \lambda $$ , perform simulations, and illustrate our method with a real, observational healthcare dataset, deriving a policy that is easy to explain in the context of the current standard of care. Our work promotes the adoption of data-driven decision aids, which have great potential to improve health outcomes.
Collapse
Affiliation(s)
- Samuel J. Weisenthal
- Department of Biostatistics and Computational Biology, University of Rochester Medical Center, New York, USA
- Medical Scientist Training Program, University of Rochester School of Medicine and Dentistry, New York, USA
| | - Sally W. Thurston
- Department of Biostatistics and Computational Biology, University of Rochester Medical Center, New York, USA
| | - Ashkan Ertefaie
- Department of Biostatistics and Computational Biology, University of Rochester Medical Center, New York, USA
| |
Collapse
|
4
|
Rose EJ, Moodie EEM, Shortreed SM. Monte Carlo sensitivity analysis for unmeasured confounding in dynamic treatment regimes. Biom J 2023; 65:e2100359. [PMID: 37017498 DOI: 10.1002/bimj.202100359] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Revised: 09/16/2022] [Accepted: 09/20/2022] [Indexed: 04/06/2023]
Abstract
Data-driven methods for personalizing treatment assignment have garnered much attention from clinicians and researchers. Dynamic treatment regimes formalize this through a sequence of decision rules that map individual patient characteristics to a recommended treatment. Observational studies are commonly used for estimating dynamic treatment regimes due to the potentially prohibitive costs of conducting sequential multiple assignment randomized trials. However, estimating a dynamic treatment regime from observational data can lead to bias in the estimated regime due to unmeasured confounding. Sensitivity analyses are useful for assessing how robust the conclusions of the study are to a potential unmeasured confounder. A Monte Carlo sensitivity analysis is a probabilistic approach that involves positing and sampling from distributions for the parameters governing the bias. We propose a method for performing a Monte Carlo sensitivity analysis of the bias due to unmeasured confounding in the estimation of dynamic treatment regimes. We demonstrate the performance of the proposed procedure with a simulation study and apply it to an observational study examining tailoring the use of antidepressant medication for reducing symptoms of depression using data from Kaiser Permanente Washington.
Collapse
Affiliation(s)
- Eric J Rose
- Department of Epidemiology and Biostatistics, McGill University, Montreal, QC, Canada
- Department of Epidemiology and Biostatistics, University at Albany, Rensselaer, New York, USA
| | - Erica E M Moodie
- Department of Epidemiology and Biostatistics, McGill University, Montreal, QC, Canada
| | - Susan M Shortreed
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
- Department of Biostatistics, University of Washington, Seattle, Washington, USA
| |
Collapse
|
5
|
Ghosh P, Yan X, Chakraborty B. A novel approach to assess dynamic treatment regimes embedded in a SMART with an ordinal outcome. Stat Med 2023; 42:1096-1111. [PMID: 36726310 DOI: 10.1002/sim.9659] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2022] [Revised: 10/21/2022] [Accepted: 01/04/2023] [Indexed: 02/03/2023]
Abstract
Sequential multiple assignment randomized trials (SMARTs) are used to construct data-driven optimal intervention strategies for subjects based on their intervention and covariate histories in different branches of health and behavioral sciences where a sequence of interventions is given to a participant. Sequential intervention strategies are often called dynamic treatment regimes (DTR). In the existing literature, the majority of the analysis methodologies for SMART data assume a continuous primary outcome. However, ordinal outcomes are also quite common in clinical practice. In this work, first, we introduce the notion of generalized odds ratio ( G O R $$ GOR $$ ) to compare two DTRs embedded in a SMART with an ordinal outcome and discuss some combinatorial properties of this measure. Next, we propose a likelihood-based approach to estimate G O R $$ GOR $$ from SMART data, and derive the asymptotic properties of its estimate. We discuss alternative ways to estimate G O R $$ GOR $$ using concordant-discordant pairs and two-sample U $$ U $$ -statistic. We derive the required sample size formula for designing SMARTs with ordinal outcomes based on G O R $$ GOR $$ . A simulation study shows the performance of the estimated G O R $$ GOR $$ in terms of the estimated power corresponding to the derived sample size. The methodology is applied to analyze data from the SMART+ study, conducted in the UK, to improve carbohydrate periodization behavior in athletes using a menu planner mobile application, Hexis Performance. A freely available Shiny web app using R is provided to make the proposed methodology accessible to other researchers and practitioners.
Collapse
Affiliation(s)
- Palash Ghosh
- Department of Mathematics, Indian Institute of Technology Guwahati, Assam, India.,Jyoti and Bhupat Mehta School of Health Sciences and Technology, Indian Institute of Technology Guwahati, Assam, India.,Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore
| | - Xiaoxi Yan
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore
| | - Bibhas Chakraborty
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore.,Department of Statistics and Data Science, National University of Singapore, Singapore, Singapore.,Department of Biostatistics and Bioinformatics, Duke University, Durham, North Carolina
| |
Collapse
|
6
|
Li Z, Chen J, Laber E, Liu F, Baumgartner R. Optimal Treatment Regimes: A Review and Empirical Comparison. Int Stat Rev 2023. [DOI: 10.1111/insr.12536] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/24/2023]
Affiliation(s)
- Zhen Li
- Department of Statistics North Carolina State University Raleigh 27607 NC USA
| | - Jie Chen
- Department of Biometrics Overland Pharmaceuticals Dover 19901 DE USA
| | - Eric Laber
- Department of Statistical Science, Department of Biostatistics and Bioinformatics Duke University Durham 27708 NC USA
| | - Fang Liu
- Biostatistics and Research Decision Sciences Merck & Co., Inc. Kenilworth NJ 07033 USA
| | - Richard Baumgartner
- Biostatistics and Research Decision Sciences Merck & Co., Inc. Kenilworth NJ 07033 USA
| |
Collapse
|
7
|
Abstract
We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.
Collapse
Affiliation(s)
- Peng Liao
- Department of Statistics, Harvard University
| | - Zhengling Qi
- Department of Decision Sciences, George Washington University
| | | | | | | |
Collapse
|
8
|
Ding Y, Li Y, Song R. Statistical Learning for Individualized Asset Allocation. J Am Stat Assoc 2022. [DOI: 10.1080/01621459.2022.2139265] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/31/2022]
Affiliation(s)
- Yi Ding
- Faculty of Business Administration, University of Macau, Macau (e-mail: )
| | | | | |
Collapse
|
9
|
Zhou W, Zhu R, Qu A. Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning. J Am Stat Assoc 2022. [DOI: 10.1080/01621459.2022.2138760] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Affiliation(s)
- Wenzhuo Zhou
- Department of Statistics, University of California Irvine, Irvine, CA;
| | - Ruoqing Zhu
- Department of Statistics, University of Illinois Urbana-Champaign, Champaign, IL;
| | - Annie Qu
- Department of Statistics, University of California Irvine, Irvine, CA,
| |
Collapse
|
10
|
Mitra N, Roy J, Small D. The Future of Causal Inference. Am J Epidemiol 2022; 191:1671-1676. [PMID: 35762132 PMCID: PMC9991894 DOI: 10.1093/aje/kwac108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2022] [Revised: 06/16/2022] [Accepted: 06/17/2022] [Indexed: 01/29/2023] Open
Abstract
The past several decades have seen exponential growth in causal inference approaches and their applications. In this commentary, we provide our top-10 list of emerging and exciting areas of research in causal inference. These include methods for high-dimensional data and precision medicine, causal machine learning, causal discovery, and others. These methods are not meant to be an exhaustive list; instead, we hope that this list will serve as a springboard for stimulating the development of new research.
Collapse
Affiliation(s)
- Nandita Mitra
- Correspondence to Dr. Nandita Mitra, Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, 423 Guardian Drive, Philadelphia, PA (e-mail: )
| | | | | |
Collapse
|
11
|
Shi C, Zhu J, Ye S, Luo S, Zhu H, Song R. Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process. J Am Stat Assoc 2022. [DOI: 10.1080/01621459.2022.2110878] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2022]
Affiliation(s)
| | | | - Shen Ye
- North Carolina State University
| | | | - Hongtu Zhu
- University of North Carolina at Chapel Hill
| | | |
Collapse
|
12
|
Shi C, Zhu J, Shen Y, Luo S, Zhu H, Song R. Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process. J Am Stat Assoc 2022. [DOI: 10.1080/01621459.2022.2110876] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2022]
Affiliation(s)
| | | | - Ye Shen
- North Carolina State University
| | | | - Hongtu Zhu
- University of North Carolina at Chapel Hill
| | | |
Collapse
|
13
|
Shi C, Luo S, Le Y, Zhu H, Song R. Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons. J Am Stat Assoc 2022. [DOI: 10.1080/01621459.2022.2106868] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/16/2022]
Affiliation(s)
| | | | - Yuan Le
- Shanghai University of Finance and Economics
| | - Hongtu Zhu
- University of North Carolina at Chapel Hill
| | | |
Collapse
|
14
|
Oh EJ, Qian M, Cheung YK. Generalization error bounds of dynamic treatment regimes in penalized regression-based learning. Ann Stat 2022. [DOI: 10.1214/22-aos2171] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Affiliation(s)
- Eun Jeong Oh
- Department of Biostatistics, Columbia University
| | - Min Qian
- Department of Biostatistics, Columbia University
| | | |
Collapse
|
15
|
Ramprasad P, Li Y, Yang Z, Wang Z, Sun WW, Cheng G. Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning. J Am Stat Assoc 2022. [DOI: 10.1080/01621459.2022.2096620] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
Affiliation(s)
| | | | - Zhuoran Yang
- Department of Statistics and Data Science, Yale University
| | - Zhaoran Wang
- Department of Industrial Engineering and Management Sciences, Northwestern University
| | - Will Wei Sun
- Krannert School of Management, Purdue University
| | | |
Collapse
|
16
|
Liu M, Shen X, Pan W. Deep reinforcement learning for personalized treatment recommendation. Stat Med 2022; 41:4034-4056. [PMID: 35716038 PMCID: PMC9427729 DOI: 10.1002/sim.9491] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2021] [Revised: 05/22/2022] [Accepted: 05/25/2022] [Indexed: 12/12/2022]
Abstract
In precision medicine, the ultimate goal is to recommend the most effective treatment to an individual patient based on patient-specific molecular and clinical profiles, possibly high-dimensional. To advance cancer treatment, large-scale screenings of cancer cell lines against chemical compounds have been performed to help better understand the relationship between genomic features and drug response; existing machine learning approaches use exclusively supervised learning, including penalized regression and recommender systems. However, it would be more efficient to apply reinforcement learning to sequentially learn as data accrue, including selecting the most promising therapy for a patient given individual molecular and clinical features and then collecting and learning from the corresponding data. In this article, we propose a novel personalized ranking system called Proximal Policy Optimization Ranking (PPORank), which ranks the drugs based on their predicted effects per cell line (or patient) in the framework of deep reinforcement learning (DRL). Modeled as a Markov decision process, the proposed method learns to recommend the most suitable drugs sequentially and continuously over time. As a proof-of-concept, we conduct experiments on two large-scale cancer cell line data sets in addition to simulated data. The results demonstrate that the proposed DRL-based PPORank outperforms the state-of-the-art competitors based on supervised learning. Taken together, we conclude that novel methods in the framework of DRL have great potential for precision medicine and should be further studied.
Collapse
Affiliation(s)
- Mingyang Liu
- School of Statistics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Xiaotong Shen
- School of Statistics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Wei Pan
- Division of Biostatistics, University of Minnesota, Minneapolis, Minnesota, USA
| |
Collapse
|
17
|
|
18
|
Affiliation(s)
| | - Xiaoyu Wang
- Key Laboratory of Systems and Control, Academy of Mathematics and Systems Science, Chinese Academy of Sciences
| | | | - Hongtu Zhu
- The Univeristy of North Carolina at Chapell Hill
| | | | | |
Collapse
|
19
|
Gao D, Liu Y, Zeng D. Non-asymptotic Properties of Individualized Treatment Rules from Sequentially Rule-Adaptive Trials. J Mach Learn Res 2022; 23:https://www.jmlr.org/papers/v23/21-0354.html. [PMID: 37576335 PMCID: PMC10419117] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Figures] [Subscribe] [Scholar Register] [Indexed: 08/15/2023]
Abstract
Learning optimal individualized treatment rules (ITRs) has become increasingly important in the modern era of precision medicine. Many statistical and machine learning methods for learning optimal ITRs have been developed in the literature. However, most existing methods are based on data collected from traditional randomized controlled trials and thus cannot take advantage of the accumulative evidence when patients enter the trials sequentially. It is also ethically important that future patients should have a high probability to be treated optimally based on the updated knowledge so far. In this work, we propose a new design called sequentially rule-adaptive trials to learn optimal ITRs based on the contextual bandit framework, in contrast to the response-adaptive design in traditional adaptive trials. In our design, each entering patient will be allocated with a high probability to the current best treatment for this patient, which is estimated using the past data based on some machine learning algorithm (for example, outcome weighted learning in our implementation). We explore the tradeoff between training and test values of the estimated ITR in single-stage problems by proving theoretically that for a higher probability of following the estimated ITR, the training value converges to the optimal value at a faster rate, while the test value converges at a slower rate. This problem is different from traditional decision problems in the sense that the training data are generated sequentially and are dependent. We also develop a tool that combines martingale with empirical process to tackle the problem that cannot be solved by previous techniques for i.i.d. data. We show by numerical examples that without much loss of the test value, our proposed algorithm can improve the training value significantly as compared to existing methods. Finally, we use a real data study to illustrate the performance of the proposed method.
Collapse
Affiliation(s)
- Daiqi Gao
- Department of Statistics and Operations Research, The University of North Carolina at Chapel Hill Chapel Hill, NC 27599, USA
| | - Yufeng Liu
- Department of Statistics and Operations Research, Department of Genetics, Department of Biostatistics, The University of North Carolina at Chapel Hill Chapel Hill, NC 27599, USA
| | - Donglin Zeng
- Department of Biostatistics, The University of North Carolina at Chapel Hill Chapel Hill, NC 27599, USA
| |
Collapse
|
20
|
Shi C, Zhang S, Lu W, Song R. Statistical inference of the value function for reinforcement learning in infinite‐horizon settings. J R Stat Soc Series B Stat Methodol 2021. [DOI: 10.1111/rssb.12465] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Affiliation(s)
- Chengchun Shi
- London School of Economics and Political Science London UK
| | - Sheng Zhang
- North Carolina State University Raleigh North Carolina USA
| | - Wenbin Lu
- North Carolina State University Raleigh North Carolina USA
| | - Rui Song
- North Carolina State University Raleigh North Carolina USA
| |
Collapse
|
21
|
Abstract
Advances in wearables and digital technology now make it possible to deliver behavioral mobile health interventions to individuals in their everyday life. The micro-randomized trial is increasingly used to provide data to inform the construction of these interventions. In a micro-randomized trial, each individual is repeatedly randomized among multiple intervention options, often hundreds or even thousands of times, over the course of the trial. This work is motivated by multiple micro-randomized trials that have been conducted or are currently in the field, in which the primary outcome is a longitudinal binary outcome. The primary aim of such micro-randomized trials is to examine whether a particular time-varying intervention has an effect on the longitudinal binary outcome, often marginally over all but a small subset of the individual's data. We propose the definition of causal excursion effect that can be used in such primary aim analysis for micro-randomized trials with binary outcomes. Under rather restrictive assumptions one can, based on existing literature, derive a semiparametric, locally efficient estimator of the causal effect. Starting from this estimator, we develop an estimator that can be used as the basis of a primary aim analysis under more plausible assumptions. Simulation studies are conducted to compare the estimators. We illustrate the developed methods using data from the micro-randomized trial, BariFit. In BariFit, the goal is to support weight maintenance for individuals who received bariatric surgery.
Collapse
Affiliation(s)
- Tianchen Qian
- Department of Statistics, University of California, Irvine, Donald Bren Hall, Irvine, California 92697, U.S.A
| | - Hyesun Yoo
- Department of Statistics, University of Michigan, 323 West Hall, 1085 South University, Ann Arbor, Michigan 48109, U.S.A
| | - Predrag Klasnja
- School of Information, University of Michigan, 4364 North Quad, 105 South State Street, Ann Arbor, Michigan 48109, U.S.A
| | - Daniel Almirall
- Department of Statistics, University of Michigan, 323 West Hall, 1085 South University, Ann Arbor, Michigan 48109, U.S.A
| | - Susan A Murphy
- Department of Statistics, Harvard University, 1 Oxford Street, Cambridge, Massachusetts 02138, U.S.A
| |
Collapse
|
22
|
Freeman NLB, Sperger J, El-Zaatari H, Kahkoska AR, Lu M, Valancius M, Virkud AV, Zikry TM, Kosorok MR. Beyond Two Cultures: Cultural Infrastructure for Data-driven Decision Support. Obs Stud 2021; 7:77-94. [PMID: 35106520 PMCID: PMC8802367 DOI: 10.1353/obs.2021.0024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
In the twenty years since Dr. Leo Breiman's incendiary paper Statistical Modeling: The Two Cultures was first published, algorithmic modeling techniques have gone from controversial to commonplace in the statistical community. While the widespread adoption of these methods as part of the contemporary statistician's toolkit is a testament to Dr. Breiman's vision, the number of high-profile failures of algorithmic models suggests that Dr. Breiman's final remark that "the emphasis needs to be on the problem and the data" has been less widely heeded. In the spirit of Dr. Breiman, we detail an emerging research community in statistics - data-driven decision support. We assert that to realize the full potential of decision support, broadly and in the context of precision health, will require a culture of social awareness and accountability, in addition to ongoing attention towards complex technical challenges.
Collapse
Affiliation(s)
- Nikki L B Freeman
- Department of Biostatistics, University of North Carolina at Chapel Hill
| | - John Sperger
- Department of Biostatistics, University of North Carolina at Chapel Hill
| | - Helal El-Zaatari
- Department of Biostatistics, University of North Carolina at Chapel Hill
| | - Anna R Kahkoska
- Department of Nutrition, University of North Carolina School of Medicine
| | - Minxin Lu
- Department of Biostatistics, University of North Carolina at Chapel Hill
| | - Michael Valancius
- Department of Biostatistics, University of North Carolina at Chapel Hill
| | - Arti V Virkud
- Department of Epidemiology, University of North Carolina at Chapel Hill
| | - Tarek M Zikry
- Department of Biostatistics, University of North Carolina at Chapel Hill
| | - Michael R Kosorok
- Department of Biostatistics, University of North Carolina at Chapel Hill
| |
Collapse
|
23
|
Affiliation(s)
- Tze Leung Lai
- Tze Leung Lai is Professor, Department of Statistics, Stanford University, 390 Jane Stanford Way, Stanford, California 94305, USA
| | - Hongsong Yuan
- Hongsong Yuan is Associate Professor, School of Information Management and Engineering, Shanghai University of Finance and Economics, 777 Guoding Road, Shanghai 200433, China
| |
Collapse
|
24
|
Guo W, Zhou XH, Ma S. Estimation of Optimal Individualized Treatment Rules Using a Covariate-Specific Treatment Effect Curve With High-Dimensional Covariates. J Am Stat Assoc 2021. [DOI: 10.1080/01621459.2020.1865167] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022]
Affiliation(s)
- Wenchuan Guo
- Department of Statistics, University of California Riverside, Riverside, CA
- Global Biometric Sciences, Bristol-Myers Squibb, Pennington, NJ
| | - Xiao-Hua Zhou
- Beijing International Center for Mathematical Research, and Department of Biostatistics, Peking University, Beijing, China
| | - Shujie Ma
- Department of Statistics, University of California Riverside, Riverside, CA
| |
Collapse
|
25
|
Koslovsky MD, Hébert ET, Businelle MS, Vannucci M. A Bayesian time-varying effect model for behavioral mHealth data. Ann Appl Stat 2020; 14:1878-1902. [DOI: 10.1214/20-aoas1402] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
26
|
Affiliation(s)
- John Sperger
- Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill North Carolina USA
| | - Nikki L. B. Freeman
- Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill North Carolina USA
| | - Xiaotong Jiang
- Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill North Carolina USA
| | - David Bang
- Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill North Carolina USA
| | - Daniel Marchi
- Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill North Carolina USA
| | - Michael R. Kosorok
- Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill North Carolina USA
| |
Collapse
|
27
|
Affiliation(s)
- Xinkun Nie
- Department of Computer Science, Stanford University, Stanford, CA
| | - Emma Brunskill
- Department of Computer Science, Stanford University, Stanford, CA
| | - Stefan Wager
- Graduate School of Business, Stanford University, Stanford, CA
| |
Collapse
|
28
|
Dong L, Laber E, Goldberg Y, Song R, Yang S. Ascertaining properties of weighting in the estimation of optimal treatment regimes under monotone missingness. Stat Med 2020; 39:3503-3520. [PMID: 32729973 DOI: 10.1002/sim.8678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2019] [Revised: 04/28/2020] [Accepted: 04/30/2020] [Indexed: 11/10/2022]
Abstract
Dynamic treatment regimes operationalize precision medicine as a sequence of decision rules, one per stage of clinical intervention, that map up-to-date patient information to a recommended intervention. An optimal treatment regime maximizes the mean utility when applied to the population of interest. Methods for estimating an optimal treatment regime assume the data to be fully observed, which rarely occurs in practice. A common approach is to first use multiple imputation and then pool the estimators across imputed datasets. However, this approach requires estimating the joint distribution of patient trajectories, which can be high-dimensional, especially when there are multiple stages of intervention. We examine the application of inverse probability weighted estimating equations as an alternative to multiple imputation in the context of monotonic missingness. This approach applies to a broad class of estimators of an optimal treatment regime including both Q-learning and a generalization of outcome weighted learning. We establish consistency under mild regularity conditions and demonstrate its advantages in finite samples using a series of simulation experiments and an application to a schizophrenia study.
Collapse
Affiliation(s)
- Lin Dong
- Department of Statistics, North Carolina State University, Raleigh, North Carolina, USA
| | - Eric Laber
- Department of Statistics, North Carolina State University, Raleigh, North Carolina, USA
| | - Yair Goldberg
- Department of Statistics, Technion Israel Institute of Technology, Haifa, Israel
| | - Rui Song
- Department of Statistics, North Carolina State University, Raleigh, North Carolina, USA
| | - Shu Yang
- Department of Statistics, North Carolina State University, Raleigh, North Carolina, USA
| |
Collapse
|
29
|
Liao P, Klasnja P, Murphy S. Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health. J Am Stat Assoc 2020; 116:382-391. [PMID: 33814653 PMCID: PMC8014957 DOI: 10.1080/01621459.2020.1807993] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2019] [Revised: 03/03/2020] [Accepted: 08/03/2020] [Indexed: 10/23/2022]
Abstract
Due to the recent advancements in wearables and sensing technology, health scientists are increasingly developing mobile health (mHealth) interventions. In mHealth interventions, mobile devices are used to deliver treatment to individuals as they go about their daily lives. These treatments are generally designed to impact a near time, proximal outcome such as stress or physical activity. The mHealth intervention policies, often called just-in-time adaptive interventions, are decision rules that map a individual's current state (e.g., individual's past behaviors as well as current observations of time, location, social activity, stress and urges to smoke) to a particular treatment at each of many time points. The vast majority of current mHealth interventions deploy expert-derived policies. In this paper, we provide an approach for conducting inference about the performance of one or more such policies using historical data collected under a possibly different policy. Our measure of performance is the average of proximal outcomes over a long time period should the particular mHealth policy be followed. We provide an estimator as well as confidence intervals. This work is motivated by HeartSteps, an mHealth physical activity intervention.
Collapse
Affiliation(s)
- Peng Liao
- Department of Statistics, University of Michigan
| | | | | |
Collapse
|
30
|
Abstract
Personalized policy represents a paradigm shift from one-decision-rule-for-all users to an individualized decision rule for each user. Developing personalized policy in mobile health applications imposes challenges. First, for lack of adherence, data from each user are limited. Second, unmeasured contextual factors can potentially impact on decision making. Aiming to optimize immediate rewards, we propose using a generalized linear mixed modeling framework where population features and individual features are modeled as fixed and random effects, respectively, and synthesized to form the personalized policy. The group lasso type penalty is imposed to avoid overfitting of individual deviations from the population model. We examine the conditions under which the proposed method work in the presence of time-varying endogenous covariates, and provide conditional optimality and marginal consistency results of the expected immediate outcome under the estimated policies. We apply our method to develop personalized push ("prompt") schedules in 294 app users, with the goal to maximize the prompt response rate given past app usage and other contextual factors. The proposed method compares favorably to existing estimation methods including using the R function "glmer" in a simulation study.
Collapse
Affiliation(s)
- Xinyu Hu
- Department of Biostatistics, Columbia University
| | - Min Qian
- Department of Biostatistics, Columbia University
| | - Bin Cheng
- Department of Biostatistics, Columbia University
| | | |
Collapse
|
31
|
Cui Y, Tchetgen ET. A semiparametric instrumental variable approach to optimal treatment regimes under endogeneity. J Am Stat Assoc 2020; 116:162-173. [PMID: 33994604 PMCID: PMC8118566 DOI: 10.1080/01621459.2020.1783272] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2019] [Revised: 02/05/2020] [Accepted: 06/09/2020] [Indexed: 01/23/2023]
Abstract
There is a fast-growing literature on estimating optimal treatment regimes based on randomized trials or observational studies under a key identifying condition of no unmeasured confounding. Because confounding by unmeasured factors cannot generally be ruled out with certainty in observational studies or randomized trials subject to noncompliance, we propose a general instrumental variable approach to learning optimal treatment regimes under endogeneity. Specifically, we establish identification of both value function E [ Y D ( L ) ] for a given regime D and optimal regimes arg max D E [ Y D ( L ) ] with the aid of a binary instrumental variable, when no unmeasured confounding fails to hold. We also construct novel multiply robust classification-based estimators. Furthermore, we propose to identify and estimate optimal treatment regimes among those who would comply to the assigned treatment under a monotonicity assumption. In this latter case, we establish the somewhat surprising result that complier optimal regimes can be consistently estimated without directly collecting compliance information and therefore without the complier average treatment effect itself being identified. Our approach is illustrated via extensive simulation studies and a data application on the effect of child rearing on labor participation.
Collapse
Affiliation(s)
- Yifan Cui
- Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104
| | - Eric Tchetgen Tchetgen
- Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104
| |
Collapse
|