Reinforcement learning in clinical medicine: a method to optimize dynamic treatment regime over time

Optimizing warfarin dosing for patients with atrial fibrillation using machine learning

While novel oral anticoagulants are increasingly used to reduce risk of stroke in patients with atrial fibrillation, vitamin K antagonists such as warfarin continue to be used extensively for stroke prevention across the world. While effective in reducing the risk of strokes, the complex pharmacodynamics of warfarin make it difficult to use clinically, with many patients experiencing under- and/or over- anticoagulation. In this study we employed a novel implementation of deep reinforcement learning to provide clinical decision support to optimize time in therapeutic International Normalized Ratio (INR) range. We used a novel semi-Markov decision process formulation of the Batch-Constrained deep Q-learning algorithm to develop a reinforcement learning model to dynamically recommend optimal warfarin dosing to achieve INR of 2.0-3.0 for patients with atrial fibrillation. The model was developed using data from 22,502 patients in the warfarin treated groups of the pivotal randomized clinical trials of edoxaban (ENGAGE AF-TIMI 48), apixaban (ARISTOTLE) and rivaroxaban (ROCKET AF). The model was externally validated on data from 5730 warfarin-treated patients in a fourth trial of dabigatran (RE-LY) using multilevel regression models to estimate the relationship between center-level algorithm consistent dosing, time in therapeutic INR range (TTR), and a composite clinical outcome of stroke, systemic embolism or major hemorrhage. External validation showed a positive association between center-level algorithm-consistent dosing and TTR (R2 = 0.56). Each 10% increase in algorithm-consistent dosing at the center level independently predicted a 6.78% improvement in TTR (95% CI 6.29, 7.28; p < 0.001) and a 11% decrease in the composite clinical outcome (HR 0.89; 95% CI 0.81, 1.00; p = 0.015). These results were comparable to those of a rules-based clinical algorithm used for benchmarking, for which each 10% increase in algorithm-consistent dosing independently predicted a 6.10% increase in TTR (95% CI 5.67, 6.54, p < 0.001) and a 10% decrease in the composite outcome (HR 0.90; 95% CI 0.83, 0.98, p = 0.018). Our findings suggest that a deep reinforcement learning algorithm can optimize time in therapeutic range for patients taking warfarin. A digital clinical decision support system to promote algorithm-consistent warfarin dosing could optimize time in therapeutic range and improve clinical outcomes in atrial fibrillation globally.

Deep reinforcement learning identifies personalized intermittent androgen deprivation therapy for prostate cancer

The evolution of drug resistance leads to treatment failure and tumor progression. Intermittent androgen deprivation therapy (IADT) helps responsive cancer cells compete with resistant cancer cells in intratumoral competition. However, conventional IADT is population-based, ignoring the heterogeneity of patients and cancer. Additionally, existing IADT relies on pre-determined thresholds of prostate-specific antigen to pause and resume treatment, which is not optimized for individual patients. To address these challenges, we framed a data-driven method in two steps. First, we developed a time-varied, mixed-effect and generative Lotka-Volterra (tM-GLV) model to account for the heterogeneity of the evolution mechanism and the pharmacokinetics of two ADT drugs Cyproterone acetate and Leuprolide acetate for individual patients. Then, we proposed a reinforcement-learning-enabled individualized IADT framework, namely, I$^{2}$ADT, to learn the patient-specific tumor dynamics and derive the optimal drug administration policy. Experiments with clinical trial data demonstrated that the proposed I$^{2}$ADT can significantly prolong the time to progression of prostate cancer patients with reduced cumulative drug dosage. We further validated the efficacy of the proposed methods with a recent pilot clinical trial data. Moreover, the adaptability of I$^{2}$ADT makes it a promising tool for other cancers with the availability of clinical data, where treatment regimens might need to be individualized based on patient characteristics and disease dynamics. Our research elucidates the application of deep reinforcement learning to identify personalized adaptive cancer therapy.

A comprehensive review of machine learning algorithms and their application in geriatric medicine: present and future

The increasing access to health data worldwide is driving a resurgence in machine learning research, including data-hungry deep learning algorithms. More computationally efficient algorithms now offer unique opportunities to enhance diagnosis, risk stratification, and individualised approaches to patient management. Such opportunities are particularly relevant for the management of older patients, a group that is characterised by complex multimorbidity patterns and significant interindividual variability in homeostatic capacity, organ function, and response to treatment. Clinical tools that utilise machine learning algorithms to determine the optimal choice of treatment are slowly gaining the necessary approval from governing bodies and being implemented into healthcare, with significant implications for virtually all medical disciplines during the next phase of digital medicine. Beyond obtaining regulatory approval, a crucial element in implementing these tools is the trust and support of the people that use them. In this context, an increased understanding by clinicians of artificial intelligence and machine learning algorithms provides an appreciation of the possible benefits, risks, and uncertainties, and improves the chances for successful adoption. This review provides a broad taxonomy of machine learning algorithms, followed by a more detailed description of each algorithm class, their purpose and capabilities, and examples of their applications, particularly in geriatric medicine. Additional focus is given on the clinical implications and challenges involved in relying on devices with reduced interpretability and the progress made in counteracting the latter via the development of explainable machine learning.

Artificial intelligence in hematological diagnostics: Game changer or gadget?

The future of clinical diagnosis and treatment of hematologic diseases will inevitably involve the integration of artificial intelligence (AI)-based systems into routine practice to support the hematologists' decision making. Several studies have shown that AI-based models can already be used to automatically differentiate cells, reliably detect malignant cell populations, support chromosome banding analysis, and interpret clinical variants, contributing to early disease detection and prognosis. However, even the best tool can become useless if it is misapplied or the results are misinterpreted. Therefore, in order to comprehensively judge and correctly apply newly developed AI-based systems, the hematologist must have a basic understanding of the general concepts of machine learning. In this review, we provide the hematologist with a comprehensive overview of various machine learning techniques, their current implementations and approaches in different diagnostic subfields (e.g., cytogenetics, molecular genetics), and the limitations and unresolved challenges of the systems.

A Bayesian Network Approach to Explainable Reinforcement Learning with Distal Information

Nowadays, Artificial Intelligence systems have expanded their competence field from research to industry and daily life, so understanding how they make decisions is becoming fundamental to reducing the lack of trust between users and machines and increasing the transparency of the model. This paper aims to automate the generation of explanations for model-free Reinforcement Learning algorithms by answering "why" and "why not" questions. To this end, we use Bayesian Networks in combination with the NOTEARS algorithm for automatic structure learning. This approach complements an existing framework very well and demonstrates thus a step towards generating explanations with as little user input as possible. This approach is computationally evaluated in three benchmarks using different Reinforcement Learning methods to highlight that it is independent of the type of model used and the explanations are then rated through a human study. The results obtained are compared to other baseline explanation models to underline the satisfying performance of the framework presented in terms of increasing the understanding, transparency and trust in the action chosen by the agent.

Reinforcement learning strategies in cancer chemotherapy treatments: A review

BACKGROUND AND OBJECTIVE

Cancer is one of the major causes of death worldwide and chemotherapies are the most significant anti-cancer therapy, in spite of the emerging precision cancer medicines in the last 2 decades. The growing interest in developing the effective chemotherapy regimen with optimal drug dosing schedule to benefit the clinical cancer patients has spawned innovative solutions involving mathematical modeling since the chemotherapy regimens are administered cyclically until the futility or the occurrence of intolerable adverse events. Thus, in this present work, we reviewed the emerging trends involved in forming a computational solution from the aspect of reinforcement learning.

METHODS

Initially, this survey in-depth focused on the details of the dynamic treatment regimens from a broad perspective and then narrowed down to inspirations from reinforcement learning that were advantageous to chemotherapy dosing, including both offline reinforcement learning and supervised reinforcement learning.

RESULTS

The insights established in the chemotherapy-planning problem associated with the Reinforcement Learning (RL) has been discussed in this study. It showed that the researchers were able to widen their perspectives in comprehending the theoretical basis, dynamic treatment regimens (DTR), use of the adaptive control on DTR, and the associated RL techniques.

CONCLUSIONS

This study reviewed the recent researches relevant to the topic, and highlighted the challenges, open questions, possible solutions, and future steps in inventing a realistic solution for the aforementioned problem.

Projection based inverse reinforcement learning for the analysis of dynamic treatment regimes

Dynamic Treatment Regimes (DTRs) are adaptive treatment strategies that allow clinicians to personalize dynamically the treatment for each patient based on their step-by-step response to their treatment. There are a series of predefined alternative treatments for each disease and any patient may associate with one of these treatments according to his/her demographics. DTRs for a certain disease are studied and evaluated by means of statistical approaches where patients are randomized at each step of the treatment and their responses are observed. Recently, the Reinforcement Learning (RL) paradigm has also been applied to determine DTRs. However, such approaches may be limited by the need to design a true reward function, which may be difficult to formalize when the expert knowledge is not well assessed, as when the DTR is in the design phase. To address this limitation, an extension of the RL paradigm, namely Inverse Reinforcement Learning (IRL), has been adopted to learn the reward function from data, such as those derived from DTR trials. In this paper, we define a Projection Based Inverse Reinforcement Learning (PB-IRL) approach to learn the true underlying reward function for given demonstrations (DTR trials). Such a reward function can be used both to evaluate the set of DTRs determined for a certain disease, as well as to enable an RL-based intelligent agent to self-learn the best way and then act as a decision support system for the clinician.

Machine learning applications in gynecological cancer: A critical review

Machine Learning (ML) represents a computer science capable of generating predictive models, by exposure to raw, training data, without being rigidly programmed. Over the last few years, ML has gained attention within the field of oncology, with considerable strides in both diagnostic, predictive, and prognostic spectrum of malignancies, but also as a catalyst of cancer research. In this review, we discuss the state of ML applications on gynecologic oncology and systematically address major technical and ethical concerns, with respect to their real-world medical practice translation. Undoubtedly, advances in ML will enable the analysis of large, rather complex, datasets for improved, cost-effective, and efficient clinical decisions.

Machine learning-based ABA treatment recommendation and personalization for autism spectrum disorder: an exploratory study

Autism spectrum is a brain development condition that impairs an individual's capacity to communicate socially and manifests through strict routines and obsessive-compulsive behavior. Applied behavior analysis (ABA) is the gold-standard treatment for autism spectrum disorder (ASD). However, as the number of ASD cases increases, there is a substantial shortage of licensed ABA practitioners, limiting the timely formulation, revision, and implementation of treatment plans and goals. Additionally, the subjectivity of the clinician and a lack of data-driven decision-making affect treatment quality. We address these obstacles by applying two machine learning algorithms to recommend and personalize ABA treatment goals for 29 study participants with ASD. The patient similarity and collaborative filtering methods predicted ABA treatment with an average accuracy of 81-84%, with a normalized discounted cumulative gain of 79-81% (NDCG) compared to clinician-prepared ABA treatment recommendations. Additionally, we assess the two models' treatment efficacy (TE) by measuring the percentage of recommended treatment goals mastered by the study participants. The proposed treatment recommendation and personalization strategy are generalizable to other intervention methods in addition to ABA and for other brain disorders. This study was registered as a clinical trial on November 5, 2020 with trial registration number CTRI/2020/11/028933.

Effects of a continuous nursing care model on elderly patients with total hip arthroplasty: a randomized controlled trial

BACKGROUND

Continuous nursing care (CNC) is an extended service based on meeting the needs of discharged patients for post-discharge treatment and rehabilitation. This research aimed to investigate the effects of CNC on older patients with total hip arthroplasty and to offer a scientific basis for improving the prognosis.

METHODS

A total of 134 patients with total hip arthroplasty were randomly divided into the control group (n = 67) and the intervention group (n = 67). The control group was treated by conventional nursing care and the intervention group was treated by CNC. Harris hip score, Barthel index, the activities of daily living (ADL) scale, self-rating depression scale (SDS) and self-rating anxiety scale (SAS) in these two groups were evaluated. Demographic characteristics between groups were analyzed by unpaired t test. The observation indexes between groups were assessed by two-way ANOVA test followed by Tukey's multiple comparisons test.

RESULTS

The scores of Harris hip score, Barthel index, ADL, SDS and SAS in the intervention group after intervention and after follow-up were better than the intervention group before intervention (all p < 0.01). Meanwhile, the scores of Harris hip score, Barthel index, ADL, SDS and SAS in the intervention group were better than the control group both after intervention and after follow-up (all p < 0.01).

CONCLUSION

In conclusion, CNC showed better efficacy than conventional nursing care in promoting hip joint function recovery, improving quality of life and alleviating anxiety and depression for older patients with total hip arthroplasty.

Individualized resuscitation strategy for septic shock formalized by finite mixture modeling and dynamic treatment regimen

BACKGROUND

Septic shock comprises a heterogeneous population, and individualized resuscitation strategy is of vital importance. The study aimed to identify subclasses of septic shock with non-supervised learning algorithms, so as to tailor resuscitation strategy for each class.

METHODS

Patients with septic shock in 25 tertiary care teaching hospitals in China from January 2016 to December 2017 were enrolled in the study. Clinical and laboratory variables were collected on days 0, 1, 2, 3 and 7 after ICU admission. Subclasses of septic shock were identified by both finite mixture modeling and K-means clustering. Individualized fluid volume and norepinephrine dose were estimated using dynamic treatment regime (DTR) model to optimize the final mortality outcome. DTR models were validated in the eICU Collaborative Research Database (eICU-CRD) dataset.

RESULTS

A total of 1437 patients with a mortality rate of 29% were included for analysis. The finite mixture modeling and K-means clustering robustly identified five classes of septic shock. Class 1 (baseline class) accounted for the majority of patients over all days; class 2 (critical class) had the highest severity of illness; class 3 (renal dysfunction) was characterized by renal dysfunction; class 4 (respiratory failure class) was characterized by respiratory failure; and class 5 (mild class) was characterized by the lowest mortality rate (21%). The optimal fluid infusion followed the resuscitation/de-resuscitation phases with initial large volume infusion and late restricted volume infusion. While class 1 transitioned to de-resuscitation phase on day 3, class 3 transitioned on day 1. Classes 1 and 3 might benefit from early use of norepinephrine, and class 2 can benefit from delayed use of norepinephrine while waiting for adequate fluid infusion.

CONCLUSIONS

Septic shock comprises a heterogeneous population that can be robustly classified into five phenotypes. These classes can be easily identified with routine clinical variables and can help to tailor resuscitation strategy in the context of precise medicine.

Recommendations for patient similarity classes: results of the AMIA 2019 workshop on defining patient similarity

Defining patient-to-patient similarity is essential for the development of precision medicine in clinical care and research. Conceptually, the identification of similar patient cohorts appears straightforward; however, universally accepted definitions remain elusive. Simultaneously, an explosion of vendors and published algorithms have emerged and all provide varied levels of functionality in identifying patient similarity categories. To provide clarity and a common framework for patient similarity, a workshop at the American Medical Informatics Association 2019 Annual Meeting was convened. This workshop included invited discussants from academics, the biotechnology industry, the FDA, and private practice oncology groups. Drawing from a broad range of backgrounds, workshop participants were able to coalesce around 4 major patient similarity classes: (1) feature, (2) outcome, (3) exposure, and (4) mixed-class. This perspective expands into these 4 subtypes more critically and offers the medical informatics community a means of communicating their work on this important topic.

Reinforcement Learning in Neurocritical and Neurosurgical Care: Principles and Possible Applications

Dynamic decision-making was essential in the clinical care of surgical patients. Reinforcement learning (RL) algorithm is a computational method to find sequential optimal decisions among multiple suboptimal options. This review is aimed at introducing RL's basic concepts, including three basic components: the state, the action, and the reward. Most medical studies using reinforcement learning methods were trained on a fixed observational dataset. This paper also reviews the literature of existing practical applications using reinforcement learning methods, which can be further categorized as a statistical RL study and a computational RL study. The review proposes several potential aspects where reinforcement learning can be applied in neurocritical and neurosurgical care. These include sequential treatment strategies of intracranial tumors and traumatic brain injury and intraoperative endoscope motion control. Several limitations of reinforcement learning are representations of basic components, the positivity violation, and validation methods.

Secure and Robust Machine Learning for Healthcare: A Survey

Recent years have witnessed widespread adoption of machine learning (ML)/deep learning (DL) techniques due to their superior performance for a variety of healthcare applications ranging from the prediction of cardiac arrest from one-dimensional heart signals to computer-aided diagnosis (CADx) using multi-dimensional medical images. Notwithstanding the impressive performance of ML/DL, there are still lingering doubts regarding the robustness of ML/DL in healthcare settings (which is traditionally considered quite challenging due to the myriad security and privacy issues involved), especially in light of recent results that have shown that ML/DL are vulnerable to adversarial attacks. In this paper, we present an overview of various application areas in healthcare that leverage such techniques from security and privacy point of view and present associated challenges. In addition, we present potential methods to ensure secure and privacy-preserving ML for healthcare applications. Finally, we provide insight into the current research challenges and promising directions for future research.