Controlling blood glucose variability under uncertainty using reinforcement learning and Gaussian processes

Auto adaptation of closed-loop insulin delivery system using continuous reward functions and incremental discretization

Several closed or hybrid loop controllers for Blood Glucose (BG) regulation, which are also known as Artificial Pancreas (AP) Systems or Automated Insulin Delivery systems (AIDs), are in development worldwide. Most AIDs are designed and evaluated for short-term performance, with a particular emphasis on the post-meal period. However, if controllers are not adapted properly to account for variations in physiology that affect Insulin Sensitivity (IS), the AIDs may perform inadequately. In this work, the performance of two Reinforcement Learning (RL) agents trained under both piecewise and continuous reward functions is evaluated in-silico for long-term adaptation of a Fully Automated Insulin Delivery (fAID) system. An automatic adaptive discretization scheme that expands the state space as needed is also implemented to avoid disproportionate state space exploration. The proposed agents are evaluated for long-term adaptation of the Automatic Regulation of Glucose (ARG) algorithm, considering variations in IS. Results show that both RL agents have improved performance compared to a rule-based decision-making approach and the baseline controller for the majority of the adult population. Moreover, the use of a continuous shaped reward function proves to enhance the performance of the agents further than a piecewise one.

Evaluation of blood glucose level control in type 1 diabetic patients using deep reinforcement learning

Diabetes mellitus is a disease associated with abnormally high levels of blood glucose due to a lack of insulin. Combining an insulin pump and continuous glucose monitor with a control algorithm to deliver insulin is an alternative to patient self-management of insulin doses to control blood glucose levels in diabetes mellitus patients. In this work, we propose a closed-loop control for blood glucose levels based on deep reinforcement learning. We describe the initial evaluation of several alternatives conducted on a realistic simulator of the glucoregulatory system and propose a particular implementation strategy based on reducing the frequency of the observations and rewards passed to the agent, and using a simple reward function. We train agents with that strategy for three groups of patient classes, evaluate and compare it with alternative control baselines. Our results show that our method is able to outperform baselines as well as similar recent proposals, by achieving longer periods of safe glycemic state and low risk.

Data-Driven Robust Control Using Reinforcement Learning

This paper proposes a robust control design method using reinforcement learning for controlling partially-unknown dynamical systems under uncertain conditions. The method extends the optimal reinforcement learning algorithm with a new learning technique based on the robust control theory. By learning from the data, the algorithm proposes actions that guarantee the stability of the closed-loop system within the uncertainties estimated also from the data. Control policies are calculated by solving a set of linear matrix inequalities. The controller was evaluated using simulations on a blood glucose model for patients with Type 1 diabetes. Simulation results show that the proposed methodology is capable of safely regulating the blood glucose within a healthy level under the influence of measurement and process noises. The controller has also significantly reduced the post-meal fluctuation of the blood glucose. A comparison between the proposed algorithm and the existing optimal reinforcement learning algorithm shows the improved robustness of the closed-loop system using our method.

GLYFE: review and benchmark of personalized glucose predictive models in type 1 diabetes

Due to the sensitive nature of diabetes-related data, preventing them from being easily shared between studies, and the wide discrepancies in their data processing pipeline, progress in the field of glucose prediction is hard to assess. To address this issue, we introduce GLYFE (GLYcemia Forecasting Evaluation), a benchmark of machine learning-based glucose predictive models. We present the accuracy and clinical acceptability of nine different models coming from the literature, from standard autoregressive to more complex neural network-based models. These results are obtained on two different datasets, namely UVA/Padova Type 1 Diabetes Metabolic Simulator (T1DMS) and Ohio Type-1 Diabetes Mellitus (OhioT1DM), featuring artificial and real type 1 diabetic patients respectively. By providing extensive details about the data flow as well as by providing the whole source code of the benchmarking process, we ensure the reproducibility of the results and the usability of the benchmark by the community. Those results serve as a basis of comparison for future studies. In a field where data are hard to obtain, and where the comparison of results from different studies is often irrelevant, GLYFE gives the opportunity of gathering researchers around a standardized common environment.

Reinforcement learning for intelligent healthcare applications: A survey

Discovering new treatments and personalizing existing ones is one of the major goals of modern clinical research. In the last decade, Artificial Intelligence (AI) has enabled the realization of advanced intelligent systems able to learn about clinical treatments and discover new medical knowledge from the huge amount of data collected. Reinforcement Learning (RL), which is a branch of Machine Learning (ML), has received significant attention in the medical community since it has the potentiality to support the development of personalized treatments in accordance with the more general precision medicine vision. This report presents a review of the role of RL in healthcare by investigating past work, and highlighting any limitations and possible future contributions.

Control of Blood Glucose for Type-1 Diabetes by Using Reinforcement Learning with Feedforward Algorithm

Background

Type-1 diabetes is a condition caused by the lack of insulin hormone, which leads to an excessive increase in blood glucose level. The glucose kinetics process is difficult to control due to its complex and nonlinear nature and with state variables that are difficult to measure.

Methods

This paper proposes a method for automatically calculating the basal and bolus insulin doses for patients with type-1 diabetes using reinforcement learning with feedforward controller. The algorithm is designed to keep the blood glucose stable and directly compensate for the external events such as food intake. Its performance was assessed using simulation on a blood glucose model. The usage of the Kalman filter with the controller was demonstrated to estimate unmeasurable state variables.

Results

Comparison simulations between the proposed controller with the optimal reinforcement learning and the proportional-integral-derivative controller show that the proposed methodology has the best performance in regulating the fluctuation of the blood glucose. The proposed controller also improved the blood glucose responses and prevented hypoglycemia condition. Simulation of the control system in different uncertain conditions provided insights on how the inaccuracies of carbohydrate counting and meal-time reporting affect the performance of the control system.

Conclusion

The proposed controller is an effective tool for reducing postmeal blood glucose rise and for countering the effects of external known events such as meal intake and maintaining blood glucose at a healthy level under uncertainties.

Estimation of plasma insulin concentration under glycemic variability using nonlinear filtering techniques

The ultimate goal of an artificial pancreas is finding the optimal insulin rates that can effectively reduce high blood glucose (BG) levels in type 1 diabetic patients. To achieve this, most closed-loop control strategies need to compute the optimal insulin action on the basis of precedent glucose and insulin levels. Unlike glucose levels which can be measured in real-time, unavailability of insulin sensors makes it essential the use of mathematical models to estimate plasma insulin concentrations. Between others, filtering techniques based on a generalization of the Kalman filter (KF) have been the most widely applied in the estimation of hidden states in nonlinear dynamic systems. Nevertheless, poor predictability of BG levels is a key issue since the glucose-insulin dynamics presents great inter- and intra-patient variability. Here, the question arises as to whether glycemic variability is not properly taken into account in models formulations and whether or it would compromise proper estimation of plasma insulin concentration. In order to tackle this point, a deterministic model describing glucose-insulin interaction plus a stochastic process to account for BG fluctuations were incorporated into the extended (EKF), cubature (CKF) and unscented (UKF) configurations of the Kalman filter to provide an estimate of the plasma insulin concentration. We found that for low glycemic variability, insulin state estimation can be attained with acceptable accuracy; however, as glycemic variability rises, Kalman filters rapidly degrade their performance as a consequence of large nonlinearities.

A review of personalized blood glucose prediction strategies for T1DM patients

This paper presents a methodological review of models for predicting blood glucose (BG) concentration, risks and BG events. The surveyed models are classified into three categories, and they are presented in summary tables containing the most relevant data regarding the experimental setup for fitting and testing each model as well as the input signals and the performance metrics. Each category exhibits trends that are presented and discussed. This document aims to be a compact guide to determine the modeling options that are currently being exploited for personalized BG prediction.