Simulation models for in silico testing of closed-loop glucose controllers in type 1 diabetes

Evaluating Deep Q-Learning Algorithms for Controlling Blood Glucose in In Silico Type 1 Diabetes

Patients with type 1 diabetes must continually decide how much insulin to inject before each meal to maintain blood glucose levels within a healthy range. Recent research has worked on a solution for this burden, showing the potential of reinforcement learning as an emerging approach for the task of controlling blood glucose levels. In this paper, we test and evaluate several deep Q-learning algorithms for automated and personalized blood glucose regulation in an in silico type 1 diabetes patient with the goal of estimating and delivering proper insulin doses. The proposed algorithms are model-free approaches with no prior information about the patient. We used the Hovorka model with meal variation and carbohydrate counting errors to simulate the patient included in this work. Our experiments compare different deep Q-learning extensions showing promising results controlling blood glucose levels, with some of the proposed algorithms outperforming standard baseline treatment.

A simulator with realistic and challenging scenarios for virtual T1D patients undergoing CSII and MDI therapy

In silico simulations have become essential for the development of diabetes treatments. However, currently available simulators are not challenging enough and often suffer from limitations in insulin and meal absorption variability, which is unable to realistically reflect the dynamics of people with type 1 diabetes (T1D). Additionally, T1D simulators are mainly designed for the testing of continuous subcutaneous insulin infusion (CSII) therapies. In this work, a simulator is presented that includes a generated virtual patient (VP) cohort and both fast- and long-acting Glargine-100 U/ml (Gla-100), Glargine-300 U/ml (Gla-300), and Degludec-100 U/ml (Deg-100) insulin models. Therefore, in addition to CSII therapies, multiple daily injections (MDI) therapies can also be tested. The Hovorka model and its published parameter probability distributions were used to generate cohorts of VPs that represent a T1D population. Valid patients are filtered through restrictions that guarantee that they are physiologically acceptable. To obtain more realistic scenarios, basal insulin profile patterns from the literature have been used to identify variability in insulin sensitivity. A library of mixed meals identified from real data has also been included. This work presents and validates a methodology for the creation of realistic VP cohorts that include physiological variability and a simulator that includes challenging and realistic scenarios for in silico testing. A cohort of 47 VPs has been generated and in silico simulations of both CSII and MDI therapies were performed in open-loop. The simulation outcome metrics were contrasted with literature results.

Mathematical Models of Meal Amount and Timing Variability With Implementation in the Type-1 Diabetes Patient Decision Simulator

BACKGROUND

In type 1 diabetes (T1D) research, in-silico clinical trials (ISCTs) have proven effective in accelerating the development of new therapies. However, published simulators lack a realistic description of some aspects of patient lifestyle which can remarkably affect glucose control. In this paper, we develop a mathematical description of meal carbohydrates (CHO) amount and timing, with the aim to improve the meal generation module in the T1D Patient Decision Simulator (T1D-PDS) published in Vettoretti et al.

METHODS

Data of 32 T1D subjects under free-living conditions for 4874 days were used. Univariate probability density function (PDF) parametric models with different candidate shapes were fitted, individually, against sample distributions of: CHO amounts of breakfast (CHOB), lunch (CHOL), dinner (CHOD), and snack (CHOS); breakfast timing (TB); and time between breakfast-lunch (TBL) and between lunch-dinner (TLD). Furthermore, a support vector machine (SVM) classifier was developed to predict the occurrence of a snack in future fixed-length time windows. Once embedded inside the T1D-PDS, an ISCT was performed.

RESULTS

Resulting PDF models were: gamma (CHOB, CHOS), lognormal (CHOL, TB), loglogistic (CHOD), and generalized-extreme-values (TBL, TLD). The SVM showed a classification accuracy of 0.8 over the test set. The distributions of simulated meal data were not statistically different from the distributions of the real data used to develop the models (α = 0.05).

CONCLUSIONS

The models of meal amount and timing variability developed are suitable for describing real data. Their inclusion in modules that describe patient behavior in the T1D-PDS can permit investigators to perform more realistic, reliable, and insightful ISCTs.

In-Silico Evaluation of Glucose Regulation Using Policy Gradient Reinforcement Learning for Patients with Type 1 Diabetes Mellitus

In this paper, we test and evaluate policy gradient reinforcement learning for automated blood glucose control in patients with Type 1 Diabetes Mellitus. Recent research has shown that reinforcement learning is a promising approach to accommodate the need for individualized blood glucose level control algorithms. The motivation for using policy gradient algorithms comes from the fact that adaptively administering insulin is an inherently continuous task. Policy gradient algorithms are known to be superior in continuous high-dimensional control tasks. Previously, most of the approaches for automated blood glucose control using reinforcement learning has used a finite set of actions. We use the Trust-Region Policy Optimization algorithm in this work. It represents the state of the art for deep policy gradient algorithms. The experiments are carried out in-silico using the Hovorka model, and stochastic behavior is modeled through simulated carbohydrate counting errors to illustrate the full potential of the framework. Furthermore, we use a model-free approach where no prior information about the patient is given to the algorithm. Our experiments show that the reinforcement learning agent is able to compete with and sometimes outperform state-of-the-art model predictive control in blood glucose regulation.

Parameter estimation of a meal glucose-insulin model for TIDM patients from therapy historical data

The effect of meal on blood glucose concentration is a key issue in diabetes mellitus because its estimation could be very useful in therapy decisions. In the case of type 1 diabetes mellitus (T1DM), the therapy based on automatic insulin delivery requires a closed-loop control system to maintain euglycaemia even in the postprandial state. Thus, the mathematical modelling of glucose metabolism is relevant to predict the metabolic state of a patient. Moreover, the eating habits are characteristic of each person, so it is of interest that the mathematical models of meal intake allow to personalise the glycaemic state of the patient using therapy historical data, that is, daily measurements of glucose and records of carbohydrate intake and insulin supply. Thus, here, a model of glucose metabolism that includes the effects of meal is analysed in order to establish criteria for data-based personalisation. The analysis includes the sensitivity and identifiability of the parameters, and the parameter estimation problem was resolved via two algorithms: particle swarm optimisation and evonorm. The results show that the mathematical model can be a useful tool to estimate the glycaemic status of a patient and personalise it according to her/his historical data.

Simulation Software for Assessment of Nonlinear and Adaptive Multivariable Control Algorithms: Glucose - Insulin Dynamics in Type 1 Diabetes

A simulator for testing automatic control algorithms for nonlinear systems with time-varying parameters, variable time delays, and uncertainties is developed. It is based on simulation of virtual patients with Type 1 diabetes (T1D). Nonlinear models are developed to describe glucose concentration (GC) variations based on user-defined scenarios for meal consumption, insulin administration, and physical activity. They compute GC values and physiological variables, such as heart rate, skin temperature, accelerometer, and energy expenditure, that are indicative of physical activities affecting GC dynamics. This is the first simulator designed for assessment of multivariable controllers that consider supplemental physiological variables in addition to GC measurements to improve glycemic control. Virtual patients are generated from distributions of identified model parameters using clinical data. The simulator will enable testing and evaluation of new control algorithms proposed for automated insulin delivery as well as various control algorithms for nonlinear systems with uncertainties, time-varying parameters and delays.

Control Limitations in Models of T1DM and the Robustness of Optimal Insulin Delivery

BACKGROUND

In insulin therapy, the blood glucose level is constrained from below by the hypoglycemic threshold, that is, the blood glucose level must remain above this threshold. It has been shown that this constraint fundamentally limits the ability to lower the maxima of the blood glucose level predicted by many mathematical models of glucose metabolism. However, it is desirable to minimize hyperglycemia as well. Hence, a desirable insulin input is one that minimizes the maximum glucose concentration while causing it to remain above the hypoglycemic, or higher, threshold. It has been shown that this input, which we call optimal, is characterized by glucose profiles for which either each maximum of the glucose concentration is followed by a minimum or each minimum is followed by a maximum.

METHODS

We discuss the implication of this inherent control limitation for clinical practice and test, through simulation, the robustness of the optimal input to a number of different model and parameter uncertainties. We further develop guidelines on how to design an optimal insulin input that is robust to such uncertainties.

RESULTS

The optimal input is in general not robust to uncertainties. However, a number of strategies may be used to ensure the blood glucose level remains above the hypoglycemic threshold and the maximum blood glucose level achieved is less than that achieved by standard therapy.

CONCLUSIONS

An understanding of the limitations on the controllability of the blood glucose level is important for future treatment improvements and the development of artificial pancreas systems.

Computer Simulation Model to Train Medical Personnel on Glucose Clamp Procedures

OBJECTIVE

A glucose clamp procedure is the most reliable way to quantify insulin pharmacokinetics and pharmacodynamics, but skilled and trained research personnel are required to frequently adjust the glucose infusion rate. A computer environment that simulates glucose clamp experiments can be used for efficient personnel training and development and testing of algorithms for automated glucose clamps.

METHODS

We built 17 virtual healthy subjects (mean age, 25±6 years; mean body mass index, 22.2±3 kg/m²), each comprising a mathematical model of glucose regulation and a unique set of parameters. Each virtual subject simulates plasma glucose and insulin concentrations in response to intravenous insulin and glucose infusions. Each virtual subject provides a unique response, and its parameters were estimated from combined intravenous glucose tolerance test-hyperinsulinemic-euglycemic clamp data using the Bayesian approach. The virtual subjects were validated by comparing their simulated predictions against data from 12 healthy individuals who underwent a hyperglycemic glucose clamp procedure.

RESULTS

Plasma glucose and insulin concentrations were predicted by the virtual subjects in response to glucose infusions determined by a trained research staff performing a simulated hyperglycemic clamp experiment. The total amount of glucose infusion was indifferent between the simulated and the real subjects (85±18 g vs. 83±23 g; p=NS) as well as plasma insulin levels (63±20 mU/L vs. 58±16 mU/L; p=NS).

CONCLUSIONS

The virtual subjects can reliably predict glucose needs and plasma insulin profiles during hyperglycemic glucose clamp conditions. These virtual subjects can be used to train personnel to make glucose infusion adjustments during clamp experiments.

Reducing risks in type 1 diabetes using H∞ control

A control scheme was designed in order to reduce the risks of hyperglycemia and hypoglycemia in type 1 diabetes mellitus (T1DM). This structure is composed of three main components: an H∞ robust controller, an insulin feedback loop (IFL), and a safety mechanism (SM). A control-relevant model that is employed to design the robust controller is identified. The identification procedure is based on the distribution version of the UVA/Padova metabolic simulator using the simulation adult cohort. The SM prevents dangerous scenarios by acting upon a prediction of future glucose levels, and the IFL modifies the loop gain in order to reduce postprandial hypoglycemia risks. The procedure is tested on the complete alic>in silico adult cohort of the UVA/Padova metabolic simulator, which has been accepted by the Food and Drug Administration (FDA) in lieu of animal trials.

The impact of mathematical modeling on the understanding of diabetes and related complications

Diabetes is a chronic and complex multifactorial disease caused by persistent hyperglycemia and for which underlying pathogenesis is still not completely understood. The mathematical modeling of glucose homeostasis, diabetic condition, and its associated complications is rapidly growing and provides new insights into the underlying mechanisms involved. Here, we discuss contributions to the diabetes modeling field over the past five decades, highlighting the areas where more focused research is required.

A closed-loop artificial pancreas using a proportional integral derivative with double phase lead controller based on a new nonlinear model of glucose metabolism

BACKGROUND

Most closed-loop insulin delivery systems rely on model-based controllers to control the blood glucose (BG) level. Simple models of glucose metabolism, which allow easy design of the control law, are limited in their parametric identification from raw data. New control models and controllers issued from them are needed.

METHODS

A proportional integral derivative with double phase lead controller was proposed. Its design was based on a linearization of a new nonlinear control model of the glucose-insulin system in type 1 diabetes mellitus (T1DM) patients validated with the University of Virginia/Padova T1DM metabolic simulator. A 36 h scenario, including six unannounced meals, was tested in nine virtual adults. A previous trial database has been used to compare the performance of our controller with their previous results. The scenario was repeated 25 times for each adult in order to take continuous glucose monitoring noise into account. The primary outcome was the time BG levels were in target (70-180 mg/dl).

RESULTS

Blood glucose values were in the target range for 77% of the time and below 50 mg/dl and above 250 mg/dl for 0.8% and 0.3% of the time, respectively. The low blood glucose index and high blood glucose index were 1.65 and 3.33, respectively.

CONCLUSION

The linear controller presented, based on the linearization of a new easily identifiable nonlinear model, achieves good glucose control with low exposure to hypoglycemia and hyperglycemia.

Prediction of postprandial blood glucose under uncertainty and intra-patient variability in type 1 diabetes: a comparative study of three interval models

The behavior of three insulin action and glucose kinetics models was assessed for an insulin therapy regime in the presence of patient variability. For this purpose, postprandial glucose in patients with type 1 diabetes was predicted by considering intra- and inter-patient variability using modal interval analysis. Equations to achieve optimal prediction are presented for models 1, 2 and 3, which are of increasing complexity. The model parameters were adjusted to reflect the "same" patient in the presence of variability. The glucose response envelope for model 1, the simplest insulin-glucose model assessed, included the responses of the other two models when a good fit of the model parameters was achieved. Thus, under variability, simple glucose-insulin models may be sufficient to describe patient dynamics in most situations.

Evaluating glycemic control algorithms by computer simulations

BACKGROUND

Numerous guidelines and algorithms exist to achieve glycemic control. Their strengths and weaknesses are difficult to assess without head-to-head comparison in time-consuming clinical trials. We hypothesized that computer simulations may be useful.

METHODS

Two open-label randomized clinical trials were replicated using computer simulations. One study compared performance of the enhanced model predictive control (eMPC) algorithm at two intensive care units in the United Kingdom and Belgium. The other study compared three glucose control algorithms-eMPC, Matias (the absolute glucose protocol), and Bath (the relative glucose change protocol)-in a single intensive care unit. Computer simulations utilized a virtual population of 56 critically ill subjects derived from routine data collected at four European surgical and medical intensive care units.

RESULTS

In agreement with the first clinical study, computer simulations reproduced the main finding and discriminated between the two intensive care units in terms of the sampling interval (1.3 h vs. 1.8 h, United Kingdom vs. Belgium; P < 0.01). Other glucose control metrics were comparable between simulations and clinical results. The principal outcome of the second study was also reproduced. The eMPC demonstrated better performance compared with the Matias and Bath algorithms as assessed by the time when plasma glucose was in the target range between 4.4 and 6.1 mmol/L (65% vs. 43% vs. 42% [P < 0.001], eMPC vs. Matias vs. Bath) without increasing the risk of severe hypoglycemia.

CONCLUSIONS

Computer simulations may provide resource-efficient means for preclinical evaluation of algorithms for glycemic control in the critically ill.

Closed-loop insulin delivery: from bench to clinical practice

Automated closed-loop insulin delivery, also referred to as the 'artificial pancreas', has been an important but elusive goal of diabetes treatment for many decades. Research milestones include the conception of continuous glucose monitoring in the early 1960s, followed by the production of the first commercial hospital-based artificial pancreas in the late 1970s that combined intravenous glucose sensing and insulin delivery. In the past 10 years, research into the artificial pancreas has gained substantial momentum and focused on the subcutaneous route for glucose measurement and insulin delivery, which reflects technological advances in interstitial glucose monitoring and the increasing use of the continuous subcutaneous insulin infusion. This Review discusses the design of an artificial pancreas, its components and clinical results, as well as the advantages and disadvantages of different types of automated closed-loop systems and potential future advances. The introduction of the artificial pancreas into clinical practice will probably occur gradually, starting with simpler approaches, such as overnight control of blood glucose concentration and temporary pump shut-off, that are adapted to more complex situations, such as glycemic control during meals and exercise.