Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine

A design specification for Critical Illness Digital Twins (CIDTs) to cure sepsis: responding to the National Academies of Sciences, Engineering and Medicine Report "Foundational Research Gaps and Future Directions for Digital Twins"

On December 15, 2023, The National Academies of Sciences, Engineering and Medicine (NASEM) released a report entitled: "Foundational Research Gaps and Future Directions for Digital Twins." The ostensible purpose of this report was to bring some structure to the burgeoning field of digital twins by providing a working definition and a series of research challenges that need to be addressed to allow this technology to fulfill its full potential. In the work presented herein we focus on five specific findings from the NASEM Report: 1) definition of a Digital Twin, 2) using "fit-for-purpose" guidance, 3) developing novel approaches to Verification, Validation and Uncertainty Quantification (VVUQ) of Digital Twins, 4) incorporating control as an explicit purpose for a Digital Twin and 5) using a Digital Twin to guide data collection and sensor development, and describe how these findings are addressed through the design specifications for a Critical Illness Digital Twin (CIDT) aimed at curing sepsis.

Reinforcement learning informs optimal treatment strategies to limit antibiotic resistance

Antimicrobial resistance was estimated to be associated with 4.95 million deaths worldwide in 2019. It is possible to frame the antimicrobial resistance problem as a feedback-control problem. If we could optimize this feedback-control problem and translate our findings to the clinic, we could slow, prevent, or reverse the development of high-level drug resistance. Prior work on this topic has relied on systems where the exact dynamics and parameters were known a priori. In this study, we extend this work using a reinforcement learning (RL) approach capable of learning effective drug cycling policies in a system defined by empirically measured fitness landscapes. Crucially, we show that it is possible to learn effective drug cycling policies despite the problems of noisy, limited, or delayed measurement. Given access to a panel of 15 [Formula: see text]-lactam antibiotics with which to treat the simulated Escherichia coli population, we demonstrate that RL agents outperform two naive treatment paradigms at minimizing the population fitness over time. We also show that RL agents approach the performance of the optimal drug cycling policy. Even when stochastic noise is introduced to the measurements of population fitness, we show that RL agents are capable of maintaining evolving populations at lower growth rates compared to controls. We further tested our approach in arbitrary fitness landscapes of up to 1,024 genotypes. We show that minimization of population fitness using drug cycles is not limited by increasing genome size. Our work represents a proof-of-concept for using AI to control complex evolutionary processes.

Current Status and Future Directions: The Application of Artificial Intelligence/Machine Learning for Precision Medicine

Technological innovations, such as artificial intelligence (AI) and machine learning (ML), have the potential to expedite the goal of precision medicine, especially when combined with increased capacity for voluminous data from multiple sources and expanded therapeutic modalities; however, they also present several challenges. In this communication, we first discuss the goals of precision medicine, and contextualize the use of AI in precision medicine by showcasing innovative applications (e.g., prediction of tumor growth and overall survival, biomarker identification using biomedical images, and identification of patient population for clinical practice) which were presented during the February 2023 virtual public workshop entitled "Application of Artificial Intelligence and Machine Learning for Precision Medicine," hosted by the US Food and Drug Administration (FDA) and University of Maryland Center of Excellence in Regulatory Science and Innovation (M-CERSI). Next, we put forward challenges brought about by the multidisciplinary nature of AI, particularly highlighting the need for AI to be trustworthy. To address such challenges, we subsequently note practical approaches, viz., differential privacy, synthetic data generation, and federated learning. The proposed strategies - some of which are highlighted presentations from the workshop - are for the protection of personal information and intellectual property. In addition, methods such as the risk-based management approach and the need for an agile regulatory ecosystem are discussed. Finally, we lay out a call for action that includes sharing of data and algorithms, development of regulatory guidance documents, and pooling of expertise from a broad-spectrum of stakeholders to enhance the application of AI in precision medicine.

Deep reinforcement learning identifies personalized intermittent androgen deprivation therapy for prostate cancer

The evolution of drug resistance leads to treatment failure and tumor progression. Intermittent androgen deprivation therapy (IADT) helps responsive cancer cells compete with resistant cancer cells in intratumoral competition. However, conventional IADT is population-based, ignoring the heterogeneity of patients and cancer. Additionally, existing IADT relies on pre-determined thresholds of prostate-specific antigen to pause and resume treatment, which is not optimized for individual patients. To address these challenges, we framed a data-driven method in two steps. First, we developed a time-varied, mixed-effect and generative Lotka-Volterra (tM-GLV) model to account for the heterogeneity of the evolution mechanism and the pharmacokinetics of two ADT drugs Cyproterone acetate and Leuprolide acetate for individual patients. Then, we proposed a reinforcement-learning-enabled individualized IADT framework, namely, I$^{2}$ADT, to learn the patient-specific tumor dynamics and derive the optimal drug administration policy. Experiments with clinical trial data demonstrated that the proposed I$^{2}$ADT can significantly prolong the time to progression of prostate cancer patients with reduced cumulative drug dosage. We further validated the efficacy of the proposed methods with a recent pilot clinical trial data. Moreover, the adaptability of I$^{2}$ADT makes it a promising tool for other cancers with the availability of clinical data, where treatment regimens might need to be individualized based on patient characteristics and disease dynamics. Our research elucidates the application of deep reinforcement learning to identify personalized adaptive cancer therapy.

Reinforcement Learning informs optimal treatment strategies to limit antibiotic resistance

Antimicrobial resistance was estimated to be associated with 4.95 million deaths worldwide in 2019. It is possible to frame the antimicrobial resistance problem as a feedback-control problem. If we could optimize this feedback-control problem and translate our findings to the clinic, we could slow, prevent or reverse the development of high-level drug resistance. Prior work on this topic has relied on systems where the exact dynamics and parameters were known a priori. In this study, we extend this work using a reinforcement learning (RL) approach capable of learning effective drug cycling policies in a system defined by empirically measured fitness landscapes. Crucially, we show that is possible to learn effective drug cycling policies despite the problems of noisy, limited, or delayed measurement. Given access to a panel of 15 β-lactam antibiotics with which to treat the simulated E. coli population, we demonstrate that RL agents outperform two naive treatment paradigms at minimizing the population fitness over time. We also show that RL agents approach the performance of the optimal drug cycling policy. Even when stochastic noise is introduced to the measurements of population fitness, we show that RL agents are capable of maintaining evolving populations at lower growth rates compared to controls. We further tested our approach in arbitrary fitness landscapes of up to 1024 genotypes. We show that minimization of population fitness using drug cycles is not limited by increasing genome size. Our work represents a proof-of-concept for using AI to control complex evolutionary processes.

Characterizing the Crosstalk Between Programmed Cell Death Pathways in Cytokine Storm With an Agent-Based Model

Background: There is increasing recognition of extensive crosstalk between programmed cell death pathways (PCDPs), such as apoptosis, pyroptosis, and necroptosis, resulting in a highly redundant system responsive to a breadth of potential pathogens. However, because pyroptosis and necroptosis propagate inflammation, these redundancies also present challenges for therapeutic control of dysregulated hyperinflammation seen in cytokine storm (CS) generated organ dysfunction. Hypothesis: We hypothesize that the conversion of existing knowledge regarding apoptosis, pyroptosis, and necroptosis into a computational model can enhance our understanding of the crosstalk between PCDPs via simulation experiments of microbe interactions and experimental interventions. Materials and Methods: Literature regarding apoptosis, pyroptosis, and necroptosis was reviewed and transposed into an agent-based model, the programmed cell death agent-based model (PCDABM). Computational experiments were performed to simulate the activation of various PCDPs as seen by differing microbes, specifically: influenza A virus (IAV), enteropathic Escherichia coli (EPEC), and Salmonella enterica (SE). The potential protective value of PCDP crosstalk was evaluated by silencing either pyroptosis, necroptosis, or both. Computational experiments were also performed simulating the effect of potential therapies blocking tumor necrosis factor (TNF) and interleukin (IL)-1. Results: The PCDABM was implemented in the agent-based modeling toolkit NetLogo. Computational experiments of infection with IAV, EPEC, and SE reproduced cross-activation of PCDPs with effective microbial clearance. Simulations of anti-TNF and anti-IL-1 did not reduce the aggregated amount of inflammation-generated system damage, the surrogate for CS-generated tissue damage. Conclusions: Redundancies have evolved in host PCDPs to maintain protection against a wide range of pathogens. However, these redundancies also challenge attempts at dampening the pathogenic hyperinflammatory state of CS using therapeutic immunomodulation. Integrative simulation models such as the PCDABM can aid in identifying potentially targetable inflection points to mitigate CS while maintaining effective host defense.

Embracing complexity in sepsis

Sepsis involves the dynamic interplay between a pathogen, the host response, the failure of organ systems, medical interventions and a myriad of other factors. This together results in a complex, dynamic and dysregulated state that has remained ungovernable thus far. While it is generally accepted that sepsis is very complex indeed, the concepts, approaches and methods that are necessary to understand this complexity remain underappreciated. In this perspective we view sepsis through the lens of complexity theory. We describe the concepts that support viewing sepsis as a state of a highly complex, non-linear and spatio-dynamic system. We argue that methods from the field of complex systems are pivotal for a fuller understanding of sepsis, and we highlight the progress that has been made over the last decades in this respect. Still, despite these considerable advancements, methods like computational modelling and network-based analyses continue to fly under the general scientific radar. We discuss what barriers contribute to this disconnect, and what we can do to embrace complexity with regards to measurements, research approaches and clinical applications. Specifically, we advocate a focus on longitudinal, more continuous biological data collection in sepsis. Understanding the complexity of sepsis will require a huge multidisciplinary effort, in which computational approaches derived from complex systems science must be supported by, and integrated with, biological data. Such integration could finetune computational models, guide validation experiments, and identify key pathways that could be targeted to modulate the system to the benefit of the host. We offer an example for immunological predictive modelling, which may inform agile trials that could be adjusted throughout the trajectory of disease. Overall, we argue that we should expand our current mental frameworks of sepsis, and embrace nonlinear, system-based thinking in order to move the field forward.

Preparing for the next pandemic: Simulation-based deep reinforcement learning to discover and test multimodal control of systemic inflammation using repurposed immunomodulatory agents

Background

Preparation to address the critical gap in a future pandemic between non-pharmacological measures and the deployment of new drugs/vaccines requires addressing two factors: 1) finding virus/pathogen-agnostic pathophysiological targets to mitigate disease severity and 2) finding a more rational approach to repurposing existing drugs. It is increasingly recognized that acute viral disease severity is heavily driven by the immune response to the infection ("cytokine storm" or "cytokine release syndrome"). There exist numerous clinically available biologics that suppress various pro-inflammatory cytokines/mediators, but it is extremely difficult to identify clinically effective treatment regimens with these agents. We propose that this is a complex control problem that resists standard methods of developing treatment regimens and accomplishing this goal requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training successful game-playing artificial intelligences (AIs). This proof-of-concept study determines if simulated sepsis (e.g. infection-driven cytokine storm) can be controlled in the absence of effective antimicrobial agents by targeting cytokines for which FDA-approved biologics currently exist.

Methods

We use a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. DRL training used a Deep Deterministic Policy Gradient (DDPG) approach with a clinically plausible control interval of 6 hours with manipulation of six cytokines for which there are existing drugs: Tumor Necrosis Factor (TNF), Interleukin-1 (IL-1), Interleukin-4 (IL-4), Interleukin-8 (IL-8), Interleukin-12 (IL-12) and Interferon-γ(IFNg).

Results

DRL trained an AI policy that could improve outcomes from a baseline Recovered Rate of 61% to one with a Recovered Rate of 90% over ~21 days simulated time. This DRL policy was then tested on four different parameterizations not seen in training representing a range of host and microbe characteristics, demonstrating a range of improvement in Recovered Rate by +33% to +56.

Discussion

The current proof-of-concept study demonstrates that significant disease severity mitigation can potentially be accomplished with existing anti-mediator drugs, but only through a multi-modal, adaptive treatment policy requiring implementation with an AI. While the actual clinical implementation of this approach is a projection for the future, the current goal of this work is to inspire the development of a research ecosystem that marries what is needed to improve the simulation models with the development of the sensing/assay technologies to collect the data needed to iteratively refine those models.

Artificial intelligence and its impact on the domains of universal health coverage, health emergencies and health promotion: An overview of systematic reviews

•

An overview of systematic reviews on the application of AI including 129 studies.

•

AI use is prominent in Universal Health Coverage, featuring image analysis in neoplasms.

•

Half of the reviews did not evaluate validation procedures nor reporting guidelines.

•

Risk of bias was only included un a third of the reviews.

•

There is not sufficient evidence to transfer AI to actual healthcare delivery.

Background

Artificial intelligence is fueling a new revolution in medicine and in the healthcare sector. Despite the growing evidence on the benefits of artificial intelligence there are several aspects that limit the measure of its impact in people’s health. It is necessary to assess the current status on the application of AI towards the improvement of people’s health in the domains defined by WHO’s Thirteenth General Programme of Work (GPW13) and the European Programme of Work (EPW), to inform about trends, gaps, opportunities, and challenges.

Objective

To perform a systematic overview of systematic reviews on the application of artificial intelligence in the people’s health domains as defined in the GPW13 and provide a comprehensive and updated map on the application specialties of artificial intelligence in terms of methodologies, algorithms, data sources, outcomes, predictors, performance, and methodological quality.

Methods

A systematic search in MEDLINE, EMBASE, Cochrane and IEEEXplore was conducted between January 2015 and June 2021 to collect systematic reviews using a combination of keywords related to the domains of universal health coverage, health emergencies protection, and better health and wellbeing as defined by the WHO’s PGW13 and EPW. Eligibility criteria was based on methodological quality and the inclusion of practical implementation of artificial intelligence. Records were classified and labeled using ICD-11 categories into the domains of the GPW13. Descriptors related to the area of implementation, type of modeling, data entities, outcomes and implementation on care delivery were extracted using a structured form and methodological aspects of the included reviews studies was assessed using the AMSTAR checklist.

Results

The search strategy resulted in the screening of 815 systematic reviews from which 203 were assessed for eligibility and 129 were included in the review. The most predominant domain for artificial intelligence applications was Universal Health Coverage (N = 98) followed by Health Emergencies (N = 16) and Better Health and Wellbeing (N = 15). Neoplasms area on Universal Health Coverage was the disease area featuring most of the applications (21.7 %, N = 28). The reviews featured analytics primarily over both public and private data sources (67.44 %, N = 87). The most used type of data was medical imaging (31.8 %, N = 41) and predictors based on regions of interest and clinical data. The most prominent subdomain of Artificial Intelligence was Machine Learning (43.4 %, N = 56), in which Support Vector Machine method was predominant (20.9 %, N = 27). Regarding the purpose, the application of Artificial Intelligence I is focused on the prediction of the diseases (36.4 %, N = 47). With respect to the validation, more than a half of the reviews (54.3 %, N = 70) did not report a validation procedure and, whenever available, the main performance indicator was the accuracy (28.7 %, N = 37). According to the methodological quality assessment, a third of the reviews (34.9 %, N = 45) implemented methods for analysis the risk of bias and the overall AMSTAR score below was 5 (4.01 ± 1.93) on all the included systematic reviews.

Conclusion

Artificial intelligence is being used for disease modelling, diagnose, classification and prediction in the three domains of GPW13. However, the evidence is often limited to laboratory and the level of adoption is largely unbalanced between ICD-11 categoriesand diseases. Data availability is a determinant factor on the developmental stage of artificial intelligence applications. Most of the reviewed studies show a poor methodological quality and are at high risk of bias, which limits the reproducibility of the results and the reliability of translating these applications to real clinical scenarios. The analyzed papers show results only in laboratory and testing scenarios and not in clinical trials nor case studies, limiting the supporting evidence to transfer artificial intelligence to actual care delivery.

Drug Development Digital Twins for Drug Discovery, Testing and Repurposing: A Schema for Requirements and Development

There has been a great deal of interest in the concept, development and implementation of medical digital twins. This interest has led to wide ranging perceptions of what constitutes a medical digital twin. This Perspectives article will provide 1) a description of fundamental features of industrial digital twins, the source of the digital twin concept, 2) aspects of biology that challenge the implementation of medical digital twins, 3) a schematic program of how a specific medical digital twin project could be defined, and 4) an example description within that schematic program for a specific type of medical digital twin intended for drug discovery, testing and repurposing, the Drug Development Digital Twin (DDDT).

Robotic laser position versus freehand in CT-guided percutaneous microwave ablation for single hepatocellular carcinoma (diameter < 3 cm): a preliminary study

PURPOSE

To compare the accuracy and safety of robotic laser position (RLP) versus freehand for antenna CT-guided microwave ablation (MWA) of single hepatocellular carcinoma (HCC) (diameter < 3 cm).

MATERIALS AND METHODS

This retrospective study was conducted between May 2020 and June 2021. A total of 40 patients with early HCC who underwent CT-guided MWA were divided into two groups: a freehand group (n = 20) and a RLP group (n = 20). Based on in-plane and out-of-plane data, the actual puncture point error (APPE), number of repositioning procedures, and operative duration were compared using the Mann-Whitney U test. Ablation-related complications were compared using the Chi-squared test.

RESULTS

The mean diameter of HCC patients who received MWA was 2.4 ± 0.5 cm. For in-plane APPE, APPE was comparable between the two groups (p = 0.299). However, for the out-of-plane position, the APPE in the freehand group was higher than that in the RLP group (p = 0.027). The number of repositioning procedures was 0 (range, 0-0) for RLP-guided procedures and 3 (range, 2-5) for freehand procedures, showing a statistically significant difference between the two groups (p < 0.001). The mean operative duration for freehand procedures was 39 min, compared with 26 min for RLP-guided procedures, showing a significant difference (p = 0.013). No deaths or major complications were directly related to MWA. Minor complications in the freehand group were comparable with those in the RLP group (p = 0.313).

CONCLUSION

RLP guidance significantly reduces the number of antenna repositioning procedures in MWA and improves puncture accuracy for target HCC out-of-plane. In addition, the operative duration of robotic guidance was shorter than that of freehand guidance.

Supporting Computational Apprenticeship Through Educational and Software Infrastructure: A Case Study in a Mathematical Oncology Research Lab

There is growing awareness of the need for mathematics and computing to quantitatively understand the complex dynamics and feedbacks in the life sciences. Although several institutions and research groups are conducting pioneering multidisciplinary research, communication and education across fields remain a bottleneck. The opportunity is ripe for using education research-supported mechanisms of cross-disciplinary training at the intersection of mathematics, computation, and biology. This case study uses the computational apprenticeship theoretical framework to describe the efforts of a computational biology lab to rapidly prototype, test, and refine a mentorship infrastructure for undergraduate research experiences. We describe the challenges, benefits, and lessons learned, as well as the utility of the computational apprenticeship framework in supporting computational/math students learning and contributing to biology, and biologists in learning computational methods. We also explore implications for undergraduate classroom instruction and cross-disciplinary scientific communication.

High-dimensional role of AI and machine learning in cancer research

The role of Artificial Intelligence and Machine Learning in cancer research offers several advantages, primarily scaling up the information processing and increasing the accuracy of the clinical decision-making. The key enabling tools currently in use in Precision, Digital and Translational Medicine, here named as 'Intelligent Systems' (IS), leverage unprecedented data volumes and aim to model their underlying heterogeneous influences and variables correlated with patients' outcomes. As functionality and performance of IS are associated with complex diagnosis and therapy decisions, a rich spectrum of patterns and features detected in high-dimensional data may be critical for inference purposes. Many challenges are also present in such discovery task. First, the generation of interpretable model results from a mix of structured and unstructured input information. Second, the design, and implementation of automated clinical decision processes for drawing disease trajectories and patient profiles. Ultimately, the clinical impacts depend on the data effectively subjected to steps such as harmonisation, integration, validation, etc. The aim of this work is to discuss the transformative value of IS applied to multimodal data acquired through various interrelated cancer domains (high-throughput genomics, experimental biology, medical image processing, radiomics, patient electronic records, etc.).

Preparing for the next COVID: Deep Reinforcement Learning trained Artificial Intelligence discovery of multi-modal immunomodulatory control of systemic inflammation in the absence of effective anti-microbials

Background

Despite a great deal of interest in the application of artificial intelligence (AI) to sepsis/critical illness, most current approaches are limited in their potential impact: prediction models do not (and cannot) address the lack of effective therapeutics and current approaches to enhancing the treatment of sepsis focus on optimizing the application of existing interventions, and thus cannot address the development of new treatment options/modalities. The inability to test new therapeutic applications was highlighted by the generally unsatisfactory results from drug repurposing efforts in COVID-19.

Hypothesis

Addressing this challenge requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training the game-playing AIs. We have previously demonstrated the potential of this method in the context of bacterial sepsis in which the microbial infection is responsive to antibiotic therapy. The current work addresses the control problem of multi-modal, adaptive immunomodulation in the circumstance where there is no effective anti-pathogen therapy (e.g., in a novel viral pandemic or in the face of resistant microbes).

Methods

This is a proof-of-concept study that determines the controllability of sepsis without the ability to pharmacologically suppress the pathogen. We use as a surrogate system a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. The DRL algorithm 'trains' an AI on simulations of infection where both the control and observation spaces are limited to operating upon the defined immune mediators included in the IIRABM (a total of 11). Policies were learned using the Deep Deterministic Policy Gradient approach, with the objective function being a return to baseline system health.

Results

DRL trained an AI policy that improved system mortality from 85% to 10.4%. Control actions affected every one of the 11 targetable cytokines and could be divided into those with static/unchanging controls and those with variable/adaptive controls. Adaptive controls primarily targeted 3 different aspects of the immune response: 2nd order pro-inflammation governing TH1/TH2 balance, primary anti-inflammation, and inflammatory cell proliferation.

Discussion

The current treatment of sepsis is hampered by limitations in therapeutic options able to affect the biology of sepsis. This is heightened in circumstances where no effective antimicrobials exist, as was the case for COVID-19. Current AI methods are intrinsically unable to address this problem; doing so requires training AIs in contexts that fully represent the counterfactual space of potential treatments. The synthetic data needed for this task is only possible through the use of high-resolution, mechanism-based simulations. Finally, being able to treat sepsis will require a reorientation as to the sensing and actuating requirements needed to develop these simulations and bring them to the bedside.

An Introduction to Machine Learning Approaches for Biomedical Research

Machine learning (ML) approaches are a collection of algorithms that attempt to extract patterns from data and to associate such patterns with discrete classes of samples in the data—e.g., given a series of features describing persons, a ML model predicts whether a person is diseased or healthy, or given features of animals, it predicts weather an animal is treated or control, or whether molecules have the potential to interact or not, etc. ML approaches can also find such patterns in an agnostic manner, i.e., without having information about the classes. Respectively, those methods are referred to as supervised and unsupervised ML. A third type of ML is reinforcement learning, which attempts to find a sequence of actions that contribute to achieving a specific goal. All of these methods are becoming increasingly popular in biomedical research in quite diverse areas including drug design, stratification of patients, medical images analysis, molecular interactions, prediction of therapy outcomes and many more. We describe several supervised and unsupervised ML techniques, and illustrate a series of prototypical examples using state-of-the-art computational approaches. Given the complexity of reinforcement learning, it is not discussed in detail here, instead, interested readers are referred to excellent reviews on that topic. We focus on concepts rather than procedures, as our goal is to attract the attention of researchers in biomedicine toward the plethora of powerful ML methods and their potential to leverage basic and applied research programs.

Iterative community-driven development of a SARS-CoV-2 tissue simulator

The 2019 novel coronavirus, SARS-CoV-2, is a pathogen of critical significance to international public health. Knowledge of the interplay between molecular-scale virus-receptor interactions, single-cell viral replication, intracellular-scale viral transport, and emergent tissue-scale viral propagation is limited. Moreover, little is known about immune system-virus-tissue interactions and how these can result in low-level (asymptomatic) infections in some cases and acute respiratory distress syndrome (ARDS) in others, particularly with respect to presentation in different age groups or pre-existing inflammatory risk factors. Given the nonlinear interactions within and among each of these processes, multiscale simulation models can shed light on the emergent dynamics that lead to divergent outcomes, identify actionable "choke points" for pharmacologic interventions, screen potential therapies, and identify potential biomarkers that differentiate patient outcomes. Given the complexity of the problem and the acute need for an actionable model to guide therapy discovery and optimization, we introduce and iteratively refine a prototype of a multiscale model of SARS-CoV-2 dynamics in lung tissue. The first prototype model was built and shared internationally as open source code and an online interactive model in under 12 hours, and community domain expertise is driving regular refinements. In a sustained community effort, this consortium is integrating data and expertise across virology, immunology, mathematical biology, quantitative systems physiology, cloud and high performance computing, and other domains to accelerate our response to this critical threat to international health. More broadly, this effort is creating a reusable, modular framework for studying viral replication and immune response in tissues, which can also potentially be adapted to related problems in immunology and immunotherapy.

The Use of Artificial Neural Networks to Forecast the Behavior of Agent-Based Models of Pathophysiology: An Example Utilizing an Agent-Based Model of Sepsis

Introduction: Disease states are being characterized at finer and finer levels of resolution via biomarker or gene expression profiles, while at the same time. Machine learning (ML) is increasingly used to analyze and potentially classify or predict the behavior of biological systems based on such characterization. As ML applications are extremely data-intensive, given the relative sparsity of biomedical data sets ML training of artificial neural networks (ANNs) often require the use of synthetic training data. Agent-based models (ABMs) that incorporate known biological mechanisms and their associated stochastic properties are a potential means of generating synthetic data. Herein we present an example of ML used to train an artificial neural network (ANN) as a surrogate system used to predict the time evolution of an ABM focusing on the clinical condition of sepsis. Methods: The disease trajectories for clinical sepsis, in terms of temporal cytokine and phenotypic dynamics, can be interpreted as a random dynamical system. The Innate Immune Response Agent-based Model (IIRABM) is a well-established model that utilizes known cellular and molecular rules to simulate disease trajectories corresponding to clinical sepsis. We have utilized two distinct neural network architectures, Long Short-Term Memory and Multi-Layer Perceptron, to take a time sequence of five measurements of eleven IIRABM simulated serum cytokine concentrations as input and to return both the future cytokine trajectories as well as an aggregate metric representing the patient's state of health. Results: The ANNs predicted model trajectories with the expected amount of error, due to stochasticity in the simulation, and recognizing that the mapping from a specific cytokine profile to a state-of-health is not unique. The Multi-Layer Perceptron neural network, generated predictions with a more accurate forecasted trajectory cone. Discussion: This work serves as a proof-of-concept for the use of ANNs to predict disease progression in sepsis as represented by an ABM. The findings demonstrate that multicellular systems with intrinsic stochasticity can be approximated with an ANN, but that forecasting a specific trajectory of the system requires sequential updating of the system state to provide a rolling forecast horizon.

Comparative Computational Modeling of the Bat and Human Immune Response to Viral Infection with the Comparative Biology Immune Agent Based Model

Given the impact of pandemics due to viruses of bat origin, there is increasing interest in comparative investigation into the differences between bat and human immune responses. The practice of comparative biology can be enhanced by computational methods used for dynamic knowledge representation to visualize and interrogate the putative differences between the two systems. We present an agent based model that encompasses and bridges differences between bat and human responses to viral infection: the comparative biology immune agent based model, or CBIABM. The CBIABM examines differences in innate immune mechanisms between bats and humans, specifically regarding inflammasome activity and type 1 interferon dynamics, in terms of tolerance to viral infection. Simulation experiments with the CBIABM demonstrate the efficacy of bat-related features in conferring viral tolerance and also suggest a crucial role for endothelial inflammasome activity as a mechanism for bat systemic viral tolerance and affecting the severity of disease in human viral infections. We hope that this initial study will inspire additional comparative modeling projects to link, compare, and contrast immunological functions shared across different species, and in so doing, provide insight and aid in preparation for future viral pandemics of zoonotic origin.

Machine Learning and Artificial Intelligence for Surgical Decision Making

Background: The use of machine learning (ML) and artificial intelligence (AI) in medical research continues to grow as the amount and availability of clinical data expands. These techniques allow complex interpretation of data and capture non-linear relations not immediately apparent by classic statistical techniques. Methods: This review of the ML/AI literature provides a brief overview for practicing surgeons and clinicians of the current and future roles these methods will have within surgical infection research. Results: A conceptual overview of the techniques is provided along with concrete examples in the surgical infections literature. Further examples of ML/AI techniques in clinical decision support as well as therapy discovery with model-based deep reinforcement learning are illustrated. Conclusions: Artificial intelligence and ML are important and increasingly utilized techniques within the expanding body of surgical infection research. This article provides a minimal baseline literacy in ML/AI to be able to view such projects in an appropriately critical fashion.

Utilizing the Heterogeneity of Clinical Data for Model Refinement and Rule Discovery Through the Application of Genetic Algorithms to Calibrate a High-Dimensional Agent-Based Model of Systemic Inflammation

Introduction: Accounting for biological heterogeneity represents one of the greatest challenges in biomedical research. Dynamic computational and mathematical models can be used to enhance the study and understanding of biological systems, but traditional methods for calibration and validation commonly do not account for the heterogeneity of biological data, which may result in overfitting and brittleness of these models. Herein we propose a machine learning approach that utilizes genetic algorithms (GAs) to calibrate and refine an agent-based model (ABM) of acute systemic inflammation, with a focus on accounting for the heterogeneity seen in a clinical data set, thereby avoiding overfitting and increasing the robustness and potential generalizability of the underlying simulation model. Methods: Agent-based modeling is a frequently used modeling method for multi-scale mechanistic modeling. However, the same properties that make ABMs well suited to representing biological systems also present significant challenges with respect to their construction and calibration, particularly with respect to the selection of potential mechanistic rules and the large number of associated free parameters. We have proposed that machine learning approaches (such as GAs) can be used to more effectively and efficiently deal with rule selection and parameter space characterization; the current work applies GAs to the challenge of calibrating a complex ABM to a specific data set, while preserving biological heterogeneity reflected in the range and variance of the data. This project uses a GA to augment the rule-set for a previously validated ABM of acute systemic inflammation, the Innate Immune Response ABM (IIRABM) to clinical time series data of systemic cytokine levels from a population of burn patients. The genome for the GA is a vector generated from the IIRABM's Model Rule Matrix (MRM), which is a matrix representation of not only the constants/parameters associated with the IIRABM's cytokine interaction rules, but also the existence of rules themselves. Capturing heterogeneity is accomplished by a fitness function that incorporates the sample value range ("error bars") of the clinical data. Results: The GA-enabled parameter space exploration resulted in a set of putative MRM rules and associated parameterizations which closely match the cytokine time course data used to design the fitness function. The number of non-zero elements in the MRM increases significantly as the model parameterizations evolve toward a fitness function minimum, transitioning from a sparse to a dense matrix. This results in a model structure that more closely resembles (at a superficial level) the structure of data generated by a standard differential gene expression experimental study. Conclusion: We present an HPC-enabled machine learning/evolutionary computing approach to calibrate a complex ABM to complex clinical data while preserving biological heterogeneity. The integration of machine learning, HPC, and multi-scale mechanistic modeling provides a pathway forward to more effectively representing the heterogeneity of clinical populations and their data.

An Integrated Approach of Mechanistic-Modeling and Machine-Learning for Thickness Optimization of Frozen Microwaveable Foods

Mechanistic-modeling has been a useful tool to help food scientists in understanding complicated microwave-food interactions, but it cannot be directly used by the food developers for food design due to its resource-intensive characteristic. This study developed and validated an integrated approach that coupled mechanistic-modeling and machine-learning to achieve efficient food product design (thickness optimization) with better heating uniformity. The mechanistic-modeling that incorporated electromagnetics and heat transfer was previously developed and validated extensively and was used directly in this study. A Bayesian optimization machine-learning algorithm was developed and integrated with the mechanistic-modeling. The integrated approach was validated by comparing the optimization performance with a parametric sweep approach, which is solely based on mechanistic-modeling. The results showed that the integrated approach had the capability and robustness to optimize the thickness of different-shape products using different initial training datasets with higher efficiency (45.9% to 62.1% improvement) than the parametric sweep approach. Three rectangular-shape trays with one optimized thickness (1.56 cm) and two non-optimized thicknesses (1.20 and 2.00 cm) were 3-D printed and used in microwave heating experiments, which confirmed the feasibility of the integrated approach in thickness optimization. The integrated approach can be further developed and extended as a platform to efficiently design complicated microwavable foods with multiple-parameter optimization.

Digital Twins for Tissue Culture Techniques—Concepts, Expectations, and State of the Art

Techniques to provide in vitro tissue culture have undergone significant changes during the last decades, and current applications involve interactions of cells and organoids, three-dimensional cell co-cultures, and organ/body-on-chip tools. Efficient computer-aided and mathematical model-based methods are required for efficient and knowledge-driven characterization, optimization, and routine manufacturing of tissue culture systems. As an alternative to purely experimental-driven research, the usage of comprehensive mathematical models as a virtual in silico representation of the tissue culture, namely a digital twin, can be advantageous. Digital twins include the mechanistic of the biological system in the form of diverse mathematical models, which describe the interaction between tissue culture techniques and cell growth, metabolism, and the quality of the tissue. In this review, current concepts, expectations, and the state of the art of digital twins for tissue culture concepts will be highlighted. In general, DT’s can be applied along the full process chain and along the product life cycle. Due to the complexity, the focus of this review will be especially on the design, characterization, and operation of the tissue culture techniques.

A framework to shift basins of attraction of gene regulatory networks through batch reinforcement learning

A major challenge in gene regulatory networks (GRN) of biological systems is to discover when and what interventions should be applied to shift them to healthy phenotypes. A set of gene activity profiles, called basin of attraction (BOA), takes this network to a specific phenotype; therefore, a healthy BOA leads the GRN to a healthy phenotype. However, without the complete observability of the genes, it is not possible to identify whether the current BOA is healthy. In this article we investigate external interventions in GRN with partial observability aiming to bring it to healthy BOAs. We propose a new batch reinforcement learning method (BRL), called mSFQI, to define intervention strategies based on the probabilities of the gene activity profiles being in healthy BOAs, which are calculated from a set of previous observed experiences. BRL uses approximation functions and repeated applications of previous experiences to accelerate learning. Results demonstrate that our proposal can quickly shift a partially observable GRN to healthy BOAs, while reducing the number of interventions. In addition, when observability is poor, mSFQI produces better results when the probabilities for a greater amount of previous observations are available.

Complex Stability and an Irrevertible Transition Reverted by Peptide and Fibroblasts in a Dynamic Model of Innate Immunity

We here apply a control analysis and various types of stability analysis to an in silico model of innate immunity that addresses the management of inflammation by a therapeutic peptide. Motivation is the observation, both in silico and in experiments, that this therapy is not robust. Our modeling results demonstrate how (1) the biological phenomena of acute and chronic modes of inflammation may reflect an inherently complex bistability with an irrevertible flip between the two modes, (2) the chronic mode of the model has stable, sometimes unique, steady states, while its acute-mode steady states are stable but not unique, (3) as witnessed by TNF levels, acute inflammation is controlled by multiple processes, whereas its chronic-mode inflammation is only controlled by TNF synthesis and washout, (4) only when the antigen load is close to the acute mode's flipping point, many processes impact very strongly on cells and cytokines, (5) there is no antigen exposure level below which reduction of the antigen load alone initiates a flip back to the acute mode, and (6) adding healthy fibroblasts makes the transition from acute to chronic inflammation revertible, although (7) there is a window of antigen load where such a therapy cannot be effective. This suggests that triple therapies may be essential to overcome chronic inflammation. These may comprise (1) anti-immunoglobulin light chain peptides, (2) a temporarily reduced antigen load, and (3a) fibroblast repopulation or (3b) stem cell strategies.

Agent-based models of inflammation in translational systems biology: A decade later

Agent-based modeling is a rule-based, discrete-event, and spatially explicit computational modeling method that employs computational objects that instantiate the rules and interactions among the individual components ("agents") of system. Agent-based modeling is well suited to translating into a computational model the knowledge generated from basic science research, particularly with respect to translating across scales the mechanisms of cellular behavior into aggregated cell population dynamics manifesting at the tissue and organ level. This capacity has made agent-based modeling an integral method in translational systems biology (TSB), an approach that uses multiscale dynamic computational modeling to explicitly represent disease processes in a clinically relevant fashion. The initial work in the early 2000s using agent-based models (ABMs) in TSB focused on examining acute inflammation and its intersection with wound healing; the decade since has seen vast growth in both the application of agent-based modeling to a wide array of disease processes as well as methodological advancements in the use and analysis of ABM. This report presents an update on an earlier review of ABMs in TSB and presents examples of exciting progress in the modeling of various organs and diseases that involve inflammation. This review also describes developments that integrate the use of ABMs with cutting-edge technologies such as high-performance computing, machine learning, and artificial intelligence, with a view toward the future integration of these methodologies. This article is categorized under: Translational, Genomic, and Systems Medicine > Translational Medicine Models of Systems Properties and Processes > Mechanistic Models Models of Systems Properties and Processes > Organ, Tissue, and Physiological Models Models of Systems Properties and Processes > Organismal Models.