An introduction to stochastic control theory, path integrals and reinforcement learning

Steering and controlling evolution - from bioengineering to fighting pathogens

Control interventions steer the evolution of molecules, viruses, microorganisms or other cells towards a desired outcome. Applications range from engineering biomolecules and synthetic organisms to drug, therapy and vaccine design against pathogens and cancer. In all these instances, a control system alters the eco-evolutionary trajectory of a target system, inducing new functions or suppressing escape evolution. Here, we synthesize the objectives, mechanisms and dynamics of eco-evolutionary control in different biological systems. We discuss how the control system learns and processes information about the target system by sensing or measuring, through adaptive evolution or computational prediction of future trajectories. This information flow distinguishes pre-emptive control strategies by humans from feedback control in biotic systems. We establish a cost-benefit calculus to gauge and optimize control protocols, highlighting the fundamental link between predictability of evolution and efficacy of pre-emptive control.

Machine Learning Methods for Small Data Challenges in Molecular Science

Small data are often used in scientific and engineering research due to the presence of various constraints, such as time, cost, ethics, privacy, security, and technical limitations in data acquisition. However, big data have been the focus for the past decade, small data and their challenges have received little attention, even though they are technically more severe in machine learning (ML) and deep learning (DL) studies. Overall, the small data challenge is often compounded by issues, such as data diversity, imputation, noise, imbalance, and high-dimensionality. Fortunately, the current big data era is characterized by technological breakthroughs in ML, DL, and artificial intelligence (AI), which enable data-driven scientific discovery, and many advanced ML and DL technologies developed for big data have inadvertently provided solutions for small data problems. As a result, significant progress has been made in ML and DL for small data challenges in the past decade. In this review, we summarize and analyze several emerging potential solutions to small data challenges in molecular science, including chemical and biological sciences. We review both basic machine learning algorithms, such as linear regression, logistic regression (LR), k-nearest neighbor (KNN), support vector machine (SVM), kernel learning (KL), random forest (RF), and gradient boosting trees (GBT), and more advanced techniques, including artificial neural network (ANN), convolutional neural network (CNN), U-Net, graph neural network (GNN), Generative Adversarial Network (GAN), long short-term memory (LSTM), autoencoder, transformer, transfer learning, active learning, graph-based semi-supervised learning, combining deep learning with traditional machine learning, and physical model-based data augmentation. We also briefly discuss the latest advances in these methods. Finally, we conclude the survey with a discussion of promising trends in small data challenges in molecular science.

Effects of water currents on fish migration through a Feynman-type path integral approach under [Formula: see text] Liouville-like quantum gravity surfaces

A stochastic differential game theoretic model has been proposed to determine optimal behavior of a fish while migrating against water currents both in rivers and oceans. Then, a dynamic objective function is maximized subject to two stochastic dynamics, one represents its location and another its relative velocity against water currents. In relative velocity stochastic dynamics, a Cucker-Smale type stochastic differential equation is introduced under white noise. As the information regarding hydrodynamic environment is incomplete and imperfect, a Feynman type path integral under [Formula: see text] Liouville-like quantum gravity surface has been introduced to obtain a Wick-rotated Schrödinger type equation to determine an optimal strategy of a fish during its migration. The advantage of having Feynman type path integral is that, it can be used in more generalized nonlinear stochastic differential equations where constructing a Hamiltonian-Jacobi-Bellman (HJB) equation is impossible. The mathematical analytic results show exact expression of an optimal strategy of a fish under imperfect information and uncertainty.

Optimal Position and Attitude Control of Quadcopter Using Stochastic Differential Dynamic Programming with Input Saturation Constraints

This paper considers the position and attitude control of a quadcopter in the presence of stochastic disturbances. Basic quadcopter dynamics is modeled as a nonlinear stochastic system described by a stochastic differential equation. Subsequently, the position and attitude control is formulated as a nonlinear stochastic optimal control problem with input saturation constraints. To solve this problem, a continuous-time stochastic differential dynamic programming (DDP) method with input saturation constraints is newly proposed. Finally, numerical simulations demonstrate the effectiveness of the proposed method by comparing it with the linear quadratic Gaussian and the deterministic DDP with input saturation constraints.

Adaptive Optimal Control for Stochastic Multiplayer Differential Games Using On-Policy and Off-Policy Reinforcement Learning

Control-theoretic differential games have been used to solve optimal control problems in multiplayer systems. Most existing studies on differential games either assume deterministic dynamics or dynamics corrupted with additive noise. In realistic environments, multidimensional environmental uncertainties often modulate system dynamics in a more complicated fashion. In this article, we study stochastic multiplayer differential games, where the players' dynamics are modulated by randomly time-varying parameters. We first formulate two differential games for systems of general uncertain linear dynamics, including the two-player zero-sum and multiplayer nonzero-sum games. We then show that optimal control policies, which constitute the Nash equilibrium solutions, can be derived from the corresponding Hamiltonian functions. Stability is proven using the Lyapunov type of analysis. In order to solve the stochastic differential games online, we integrate reinforcement learning (RL) and an effective uncertainty sampling method called the multivariate probabilistic collocation method (MPCM). Two learning algorithms, including the on-policy integral RL (IRL) and off-policy IRL, are designed for the formulated games, respectively. We show that the proposed learning algorithms can effectively find the Nash equilibrium solutions for the stochastic multiplayer differential games.

A Path-Integral-Based Reinforcement Learning Algorithm for Path Following of an Autoassembly Mobile Robot

Reinforcement learning (RL) combined with deep neural networks has led to a number of great achievements for robot control in virtual computer environments, where sufficient data can be obtained without any difficulty to train various models. However, thus far, only few and relatively simple tasks have been accomplished for practical robots, which is mainly caused by the following two reasons. First, training with real robots, especially with dynamic systems, is too complicated to be fully and accurately represented in simulations. Second, it is very costly to obtain training data from real systems. To address these two problems effectively, in this article, a path-integral-based RL algorithm is proposed for the task of path following of an autoassembly mobile robot, wherein three kernel techniques are introduced. First, a generalized path-integral-control approach is proposed to obtain the numerical solution of a stochastic dynamical system, wherein the calculation of the gradient and kinematics inverse is avoided to ensure fast and reliable training convergence. Second, a novel parameterization method using Lyapunov techniques is introduced into the RL algorithm to ensure good performance of the system when directly transferring simulation results into practical systems. Third, the optimal parameters for all discrete initial states are first learned offline and then tuned online to improve the generalization and real-time performance. In addition to the optimization control for the mobile robot, the proposed method also possesses general applicability for a class of nonlinear systems such as crane systems. Simulation and experimental results are included and analyzed to illustrate the superior performance of the proposed algorithm.

Eco-evolutionary control of pathogens

Vaccinations and therapies targeting evolving pathogens aim to curb the pathogen and to steer it toward a controlled evolutionary state. Control is leveraged against the pathogen’s intrinsic evolutionary forces, which in turn, can drive an escape from control. Here, we analyze a simple model of control, in which a host produces antibodies that bind the pathogen. We show that the leverages of host (or external intervention) and pathogen are often highly imbalanced: an error threshold separates parameter regions of efficient control from regions of compromised control, where the pathogen retains the upper hand. Because control efficiency can be predicted from few measurable fitness parameters, our results establish a proof of principle how control theory can guide interventions against evolving pathogens.

Control can alter the eco-evolutionary dynamics of a target pathogen in two ways, by changing its population size and by directed evolution of new functions. Here, we develop a payoff model of eco-evolutionary control based on strategies of evolution, regulation, and computational forecasting. We apply this model to pathogen control by molecular antibody–antigen binding with a tunable dosage of antibodies. By analytical solution, we obtain optimal dosage protocols and establish a phase diagram with an error threshold delineating parameter regimes of successful and compromised control. The solution identifies few independently measurable fitness parameters that predict the outcome of control. Our analysis shows how optimal control strategies depend on mutation rate and population size of the pathogen, and how monitoring and computational forecasting affect protocols and efficiency of control. We argue that these results carry over to more general systems and are elements of an emerging eco-evolutionary control theory.

Multimodal trajectory optimization for motion planning

Existing motion planning methods often have two drawbacks: (1) goal configurations need to be specified by a user, and (2) only a single solution is generated under a given condition. In practice, multiple possible goal configurations exist to achieve a task. Although the choice of the goal configuration significantly affects the quality of the resulting trajectory, it is not trivial for a user to specify the optimal goal configuration. In addition, the objective function used in the trajectory optimization is often non-convex, and it can have multiple solutions that achieve comparable costs. In this study, we propose a framework that determines multiple trajectories that correspond to the different modes of the cost function. We reduce the problem of identifying the modes of the cost function to that of estimating the density induced by a distribution based on the cost function. The proposed framework enables users to select a preferable solution from multiple candidate trajectories, thereby making it easier to tune the cost function and obtain a satisfactory solution. We evaluated our proposed method with motion planning tasks in 2D and 3D space. Our experiments show that the proposed algorithm is capable of determining multiple solutions for those tasks.

Behavior Stability and Individual Differences in Pavlovian Extended Conditioning

How stable and general is behavior once maximum learning is reached? To answer this question and understand post-acquisition behavior and its related individual differences, we propose a psychological principle that naturally extends associative models of Pavlovian conditioning to a dynamical oscillatory model where subjects have a greater memory capacity than usually postulated, but with greater forecast uncertainty. This results in a greater resistance to learning in the first few sessions followed by an over-optimal response peak and a sequence of progressively damped response oscillations. We detected the first peak and trough of the new learning curve in our data, but their dispersion was too large to also check the presence of oscillations with smaller amplitude. We ran an unusually long experiment with 32 rats over 3,960 trials, where we excluded habituation and other well-known phenomena as sources of variability in the subjects' performance. Using the data of this and another Pavlovian experiment by Harris et al. (2015), as an illustration of the principle we tested the theory against the basic associative single-cue Rescorla–Wagner (RW) model. We found evidence that the RW model is the best non-linear regression to data only for a minority of the subjects, while its dynamical extension can explain the almost totality of data with strong to very strong evidence. Finally, an analysis of short-scale fluctuations of individual responses showed that they are described by random white noise, in contrast with the colored-noise findings in human performance.

Noncooperative dynamics in election interference

Foreign power interference in domestic elections is an existential threat to societies. Manifested through myriad methods from war to words, such interference is a timely example of strategic interaction between economic and political agents. We model this interaction between rational game players as a continuous-time differential game, constructing an analytical model of this competition with a variety of payoff structures. All-or-nothing attitudes by only one player regarding the outcome of the game lead to an arms race in which both countries spend increasing amounts on interference and counterinterference operations. We then confront our model with data pertaining to the Russian interference in the 2016 United States presidential election contest. We introduce and estimate a Bayesian structural time-series model of election polls and social media posts by Russian Twitter troll accounts. Our analytical model, while purposefully abstract and simple, adequately captures many temporal characteristics of the election and social media activity. We close with a discussion of our model's shortcomings and suggestions for future research.

Learning Environmental Field Exploration with Computationally Constrained Underwater Robots: Gaussian Processes Meet Stochastic Optimal Control

Autonomous exploration of environmental fields is one of the most promising tasks to be performed by fleets of mobile underwater robots. The goal is to maximize the information gain during the exploration process by integrating an information-metric into the path-planning and control step. Therefore, the system maintains an internal belief representation of the environmental field which incorporates previously collected measurements from the real field. In contrast to surface robots, mobile underwater systems are forced to run all computations on-board due to the limited communication bandwidth in underwater domains. Thus, reducing the computational cost of field exploration algorithms constitutes a key challenge for in-field implementations on micro underwater robot teams. In this work, we present a computationally efficient exploration algorithm which utilizes field belief models based on Gaussian Processes, such as Gaussian Markov random fields or Kalman regression, to enable field estimation with constant computational cost over time. We extend the belief models by the use of weighted shape functions to directly incorporate spatially continuous field observations. The developed belief models function as information-theoretic value functions to enable path planning through stochastic optimal control with path integrals. We demonstrate the efficiency of our exploration algorithm in a series of simulations including the case of a stationary spatio-temporal field.

Hierarchical Decision and Control for Continuous Multitarget Problem: Policy Evaluation With Action Delay

This paper proposes a hierarchical decision-making and control algorithm for the shepherd game, the seventh mission in the International Aerial Robotics Competition (IARC). In this game, the agent (a multirotor aerial robot) is required to contact targets (ground vehicles) sequentially and drive them to a certain boundary to earn score. During the game of 10 min, the agent should be fully autonomous without any human interference. Regarding the lower-level controller and dynamics of the agent, each action takes a duration of time to accomplish. Denoted as an action delay, in this paper, this action duration is nonconstant and is related to the final reward. Therefore, the challenging point is making the agent "aware of time" when applying a certain action. We solve this problem by two approaches: deep Q-networks and lookup table. The action delay predictor in the decision-level is fitted by a lower-level controller. Through simulations by the example of the shepherd game, the effectiveness and efficiency of this approach are validated. This paper helps our team winning the first prize in IARC 2017, and keeps the best record of this mission since it was released in 2013.

Development of a variational scheme for model inversion of multi-area model of brain. Part I: simulation evaluation

We previously developed an integrated model of the brain within a single cortical area for functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and magnetoencephalography (MEG) using an extended neural mass model (ENMM). We then extended ENMM from a single-area to a multi-area model to develop a neural mass model of the entire brain. To this end, we derived a nonlinear state-space representation of the multi-area model. In Parts I and II of these two companion papers (henceforth called Part I and Part II), we develop and evaluate a variational Bayesian expectation maximization (VBEM) method to estimate parameters of multi-area ENMM (MEN) using E/MEG data. In Part I, we derive a state-space representation of MEN and use VBEM method for model inversion (parameter estimation). We evaluate and validate performance of VBEM method for model inversion of MEN using simulation studies in various signal-to-noise ratios. Details of VBEM method are presented in Part II. The proposed approach provides a useful technique for analyzing effective connectivity using non-invasive EEG and MEG methods.

Model driven EEG/fMRI fusion of brain oscillations

This article reviews progress and challenges in model driven EEG/fMRI fusion with a focus on brain oscillations. Fusion is the combination of both imaging modalities based on a cascade of forward models from ensemble of post-synaptic potentials (ePSP) to net primary current densities (nPCD) to EEG; and from ePSP to vasomotor feed forward signal (VFFSS) to BOLD. In absence of a model, data driven fusion creates maps of correlations between EEG and BOLD or between estimates of nPCD and VFFS. A consistent finding has been that of positive correlations between EEG alpha power and BOLD in both frontal cortices and thalamus and of negative ones for the occipital region. For model driven fusion we formulate a neural mass EEG/fMRI model coupled to a metabolic hemodynamic model. For exploratory simulations we show that the Local Linearization (LL) method for integrating stochastic differential equations is appropriate for highly nonlinear dynamics. It has been successfully applied to small and medium sized networks, reproducing the described EEG/BOLD correlations. A new LL-algebraic method allows simulations with hundreds of thousands of neural populations, with connectivities and conduction delays estimated from diffusion weighted MRI. For parameter and state estimation, Kalman filtering combined with the LL method estimates the innovations or prediction errors. From these the likelihood of models given data are obtained. The LL-innovation estimation method has been already applied to small and medium scale models. With improved Bayesian computations the practical estimation of very large scale EEG/fMRI models shall soon be possible.