Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control

From biological data to oscillator models using SINDy

Periodic changes in the concentration or activity of different molecules regulate vital cellular processes such as cell division and circadian rhythms. Developing mathematical models is essential to better understand the mechanisms underlying these oscillations. Recent data-driven methods like SINDy have fundamentally changed model identification, yet their application to experimental biological data remains limited. This study investigates SINDy's constraints by directly applying it to biological oscillatory data. We identify insufficient resolution, noise, dimensionality, and limited prior knowledge as primary limitations. Using various generic oscillator models of different complexity and/or dimensionality, we systematically analyze these factors. We then propose a comprehensive guide for inferring models from biological data, addressing these challenges step by step. Our approach is validated using glycolytic oscillation data from yeast.

Data-driven modeling and forecasting of chaotic dynamics on inertial manifolds constructed as spectral submanifolds

We present a data-driven and interpretable approach for reducing the dimensionality of chaotic systems using spectral submanifolds (SSMs). Emanating from fixed points or periodic orbits, these SSMs are low-dimensional inertial manifolds containing the chaotic attractor of the underlying high-dimensional system. The reduced dynamics on the SSMs turn out to predict chaotic dynamics accurately over a few Lyapunov times and also reproduce long-term statistical features, such as the largest Lyapunov exponents and probability distributions, of the chaotic attractor. We illustrate this methodology on numerical data sets including delay-embedded Lorenz and Rössler attractors, a nine-dimensional Lorenz model, a periodically forced Duffing oscillator chain, and the Kuramoto-Sivashinsky equation. We also demonstrate the predictive power of our approach by constructing an SSM-reduced model from unforced trajectories of a buckling beam and then predicting its periodically forced chaotic response without using data from the forced beam.

Estimating stable fixed points and Langevin potentials for financial dynamics

The geometric Brownian motion (GBM) is a standard model in quantitative finance, but the potential function of its stochastic differential equation (SDE) cannot include stable nonzero prices. This article generalizes the GBM to an SDE with polynomial drift of order q and shows via model selection that q=2 is most frequently the optimal model to describe the data. Moreover, Markov chain Monte Carlo ensembles of the accompanying potential functions show a clear and pronounced potential well, indicating the existence of a stable price.

Data-driven discovery of sparse dynamical model of cardiovascular system for model predictive control

Cardiovascular diseases remain the leading cause of death globally. In recent years, vagal nerve stimulation (VNS) has shown promising results in the treatment of a number of cardiovascular diseases. In this approach, mild electrical pulses are sent to the brain via the vagus nerve. This open-loop neurostimulation, however, leads to various side effects due to physiological and inter-patient variability and therefore a closed-loop delivery strategy of electrical pulses that accounts for this variability is desired. In this context, we envision data-driven sparse dynamical model parameterized by patient-specific data as appropriate for use in closed loop controller design. In this work, we build a dynamical model for mean arterial pressure and heart rate using the method sparse identification of nonlinear dynamics (SINDy). As a proxy for real datasets or measurements from a patient, we simulate a mechanistic model from the literature and then discover a data-driven model for predicting mean arterial pressure and heart rate in response to neural stimulus. This discovered model is then used to design a controller to be implemented in closed-loop via model predictive control. We observe that this data-driven model is interpretable, consistent with experiments, provides insights on the sensitivity of different stimulation locations and simplifies the formulation of the optimal control problem. Noting the set-point tracking performance of this closed-loop model-based controller that uses this discovered model, we conclude that the model is adequate in capturing the dynamics of a highly nonlinear cardiovascular system for the purpose of optimal predictive controller design.

Statistical Post-Processing Method for Evaluating Bioaccumulation in Fish Due to Dietary Exposure in Japan

In 2018, the dietary exposure bioaccumulation fish test of the Organization for Economic Co-operation and Development Test Guideline No. 305 was introduced into Japan's Chemical Substances Control Law. The Japanese government has adopted a single definitive testing criterion for the absence of high bioaccumulation: the growth-corrected kinetic dietary magnification factor (BMFKg) must be less than 0.007. The aim of this study was to decrease regulatory restrictions in order to increase newly developed chemical substances and their subsequent approval of their manufacture and import, i.e., the present study was motivated by concerns over the criterion being too restrictive, rather than scientific concerns, such as uncertainty in criterion. We used statistical post-processing to assess the possibility of expanding the criteria for not being highly bioaccumulative. Based on our results, we proposed the criterion that the test substance should be considered not highly bioaccumulative if the following two conditions are met: (1) The ratio of the maximum to the minimum measured 5% lipid-standardized biomagnification factor at the end of the uptake phase (BMF5%, n = 5) for the test substance and reference substance should be less than 3.0, and (2) For the measured BMF5% of the test substance (n = 5), the probability that the next (the sixth) BMF5% is below 0.0334 should exceed 95% based on statistical post-processing. It is worth noting that the BMF5% values should only be applied for non-ionizable lipid soluble compounds. Application of our suggested approach to Japan implies that the criterion for chemical substances that are not highly bioaccumulative in the dietary exposure bioaccumulation fish test would be increased from 0.007 to 0.0149.

Direct Estimation of Parameters in ODE Models Using WENDy: Weak-Form Estimation of Nonlinear Dynamics

We introduce the Weak-form Estimation of Nonlinear Dynamics (WENDy) method for estimating model parameters for non-linear systems of ODEs. Without relying on any numerical differential equation solvers, WENDy computes accurate estimates and is robust to large (biologically relevant) levels of measurement noise. For low dimensional systems with modest amounts of data, WENDy is competitive with conventional forward solver-based nonlinear least squares methods in terms of speed and accuracy. For both higher dimensional systems and stiff systems, WENDy is typically both faster (often by orders of magnitude) and more accurate than forward solver-based approaches. The core mathematical idea involves an efficient conversion of the strong form representation of a model to its weak form, and then solving a regression problem to perform parameter inference. The core statistical idea rests on the Errors-In-Variables framework, which necessitates the use of the iteratively reweighted least squares algorithm. Further improvements are obtained by using orthonormal test functions, created from a set of [Formula: see text] bump functions of varying support sizes.We demonstrate the high robustness and computational efficiency by applying WENDy to estimate parameters in some common models from population biology, neuroscience, and biochemistry, including logistic growth, Lotka-Volterra, FitzHugh-Nagumo, Hindmarsh-Rose, and a Protein Transduction Benchmark model. Software and code for reproducing the examples is available at https://github.com/MathBioCU/WENDy .

Distilling identifiable and interpretable dynamic models from biological data

Mechanistic dynamical models allow us to study the behavior of complex biological systems. They can provide an objective and quantitative understanding that would be difficult to achieve through other means. However, the systematic development of these models is a non-trivial exercise and an open problem in computational biology. Currently, many research efforts are focused on model discovery, i.e. automating the development of interpretable models from data. One of the main frameworks is sparse regression, where the sparse identification of nonlinear dynamics (SINDy) algorithm and its variants have enjoyed great success. SINDy-PI is an extension which allows the discovery of rational nonlinear terms, thus enabling the identification of kinetic functions common in biochemical networks, such as Michaelis-Menten. SINDy-PI also pays special attention to the recovery of parsimonious models (Occam's razor). Here we focus on biological models composed of sets of deterministic nonlinear ordinary differential equations. We present a methodology that, combined with SINDy-PI, allows the automatic discovery of structurally identifiable and observable models which are also mechanistically interpretable. The lack of structural identifiability and observability makes it impossible to uniquely infer parameter and state variables, which can compromise the usefulness of a model by distorting its mechanistic significance and hampering its ability to produce biological insights. We illustrate the performance of our method with six case studies. We find that, despite enforcing sparsity, SINDy-PI sometimes yields models that are unidentifiable. In these cases we show how our method transforms their equations in order to obtain a structurally identifiable and observable model which is also interpretable.

State estimation of a physical system with unknown governing equations

State estimation is concerned with reconciling noisy observations of a physical system with the mathematical model believed to predict its behaviour for the purpose of inferring unmeasurable states and denoising measurable ones1,2. Traditional state-estimation techniques rely on strong assumptions about the form of uncertainty in mathematical models, typically that it manifests as an additive stochastic perturbation or is parametric in nature3. Here we present a reparametrization trick for stochastic variational inference with Markov Gaussian processes that enables an approximate Bayesian approach for state estimation in which the equations governing how the system evolves over time are partially or completely unknown. In contrast to classical state-estimation techniques, our method learns the missing terms in the mathematical model and a state estimate simultaneously from an approximate Bayesian perspective. This development enables the application of state-estimation methods to problems that have so far proved to be beyond reach. Finally, although we focus on state estimation, the advancements to stochastic variational inference made here are applicable to a broader class of problems in machine learning.

CEBoosting: Online sparse identification of dynamical systems with regime switching by causation entropy boosting

Regime switching is ubiquitous in many complex dynamical systems with multiscale features, chaotic behavior, and extreme events. In this paper, a causation entropy boosting (CEBoosting) strategy is developed to facilitate the detection of regime switching and the discovery of the dynamics associated with the new regime via online model identification. The causation entropy, which can be efficiently calculated, provides a logic value of each candidate function in a pre-determined library. The reversal of one or a few such causation entropy indicators associated with the model calibrated for the current regime implies the detection of regime switching. Despite the short length of each batch formed by the sequential data, the accumulated value of causation entropy corresponding to a sequence of data batches leads to a robust indicator. With the detected rectification of the model structure, the subsequent parameter estimation becomes a quadratic optimization problem, which is solved using closed analytic formulas. Using the Lorenz 96 model, it is shown that the causation entropy indicator can be efficiently calculated, and the method applies to moderately large dimensional systems. The CEBoosting algorithm is also adaptive to the situation with partial observations. It is shown via a stochastic parameterized model that the CEBoosting strategy can be combined with data assimilation to identify regime switching triggered by the unobserved latent processes. In addition, the CEBoosting method is applied to a nonlinear paradigm model for topographic mean flow interaction, demonstrating the online detection of regime switching in the presence of strong intermittency and extreme events.

Neural ordinary differential equations with irregular and noisy data

Measurement noise is an integral part of collecting data of a physical process. Thus, noise removal is necessary to draw conclusions from these data, and it often becomes essential to construct dynamical models using these data. We discuss a methodology to learn differential equation(s) using noisy and irregularly sampled measurements. In our methodology, the main innovation can be seen in the integration of deep neural networks with the neural ordinary differential equations (ODEs) approach. Precisely, we aim at learning a neural network that provides (approximately) an implicit representation of the data and an additional neural network that models the vector fields of the dependent variables. We combine these two networks by constraints using neural ODEs. The proposed framework to learn a model describing the vector field is highly effective under noisy measurements. The approach can handle scenarios where dependent variables are unavailable at the same temporal grid. Moreover, a particular structure, e.g. second order with respect to time, can easily be incorporated. We demonstrate the effectiveness of the proposed method for learning models using data obtained from various differential equations and present a comparison with the neural ODE method that does not make any special treatment to noise. Additionally, we discuss an ensemble approach to improve the performance of the proposed approach further.

Learning dynamics on invariant measures using PDE-constrained optimization

We extend the methodology in Yang et al. [SIAM J. Appl. Dyn. Syst. 22, 269-310 (2023)] to learn autonomous continuous-time dynamical systems from invariant measures. The highlight of our approach is to reformulate the inverse problem of learning ODEs or SDEs from data as a PDE-constrained optimization problem. This shift in perspective allows us to learn from slowly sampled inference trajectories and perform uncertainty quantification for the forecasted dynamics. Our approach also yields a forward model with better stability than direct trajectory simulation in certain situations. We present numerical results for the Van der Pol oscillator and the Lorenz-63 system, together with real-world applications to Hall-effect thruster dynamics and temperature prediction, to demonstrate the effectiveness of the proposed approach.

Challenges in identifying simple pattern-forming mechanisms in the development of settlements using demographic data

The rapid increase of population and settlement structures in the Global South during recent decades has motivated the development of suitable models to describe their formation and evolution. Such settlement formation has been previously suggested to be dynamically driven by simple pattern-forming mechanisms. Here, we explore the use of a data-driven white-box approach, called SINDy, to discover differential equation models directly from available spatiotemporal demographic data for three representative regions of the Global South. We show that the current resolution and observation time of the available data are insufficient to uncover relevant pattern-forming mechanisms in settlement development. Using synthetic data generated with a generic pattern-forming model, the Allen-Cahn equation, we characterize what the requirements are for spatial and temporal resolution, as well as observation time, to successfully identify possible model system equations. Overall, the study provides a theoretical framework for the analysis of large-scale geographical and/or ecological systems, and it motivates further improvements in optimization approaches and data collection.

Direct Estimation of Parameters in ODE Models Using WENDy: Weak-form Estimation of Nonlinear Dynamics

We introduce the Weak-form Estimation of Nonlinear Dynamics (WENDy) method for estimating model parameters for non-linear systems of ODEs. Without relying on any numerical differential equation solvers, WENDy computes accurate estimates and is robust to large (biologically relevant) levels of measurement noise. For low dimensional systems with modest amounts of data, WENDy is competitive with conventional forward solver-based nonlinear least squares methods in terms of speed and accuracy. For both higher dimensional systems and stiff systems, WENDy is typically both faster (often by orders of magnitude) and more accurate than forward solver-based approaches. The core mathematical idea involves an efficient conversion of the strong form representation of a model to its weak form, and then solving a regression problem to perform parameter inference. The core statistical idea rests on the Errors-In-Variables framework, which necessitates the use of the iteratively reweighted least squares algorithm. Further improvements are obtained by using orthonormal test functions, created from a set of C-infinity bump functions of varying support sizes. We demonstrate the high robustness and computational efficiency by applying WENDy to estimate parameters in some common models from population biology, neuroscience, and biochemistry, including logistic growth, Lotka-Volterra, FitzHugh-Nagumo, Hindmarsh-Rose, and a Protein Transduction Benchmark model. Software and code for reproducing the examples is available at (https://github.com/MathBioCU/WENDy).

Deep kernel learning of dynamical models from high-dimensional noisy data

This work proposes a stochastic variational deep kernel learning method for the data-driven discovery of low-dimensional dynamical models from high-dimensional noisy data. The framework is composed of an encoder that compresses high-dimensional measurements into low-dimensional state variables, and a latent dynamical model for the state variables that predicts the system evolution over time. The training of the proposed model is carried out in an unsupervised manner, i.e., not relying on labeled data. Our learning method is evaluated on the motion of a pendulum-a well studied baseline for nonlinear model identification and control with continuous states and control inputs-measured via high-dimensional noisy RGB images. Results show that the method can effectively denoise measurements, learn compact state representations and latent dynamical models, as well as identify and quantify modeling uncertainties.

An improved sparse identification of nonlinear dynamics with Akaike information criterion and group sparsity

A crucial challenge encountered in diverse areas of engineering applications involves speculating the governing equations based upon partial observations. On this basis, a variant of the sparse identification of nonlinear dynamics (SINDy) algorithm is developed. First, the Akaike information criterion (AIC) is integrated to enforce model selection by hierarchically ranking the most informative model from several manageable candidate models. This integration avoids restricting the number of candidate models, which is a disadvantage of the traditional methods for model selection. The subsequent procedure expands the structure of dynamics from ordinary differential equations (ODEs) to partial differential equations (PDEs), while group sparsity is employed to identify the nonconstant coefficients of partial differential equations. Of practical consideration within an integrated frame is data processing, which tends to treat noise separate from signals and tends to parametrize the noise probability distribution. In particular, the coefficients of a species of canonical ODEs and PDEs, such as the Van der Pol, Rössler, Burgers' and Kuramoto-Sivashinsky equations, can be identified correctly with the introduction of noise. Furthermore, except for normal noise, the proposed approach is able to capture the distribution of uniform noise. In accordance with the results of the experiments, the computational speed is markedly advanced and possesses robustness.

Learning anisotropic interaction rules from individual trajectories in a heterogeneous cellular population

Interacting particle system (IPS) models have proven to be highly successful for describing the spatial movement of organisms. However, it is challenging to infer the interaction rules directly from data. In the field of equation discovery, the weak-form sparse identification of nonlinear dynamics (WSINDy) methodology has been shown to be computationally efficient for identifying the governing equations of complex systems from noisy data. Motivated by the success of IPS models to describe the spatial movement of organisms, we develop WSINDy for the second-order IPS to learn equations for communities of cells. Our approach learns the directional interaction rules for each individual cell that in aggregate govern the dynamics of a heterogeneous population of migrating cells. To sort a cell according to the active classes present in its model, we also develop a novel ad hoc classification scheme (which accounts for the fact that some cells do not have enough evidence to accurately infer a model). Aggregated models are then constructed hierarchically to simultaneously identify different species of cells present in the population and determine best-fit models for each species. We demonstrate the efficiency and proficiency of the method on several test scenarios, motivated by common cell migration experiments.

Process Model Inversion in the Data-Driven Engineering Context for Improved Parameter Sensitivities

Industry 4.0 has embraced process models in recent years, and the use of model-based digital twins has become even more critical in process systems engineering, monitoring, and control. However, the reliability of these models depends on the model parameters available. The accuracy of the estimated parameters is, in turn, determined by the amount and quality of the measurement data and the algorithm used for parameter identification. For the definition of the parameter identification problem, the ordinary least squares framework is still state-of-the-art in the literature, and better parameter estimates are only possible with additional data. In this work, we present an alternative strategy to identify model parameters by incorporating differential flatness for model inversion and neural ordinary differential equations for surrogate modeling. The novel concept results in an input-least-squares-based parameter identification problem with significant parameter sensitivity changes. To study these sensitivity effects, we use a classic one-dimensional diffusion-type problem, i.e., an omnipresent equation in process systems engineering and transport phenomena. As shown, the proposed concept ensures higher parameter sensitivities for two relevant scenarios. Based on the results derived, we also discuss general implications for data-driven engineering concepts used to identify process model parameters in the recent literature.

Data-driven model discovery of ideal four-wave mixing in nonlinear fibre optics

We show using numerical simulations that data driven discovery using sparse regression can be used to extract the governing differential equation model of ideal four-wave mixing in a nonlinear Schrödinger equation optical fibre system. Specifically, we consider the evolution of a strong single frequency pump interacting with two frequency detuned sidebands where the dynamics are governed by a reduced Hamiltonian system describing pump-sideband coupling. Based only on generated dynamical data from this system, sparse regression successfully recovers the underlying physical model, fully capturing the dynamical landscape on both sides of the system separatrix. We also discuss how analysing an ensemble over different initial conditions allows us to reliably identify the governing model in the presence of noise. These results extend the use of data driven discovery to ideal four-wave mixing in nonlinear Schrödinger equation systems.

Data-driven discovery of the governing equations of dynamical systems via moving horizon optimization

Discovering the governing laws underpinning physical and chemical phenomena entirely from data is a key step towards understanding and ultimately controlling systems in science and engineering. Noisy measurements and complex, highly nonlinear underlying dynamics hinder the identification of such governing laws. In this work, we introduce a machine learning framework rooted in moving horizon nonlinear optimization for identifying governing equations in the form of ordinary differential equations from noisy experimental data sets. Our approach evaluates sequential subsets of measurement data, and exploits statistical arguments to learn truly parsimonious governing equations from a large dictionary of basis functions. The proposed framework reduces gradient approximation errors by implicitly embedding an advanced numerical discretization scheme, which improves robustness to noise as well as to model stiffness. Canonical nonlinear dynamical system examples are used to demonstrate that our approach can accurately recover parsimonious governing laws under increasing levels of measurement noise, and outperform state of the art frameworks in the literature. Further, we consider a non-isothermal chemical reactor example to demonstrate that the proposed framework can cope with basis functions that have nonlinear (unknown) parameterizations.

Discovery of nonlinear dynamical systems using a Runge-Kutta inspired dictionary-based sparse regression approach

In this work, we blend machine learning and dictionary-based learning with numerical analysis tools to discover differential equations from noisy and sparsely sampled measurement data of time-dependent processes. We use the fact that given a dictionary containing large candidate nonlinear functions, dynamical models can often be described by a few appropriately chosen basis functions. As a result, we obtain parsimonious models that can be better interpreted by practitioners, and potentially generalize better beyond the sampling regime than black-box modelling. In this work, we integrate a numerical integration framework with dictionary learning that yields differential equations without requiring or approximating derivative information at any stage. Hence, it is utterly effective for corrupted and sparsely sampled data. We discuss its extension to governing equations, containing rational nonlinearities that typically appear in biological networks. Moreover, we generalized the method to governing equations subject to parameter variations and externally controlled inputs. We demonstrate the efficiency of the method to discover a number of diverse differential equations using noisy measurements, including a model describing neural dynamics, chaotic Lorenz model, Michaelis-Menten kinetics and a parameterized Hopf normal form.