Model selection for hybrid dynamical systems via sparse regression

Quantifying changes in individual-specific template-based representations of center-of-mass dynamics during walking with ankle exoskeletons using Hybrid-SINDy

Ankle exoskeletons alter whole-body walking mechanics, energetics, and stability by altering center-of-mass (CoM) motion. Controlling the dynamics governing CoM motion is, therefore, critical for maintaining efficient and stable gait. However, how CoM dynamics change with ankle exoskeletons is unknown, and how to optimally model individual-specific CoM dynamics, especially in individuals with neurological injuries, remains a challenge. Here, we evaluated individual-specific changes in CoM dynamics in unimpaired adults and one individual with post-stroke hemiparesis while walking in shoes-only and with zero-stiffness and high-stiffness passive ankle exoskeletons. To identify optimal sets of physically interpretable mechanisms describing CoM dynamics, termed template signatures, we leveraged hybrid sparse identification of nonlinear dynamics (Hybrid-SINDy), an equation-free data-driven method for inferring sparse hybrid dynamics from a library of candidate functional forms. In unimpaired adults, Hybrid-SINDy automatically identified spring-loaded inverted pendulum-like template signatures, which did not change with exoskeletons (p > 0.16), except for small changes in leg resting length (p < 0.001). Conversely, post-stroke paretic-leg rotary stiffness mechanisms increased by 37-50% with zero-stiffness exoskeletons. While unimpaired CoM dynamics appear robust to passive ankle exoskeletons, how neurological injuries alter exoskeleton impacts on CoM dynamics merits further investigation. Our findings support Hybrid-SINDy's potential to discover mechanisms describing individual-specific CoM dynamics with assistive devices.

Regularized least absolute deviation-based sparse identification of dynamical systems

This work develops a regularized least absolute deviation-based sparse identification of dynamics (RLAD-SID) method to address outlier problems in the classical metric-based loss function and the sparsity constraint framework. Our method uses absolute derivation loss as a substitute of Euclidean loss. Moreover, a corresponding computationally efficient optimization algorithm is derived on the basis of the alternating direction method of multipliers due to the non-smoothness of both the new proposed loss function and the regularization term. Numerical experiments are performed to evaluate the effectiveness of RLAD-SID using several exemplary nonlinear dynamical systems, such as the van der Pol equation, the Lorenz system, and the 1D discrete logistic map. Furthermore, detailed numerical comparisons are provided with other existing methods in metric-based sparse regression. Numerical results demonstrate that (1) RLAD-SID shows significant robustness toward a large outlier and (2) RLAD-SID can be seen as a particular metric-based sparse regression strategy that exhibits the effectiveness of the metric-based sparse regression framework for solving outlier problems in a dynamical system identification.

An improved sparse identification of nonlinear dynamics with Akaike information criterion and group sparsity

A crucial challenge encountered in diverse areas of engineering applications involves speculating the governing equations based upon partial observations. On this basis, a variant of the sparse identification of nonlinear dynamics (SINDy) algorithm is developed. First, the Akaike information criterion (AIC) is integrated to enforce model selection by hierarchically ranking the most informative model from several manageable candidate models. This integration avoids restricting the number of candidate models, which is a disadvantage of the traditional methods for model selection. The subsequent procedure expands the structure of dynamics from ordinary differential equations (ODEs) to partial differential equations (PDEs), while group sparsity is employed to identify the nonconstant coefficients of partial differential equations. Of practical consideration within an integrated frame is data processing, which tends to treat noise separate from signals and tends to parametrize the noise probability distribution. In particular, the coefficients of a species of canonical ODEs and PDEs, such as the Van der Pol, Rössler, Burgers' and Kuramoto-Sivashinsky equations, can be identified correctly with the introduction of noise. Furthermore, except for normal noise, the proposed approach is able to capture the distribution of uniform noise. In accordance with the results of the experiments, the computational speed is markedly advanced and possesses robustness.

Stability selection enables robust learning of differential equations from limited noisy data

We present a statistical learning framework for robust identification of differential equations from noisy spatio-temporal data. We address two issues that have so far limited the application of such methods, namely their robustness against noise and the need for manual parameter tuning, by proposing stability-based model selection to determine the level of regularization required for reproducible inference. This avoids manual parameter tuning and improves robustness against noise in the data. Our stability selection approach, termed PDE-STRIDE, can be combined with any sparsity-promoting regression method and provides an interpretable criterion for model component importance. We show that the particular combination of stability selection with the iterative hard-thresholding algorithm from compressed sensing provides a fast and robust framework for equation inference that outperforms previous approaches with respect to accuracy, amount of data required, and robustness. We illustrate the performance of PDE-STRIDE on a range of simulated benchmark problems, and we demonstrate the applicability of PDE-STRIDE on real-world data by considering purely data-driven inference of the protein interaction network for embryonic polarization in Caenorhabditis elegans. Using fluorescence microscopy images of C. elegans zygotes as input data, PDE-STRIDE is able to learn the molecular interactions of the proteins.

Model selection of chaotic systems from data with hidden variables using sparse data assimilation

Many natural systems exhibit chaotic behavior, including the weather, hydrology, neuroscience, and population dynamics. Although many chaotic systems can be described by relatively simple dynamical equations, characterizing these systems can be challenging due to sensitivity to initial conditions and difficulties in differentiating chaotic behavior from noise. Ideally, one wishes to find a parsimonious set of equations that describe a dynamical system. However, model selection is more challenging when only a subset of the variables are experimentally accessible. Manifold learning methods using time-delay embeddings can successfully reconstruct the underlying structure of the system from data with hidden variables, but not the equations. Recent work in sparse-optimization based model selection has enabled model discovery given a library of possible terms, but regression-based methods require measurements of all state variables. We present a method combining variational annealing-a technique previously used for parameter estimation in chaotic systems with hidden variables-with sparse-optimization methods to perform model identification for chaotic systems with unmeasured variables. We applied the method to ground-truth time-series simulated from the classic Lorenz system and experimental data from an electrical circuit with Lorenz-system like behavior. In both cases, we successfully recover the expected equations with two measured and one hidden variable. Application to simulated data from the Colpitts oscillator demonstrates successful model selection of terms within nonlinear functions. We discuss the robustness of our method to varying noise.

Quantifying nanoscale forces using machine learning in dynamic atomic force microscopy

Dynamic atomic force microscopy (AFM) is a key platform that enables topological and nanomechanical characterization of novel materials. This is achieved by linking the nanoscale forces that exist between the AFM tip and the sample to specific mathematical functions through modeling. However, the main challenge in dynamic AFM is to quantify these nanoscale forces without the use of complex models that are routinely used to explain the physics of tip-sample interaction. Here, we make use of machine learning and data science to characterize tip-sample forces purely from experimental data with sub-microsecond resolution. Our machine learning approach is first trained on standard AFM models and then showcased experimentally on a polymer blend of polystyrene (PS) and low density polyethylene (LDPE) sample. Using this algorithm we probe the complex physics of tip-sample contact in polymers, estimate elasticity, and provide insight into energy dissipation during contact. Our study opens a new route in dynamic AFM characterization where machine learning can be combined with experimental methodologies to probe transient processes involved in phase transformation as well as complex chemical and biological phenomena in real-time.

Automatic differentiation to simultaneously identify nonlinear dynamics and extract noise probability distributions from data

The sparse identification of nonlinear dynamics (SINDy) is a regression framework for the discovery of parsimonious dynamic models and governing equations from time-series data. As with all system identification methods, noisy measurements compromise the accuracy and robustness of the model discovery procedure. In this work we develop a variant of the SINDy algorithm that integrates automatic differentiation and recent time-stepping constrained motivated by Rudy et al (2019 J. Computat. Phys. 396 483–506) for simultaneously (1) denoising the data, (2) learning and parametrizing the noise probability distribution, and (3) identifying the underlying parsimonious dynamical system responsible for generating the time-series data. Thus within an integrated optimization framework, noise can be separated from signal, resulting in an architecture that is approximately twice as robust to noise as state-of-the-art methods, handling as much as 40% noise on a given time-series signal and explicitly parametrizing the noise probability distribution. We demonstrate this approach on several numerical examples, from Lotka-Volterra models to the spatio-temporal Lorenz 96 model. Further, we show the method can learn a diversity of probability distributions for the measurement noise, including Gaussian, uniform, Gamma, and Rayleigh distributions.

Data-driven identification of dynamical models using adaptive parameter sets

This paper presents two data-driven model identification techniques for dynamical systems with fixed point attractors. Both strategies implement adaptive parameter update rules to limit truncation errors in the inferred dynamical models. The first strategy can be considered an extension of the dynamic mode decomposition with control (DMDc) algorithm. The second strategy uses a reduced order isostable coordinate basis that captures the behavior of the slowest decaying modes of the Koopman operator. The accuracy and robustness of both model identification algorithms is considered in a simple model with dynamics near a Hopf bifurcation. A more complicated model for nonlinear convective flow past an obstacle is also considered. In these examples, the proposed strategies outperform a collection of other commonly used data-driven model identification algorithms including Koopman model predictive control, Galerkin projection, and DMDc.

Uncertainty Analysis and Experimental Validation of Identifying the Governing Equation of an Oscillator Using Sparse Regression

In recent years, the rapid growth of computing technology has enabled identifying mathematical models for vibration systems using measurement data instead of domain knowledge. Within this category, the method Sparse Identification of Nonlinear Dynamical Systems (SINDy) shows potential for interpretable identification. Therefore, in this work, a procedure of system identification based on the SINDy framework is developed and validated on a single-mass oscillator. To estimate the parameters in the SINDy model, two sparse regression methods are discussed. Compared with the Least Squares method with Sequential Threshold (LSST), which is the original estimation method from SINDy, the Least Squares method Post-LASSO (LSPL) shows better performance in numerical Monte Carlo Simulations (MCSs) of a single-mass oscillator in terms of sparseness, convergence, identified eigenfrequency, and coefficient of determination. Furthermore, the developed method SINDy-LSPL was successfully implemented with real measurement data of a single-mass oscillator with known theoretical parameters. The identified parameters using a sweep signal as excitation are more consistent and accurate than those identified using impulse excitation. In both cases, there exists a dependency of the identified parameter on the excitation amplitude that should be investigated in further research.

Multiscale modeling meets machine learning: What can we learn?

Machine learning is increasingly recognized as a promising technology in the biological, biomedical, and behavioral sciences. There can be no argument that this technique is incredibly successful in image recognition with immediate applications in diagnostics including electrophysiology, radiology, or pathology, where we have access to massive amounts of annotated data. However, machine learning often performs poorly in prognosis, especially when dealing with sparse data. This is a field where classical physics-based simulation seems to remain irreplaceable. In this review, we identify areas in the biomedical sciences where machine learning and multiscale modeling can mutually benefit from one another: Machine learning can integrate physics-based knowledge in the form of governing equations, boundary conditions, or constraints to manage ill-posted problems and robustly handle sparse and noisy data; multiscale modeling can integrate machine learning to create surrogate models, identify system dynamics and parameters, analyze sensitivities, and quantify uncertainty to bridge the scales and understand the emergence of function. With a view towards applications in the life sciences, we discuss the state of the art of combining machine learning and multiscale modeling, identify applications and opportunities, raise open questions, and address potential challenges and limitations. We anticipate that it will stimulate discussion within the community of computational mechanics and reach out to other disciplines including mathematics, statistics, computer science, artificial intelligence, biomedicine, systems biology, and precision medicine to join forces towards creating robust and efficient models for biological systems.

Biochemical adaptations of the stout razor clam (Tagelus plebeius) to changes in oxygen availability: resilience in a changing world?

Climate change is producing sea level rise and deoxygenation of the ocean, altering estuaries and coastal areas. Changes in oxygen availability are expected to have consequences on the physiological fitness of intertidal species. In this work we analyze the coping response of the intertidal stout razor clam (Tagelus plebeius (Lightfoot, 1786)) to extreme environmental changes in oxygen concentration. Their biochemical responses to normoxia, hypoxia, and hyperoxia transition at different intertidal level (low–high) were measured through an in situ transplant experiment. The high intertidal level negatively affected the analyzed traits of the T. plebeius populations. The differences in reactive oxygen species production, total oxyradical scavenger capacities, and catalase activity also suggested more stressful conditions at the high level where long-term hypoxia periods occur. Both hypoxia and re-oxygenation provoked re-adjustments in the antioxidant responses and higher lipid oxidative damage (normoxia < hypoxia < re-oxygenation). The observed responses in transplanted clams at the opposite intertidal level suggested the potential acclimation of T. plebeius to cope with new environmental conditions. These findings are discussed within a global changing context where both increasing deoxygenation conditions and sea level rise are predicted to be exacerbated in the area driven by climate change.

Modeling of dynamical systems through deep learning

This review presents a modern perspective on dynamical systems in the context of current goals and open challenges. In particular, our review focuses on the key challenges of discovering dynamics from data and finding data-driven representations that make nonlinear systems amenable to linear analysis. We explore various challenges in modern dynamical systems, along with emerging techniques in data science and machine learning to tackle them. The two chief challenges are (1) nonlinear dynamics and (2) unknown or partially known dynamics. Machine learning is providing new and powerful techniques for both challenges. Dimensionality reduction methods are used for projecting dynamical methods in reduced form, and these methods perform computational efficiency on real-world data. Data-driven models drive to discover the governing equations and give laws of physics. The identification of dynamical systems through deep learning techniques succeeds in inferring physical systems. Machine learning provides advanced new and powerful algorithms for nonlinear dynamics. Advanced deep learning methods like autoencoders, recurrent neural networks, convolutional neural networks, and reinforcement learning are used in modeling of dynamical systems.

SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics

Accurately modelling the nonlinear dynamics of a system from measurement data is a challenging yet vital topic. The sparse identification of nonlinear dynamics (SINDy) algorithm is one approach to discover dynamical systems models from data. Although extensions have been developed to identify implicit dynamics, or dynamics described by rational functions, these extensions are extremely sensitive to noise. In this work, we develop SINDy-PI (parallel, implicit), a robust variant of the SINDy algorithm to identify implicit dynamics and rational nonlinearities. The SINDy-PI framework includes multiple optimization algorithms and a principled approach to model selection. We demonstrate the ability of this algorithm to learn implicit ordinary and partial differential equations and conservation laws from limited and noisy data. In particular, we show that the proposed approach is several orders of magnitude more noise robust than previous approaches, and may be used to identify a class of ODE and PDE dynamics that were previously unattainable with SINDy, including for the double pendulum dynamics and simplified model for the Belousov-Zhabotinsky (BZ) reaction.

Algorithmic discovery of dynamic models from infectious disease data

Theoretical models are typically developed through a deductive process where a researcher formulates a system of dynamic equations from hypothesized mechanisms. Recent advances in algorithmic methods can discover dynamic models inductively-directly from data. Most previous research has tested these methods by rediscovering models from synthetic data generated by the already known model. Here we apply Sparse Identification of Nonlinear Dynamics (SINDy) to discover mechanistic equations for disease dynamics from case notification data for measles, chickenpox, and rubella. The discovered models provide a good qualitative fit to the observed dynamics for all three diseases, However, the SINDy chickenpox model appears to overfit the empirical data, and recovering qualitatively correct rubella dynamics requires using power spectral density in the goodness-of-fit criterion. When SINDy uses a library of second-order functions, the discovered models tend to include mass action incidence and a seasonally varying transmission rate-a common feature of existing epidemiological models for childhood infectious diseases. We also find that the SINDy measles model is capable of out-of-sample prediction of a dynamical regime shift in measles case notification data. These results demonstrate the potential for algorithmic model discovery to enrich scientific understanding by providing a complementary approach to developing theoretical models.

Integrating machine learning and multiscale modeling-perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences

Fueled by breakthrough technology developments, the biological, biomedical, and behavioral sciences are now collecting more data than ever before. There is a critical need for time- and cost-efficient strategies to analyze and interpret these data to advance human health. The recent rise of machine learning as a powerful technique to integrate multimodality, multifidelity data, and reveal correlations between intertwined phenomena presents a special opportunity in this regard. However, machine learning alone ignores the fundamental laws of physics and can result in ill-posed problems or non-physical solutions. Multiscale modeling is a successful strategy to integrate multiscale, multiphysics data and uncover mechanisms that explain the emergence of function. However, multiscale modeling alone often fails to efficiently combine large datasets from different sources and different levels of resolution. Here we demonstrate that machine learning and multiscale modeling can naturally complement each other to create robust predictive models that integrate the underlying physics to manage ill-posed problems and explore massive design spaces. We review the current literature, highlight applications and opportunities, address open questions, and discuss potential challenges and limitations in four overarching topical areas: ordinary differential equations, partial differential equations, data-driven approaches, and theory-driven approaches. Towards these goals, we leverage expertise in applied mathematics, computer science, computational biology, biophysics, biomechanics, engineering mechanics, experimentation, and medicine. Our multidisciplinary perspective suggests that integrating machine learning and multiscale modeling can provide new insights into disease mechanisms, help identify new targets and treatment strategies, and inform decision making for the benefit of human health.

Time-dependent antagonist-agonist switching in receptor tyrosine kinase-mediated signaling

BACKGROUND

ErbB4/HER4 is a unique member of the ErbB family of receptor tyrosine kinases concerning its activation of anti-proliferative JAK2-STAT5 pathway when stimulated by ligand Neuregulin (NRG). Activation of this pathway leads to expression of genes like β-casein which promote cell differentiation. Recent experimental studies on mouse HC11 mammary epithelial cells stimulated by ligand Neuregulin (NRG) showed a time-dependent switching behavior in the β-casein expression. This behavior cannot be explained using currently available mechanistic models of the JAK-STAT pathway. We constructed an improved mechanistic model which introduces two crucial modifications to the canonical HER4-JAK2-STAT5 pathway based on literature findings. These modifications include competitive HER4 heterodimerization with other members of the ErbB family and a slower JAK2 independent activation STAT5 through HER4. We also performed global sensitivity analysis on the model to test the robustness of the predictions and parameter combinations that are sensitive to the outcome.

RESULTS

Our model was able to reproduce the time-dependent switching behavior of β-casein and also establish that the modifications mentioned above to the canonical JAK-STAT pathway are necessary to reproduce this behavior. The sensitivity studies show that the competitive HER4 heterodimerization reactions have a profound impact on the sensitivity of the pathway to NRG stimulation, while the slower JAK2-independent pathway is necessary for the late stage promotion of β-casein mRNA transcription. The difference in the time scales of the JAK-dependent and JAK-independent pathways was found to be the main contributing factor to the time-dependent switch. The transport rates controlling activated STAT5 dimer nuclear import and β-casein mRNA export to cytoplasm affected the time delay between NRG stimulation and peak β-casein mRNA activity.

CONCLUSION

This study highlights the effect of competitive and parallel reaction pathways on both short and long-term dynamics of receptor-mediated signaling. It provides robust and testable predictions of the dynamical behavior of the HER4 mediated JAK-STAT pathway which could be useful in designing treatments for various cancers where this pathway is activated/altered.