Machine-learning inference of fluid variables from data using reservoir computing

Data-driven ordinary-differential-equation modeling of high-frequency complex dynamics via a low-frequency dynamics model

In our previous paper [N. Tsutsumi et al., Chaos 32, 091101 (2022)10.1063/5.0100166], we proposed a method for constructing a system of differential equations of chaotic behavior from only observable deterministic time series, which we call the radial function-based regression (RfR) method. However, when the targeted variable's behavior is rather complex, the direct application of the RfR method does not function well. In this study, we propose a method of modeling such dynamics, including the high-frequency intermittent behavior of a fluid flow, by considering another variable (base variable) showing relatively simple, less intermittent behavior. We construct an autonomous joint model composed of two parts: the first is an autonomous system of a base variable, and the other concerns the targeted variable being affected by a term involving the base variable to demonstrate complex dynamics. The constructed joint model succeeded in not only inferring a short trajectory but also reconstructing chaotic sets and statistical properties obtained from a long trajectory such as the density distributions of the actual dynamics.

Learning noise-induced transitions by multi-scaling reservoir computing

Noise is usually regarded as adversarial to extracting effective dynamics from time series, such that conventional approaches usually aim at learning dynamics by mitigating the noisy effect. However, noise can have a functional role in driving transitions between stable states underlying many stochastic dynamics. We find that leveraging a machine learning model, reservoir computing, can learn noise-induced transitions. We propose a concise training protocol with a focus on a pivotal hyperparameter controlling the time scale. The approach is widely applicable, including a bistable system with white noise or colored noise, where it generates accurate statistics of transition time for white noise and specific transition time for colored noise. Instead, the conventional approaches such as SINDy and the recurrent neural network do not faithfully capture stochastic transitions even for the case of white noise. The present approach is also aware of asymmetry of the bistable potential, rotational dynamics caused by non-detailed balance, and transitions in multi-stable systems. For the experimental data of protein folding, it learns statistics of transition time between folded states, enabling us to characterize transition dynamics from a small dataset. The results portend the exploration of extending the prevailing approaches in learning dynamics from noisy time series.

Reservoir-computing based associative memory and itinerancy for complex dynamical attractors

Traditional neural network models of associative memories were used to store and retrieve static patterns. We develop reservoir-computing based memories for complex dynamical attractors, under two common recalling scenarios in neuropsychology: location-addressable with an index channel and content-addressable without such a channel. We demonstrate that, for location-addressable retrieval, a single reservoir computing machine can memorize a large number of periodic and chaotic attractors, each retrievable with a specific index value. We articulate control strategies to achieve successful switching among the attractors, unveil the mechanism behind failed switching, and uncover various scaling behaviors between the number of stored attractors and the reservoir network size. For content-addressable retrieval, we exploit multistability with cue signals, where the stored attractors coexist in the high-dimensional phase space of the reservoir network. As the length of the cue signal increases through a critical value, a high success rate can be achieved. The work provides foundational insights into developing long-term memories and itinerancy for complex dynamical patterns.

Attractor reconstruction with reservoir computers: The effect of the reservoir's conditional Lyapunov exponents on faithful attractor reconstruction

Reservoir computing is a machine learning framework that has been shown to be able to replicate the chaotic attractor, including the fractal dimension and the entire Lyapunov spectrum, of the dynamical system on which it is trained. We quantitatively relate the generalized synchronization dynamics of a driven reservoir during the training stage to the performance of the trained reservoir computer at the attractor reconstruction task. We show that, in order to obtain successful attractor reconstruction and Lyapunov spectrum estimation, the maximal conditional Lyapunov exponent of the driven reservoir must be significantly more negative than the most negative Lyapunov exponent of the target system. We also find that the maximal conditional Lyapunov exponent of the reservoir depends strongly on the spectral radius of the reservoir adjacency matrix; therefore, for attractor reconstruction and Lyapunov spectrum estimation, small spectral radius reservoir computers perform better in general. Our arguments are supported by numerical examples on well-known chaotic systems.

Constructing polynomial libraries for reservoir computing in nonlinear dynamical system forecasting

Reservoir computing is an effective model for learning and predicting nonlinear and chaotic dynamical systems; however, there remains a challenge in achieving a more dependable evolution for such systems. Based on the foundation of Koopman operator theory, considering the effectiveness of the sparse identification of nonlinear dynamics algorithm to construct candidate nonlinear libraries in the application of nonlinear data, an alternative reservoir computing method is proposed, which creates the linear Hilbert space of the nonlinear system by including nonlinear terms in the optimization process of reservoir computing, allowing for the application of linear optimization. We introduce an implementation that incorporates a polynomial transformation of arbitrary order when fitting the readout matrix. Constructing polynomial libraries with reservoir-state vectors as elements enhances the nonlinear representation of reservoir states and more easily captures the complexity of nonlinear systems. The Lorenz-63 system, the Lorenz-96 system, and the Kuramoto-Sivashinsky equation are used to validate the effectiveness of constructing polynomial libraries for reservoir states in the field of state-evolution prediction of nonlinear and chaotic dynamical systems. This study not only promotes the theoretical study of reservoir computing, but also provides a theoretical and practical method for the prediction of nonlinear and chaotic dynamical system evolution.

Global forecasts in reservoir computers

A reservoir computer is a machine learning model that can be used to predict the future state(s) of time-dependent processes, e.g., dynamical systems. In practice, data in the form of an input-signal are fed into the reservoir. The trained reservoir is then used to predict the future state of this signal. We develop a new method for not only predicting the future dynamics of the input-signal but also the future dynamics starting at an arbitrary initial condition of a system. The systems we consider are the Lorenz, Rossler, and Thomas systems restricted to their attractors. This method, which creates a global forecast, still uses only a single input-signal to train the reservoir but breaks the signal into many smaller windowed signals. We examine how well this windowed method is able to forecast the dynamics of a system starting at an arbitrary point on a system's attractor and compare this to the standard method without windows. We find that the standard method has almost no ability to forecast anything but the original input-signal while the windowed method can capture the dynamics starting at most points on an attractor with significant accuracy.

Variability of echo state network prediction horizon for partially observed dynamical systems

Study of dynamical systems using partial state observation is an important problem due to its applicability to many real-world systems. We address the problem by studying an echo state network (ESN) framework with partial state input with partial or full state output. Application to the Lorenz system and Chua's oscillator (both numerically simulated and experimental systems) demonstrate the effectiveness of our method. We show that the ESN, as an autonomous dynamical system, is capable of making short-term predictions up to a few Lyapunov times. However, the prediction horizon has high variability depending on the initial condition-an aspect that we explore in detail using the distribution of the prediction horizon. Further, using a variety of statistical metrics to compare the long-term dynamics of the ESN predictions with numerically simulated or experimental dynamics and observed similar results, we show that the ESN can effectively learn the system's dynamics even when trained with noisy numerical or experimental data sets. Thus, we demonstrate the potential of ESNs to serve as cheap surrogate models for simulating the dynamics of systems where complete observations are unavailable.

Constructing low-dimensional ordinary differential equations from chaotic time series of high- or infinite-dimensional systems using radial-function-based regression

In our previous study [N. Tsutsumi, K. Nakai, and Y. Saiki, Chaos 32, 091101 (2022)1054-150010.1063/5.0100166] we proposed a method of constructing a system of ordinary differential equations of chaotic behavior only from observable deterministic time series, which we will call the radial-function-based regression (RfR) method. The RfR method employs a regression using Gaussian radial basis functions together with polynomial terms to facilitate the robust modeling of chaotic behavior. In this paper, we apply the RfR method to several example time series of high- or infinite-dimensional deterministic systems, and we construct a system of relatively low-dimensional ordinary differential equations with a large number of terms. The examples include time series generated from a partial differential equation, a delay differential equation, a turbulence model, and intermittent dynamics. The case when the observation includes noise is also tested. We have effectively constructed a system of differential equations for each of these examples, which is assessed from the point of view of time series forecast, reconstruction of invariant sets, and invariant densities. We find that in some of the models, an appropriate trajectory is realized on the chaotic saddle and is identified by the stagger-and-step method.

Toward accelerated data-driven Rayleigh-Bénard convection simulations

A hybrid data-driven/finite volume method for 2D and 3D thermal convective flows is introduced. The approach relies on a single-step loss, convolutional neural network that is active only in the near-wall region of the flow. We demonstrate that the method significantly reduces errors in the prediction of the heat flux over the long-time horizon and increases pointwise accuracy in coarse simulations, when compared to direct computations on the same grids with and without a traditional subgrid model. We trace the success of our machine learning model to the choice of the training procedure, incorporating both the temporal flow development and distributional bias.

Extreme Events Prediction from Nonlocal Partial Information in a Spatiotemporally Chaotic Microcavity Laser

The forecasting of high-dimensional, spatiotemporal nonlinear systems has made tremendous progress with the advent of model-free machine learning techniques. However, in real systems it is not always possible to have all the information needed; only partial information is available for learning and forecasting. This can be due to insufficient temporal or spatial samplings, to inaccessible variables, or to noisy training data. Here, we show that it is nevertheless possible to forecast extreme event occurrences in incomplete experimental recordings from a spatiotemporally chaotic microcavity laser using reservoir computing. Selecting regions of maximum transfer entropy, we show that it is possible to get higher forecasting accuracy using nonlocal data vs local data, thus allowing greater warning times of at least twice the time horizon predicted from the nonlinear local Lyapunov exponent.

Reservoir computing as digital twins for nonlinear dynamical systems

We articulate the design imperatives for machine learning based digital twins for nonlinear dynamical systems, which can be used to monitor the "health" of the system and anticipate future collapse. The fundamental requirement for digital twins of nonlinear dynamical systems is dynamical evolution: the digital twin must be able to evolve its dynamical state at the present time to the next time step without further state input-a requirement that reservoir computing naturally meets. We conduct extensive tests using prototypical systems from optics, ecology, and climate, where the respective specific examples are a chaotic CO₂ laser system, a model of phytoplankton subject to seasonality, and the Lorenz-96 climate network. We demonstrate that, with a single or parallel reservoir computer, the digital twins are capable of a variety of challenging forecasting and monitoring tasks. Our digital twin has the following capabilities: (1) extrapolating the dynamics of the target system to predict how it may respond to a changing dynamical environment, e.g., a driving signal that it has never experienced before, (2) making continual forecasting and monitoring with sparse real-time updates under non-stationary external driving, (3) inferring hidden variables in the target system and accurately reproducing/predicting their dynamical evolution, (4) adapting to external driving of different waveform, and (5) extrapolating the global bifurcation behaviors to network systems of different sizes. These features make our digital twins appealing in applications, such as monitoring the health of critical systems and forecasting their potential collapse induced by environmental changes or perturbations. Such systems can be an infrastructure, an ecosystem, or a regional climate system.

Forecasting macroscopic dynamics in adaptive Kuramoto network using reservoir computing

Forecasting a system's behavior is an essential task encountering the complex systems theory. Machine learning offers supervised algorithms, e.g., recurrent neural networks and reservoir computers that predict the behavior of model systems whose states consist of multidimensional time series. In real life, we often have limited information about the behavior of complex systems. The brightest example is the brain neural network described by the electroencephalogram. Forecasting the behavior of these systems is a more challenging task but provides a potential for real-life application. Here, we trained reservoir computer to predict the macroscopic signal produced by the network of phase oscillators. The Lyapunov analysis revealed the chaotic nature of the signal and reservoir computer failed to forecast it. Augmenting the feature space using Takkens' theorem improved the quality of forecasting. RC achieved the best prediction score when the number of signals coincided with the embedding dimension estimated via the nearest false neighbors method. We found that short-time prediction required a large number of features, while long-time prediction utilizes a limited number of features. These results refer to the bias-variance trade-off, an important concept in machine learning.

Time series reconstructing using calibrated reservoir computing

Reservoir computing, a new method of machine learning, has recently been used to predict the state evolution of various chaotic dynamic systems. It has significant advantages in terms of training cost and adjusted parameters; however, the prediction length is limited. For classic reservoir computing, the prediction length can only reach five to six Lyapunov times. Here, we modified the method of reservoir computing by adding feedback, continuous or discrete, to “calibrate” the input of the reservoir and then reconstruct the entire dynamic systems. The reconstruction length appreciably increased and the training length obviously decreased. The reconstructing of dynamical systems is studied in detail under this method. The reconstruction can be significantly improved both in length and accuracy. Additionally, we summarized the effect of different kinds of input feedback. The more it interacts with others in dynamical equations, the better the reconstructions. Nonlinear terms can reveal more information than linear terms once the interaction terms are equal. This method has proven effective via several classical chaotic systems. It can be superior to traditional reservoir computing in reconstruction, provides new hints in computing promotion, and may be used in some real applications.

Constructing differential equations using only a scalar time-series about continuous time chaotic dynamics

We propose a simple method of constructing a system of differential equations of chaotic behavior based on the regression only from scalar observable time-series data. The estimated system enables us to reconstruct invariant sets and statistical properties as well as to infer short time-series. Our successful modeling relies on the introduction of a set of Gaussian radial basis functions to capture local structures. The proposed method is used to construct a system of ordinary differential equations whose orbit reconstructs a time-series of a variable of the well-known Lorenz system as a simple but typical example. A system for a macroscopic fluid variable is also constructed.

Dynamical system analysis of a data-driven model constructed by reservoir computing

This study evaluates data-driven models from a dynamical system perspective, such as unstable fixed points, periodic orbits, chaotic saddle, Lyapunov exponents, manifold structures, and statistical values. We find that these dynamical characteristics can be reconstructed much more precisely by a data-driven model than by computing directly from training data. With this idea, we predict the laminar lasting time distribution of a particular macroscopic variable of chaotic fluid flow, which cannot be calculated from a direct numerical simulation of the Navier-Stokes equation because of its high computational cost.

Predicting amplitude death with machine learning

In nonlinear dynamics, a parameter drift can lead to a sudden and complete cessation of the oscillations of the state variables-the phenomenon of amplitude death. The underlying bifurcation is one at which the system settles into a steady state from chaotic or regular oscillations. As the normal functioning of many physical, biological, and physiological systems hinges on oscillations, amplitude death is undesired. To predict amplitude death in advance of its occurrence based solely on oscillatory time series collected while the system still functions normally is a challenge problem. We exploit machine learning to meet this challenge. In particular, we develop the scheme of "parameter-aware" reservoir computing, where training is conducted for a small number of bifurcation parameter values in the oscillatory regime to enable prediction upon a parameter drift into the regime of amplitude death. We demonstrate successful prediction of amplitude death for three prototypical dynamical systems in which the transition to death is preceded by either chaotic or regular oscillations. Because of the completely data-driven nature of the prediction framework, potential applications to real-world systems can be anticipated.

Finding nonlinear system equations and complex network structures from data: A sparse optimization approach

In applications of nonlinear and complex dynamical systems, a common situation is that the system can be measured, but its structure and the detailed rules of dynamical evolution are unknown. The inverse problem is to determine the system equations and structure from time series. The principle of exploiting sparse optimization to find the equations of dynamical systems from data was first articulated in 2011 by the ASU group. The basic idea is to expand the system equations into a power series or a Fourier series of a finite number of terms and then to determine the vector of the expansion coefficients based solely on data through sparse optimization. This Tutorial presents a brief review of the recent progress in this area. Issues discussed include discovering the equations of stationary or nonstationary chaotic systems to enable the prediction of critical transition and system collapse, inferring the full topology of complex oscillator networks and social networks hosting evolutionary game dynamics, and identifying partial differential equations for spatiotemporal dynamical systems. Situations where sparse optimization works or fails are pointed out. The relation with the traditional delay-coordinate embedding method is discussed, and the recent development of a model-free, data-driven prediction framework based on machine learning is mentioned.

Robust Optimization and Validation of Echo State Networks for learning chaotic dynamics

An approach to the time-accurate prediction of chaotic solutions is by learning temporal patterns from data. Echo State Networks (ESNs), which are a class of Reservoir Computing, can accurately predict the chaotic dynamics well beyond the predictability time. Existing studies, however, also showed that small changes in the hyperparameters may markedly affect the network's performance. The overarching aim of this paper is to improve the robustness in the selection of hyperparameters in Echo State Networks for the time-accurate prediction of chaotic solutions. We define the robustness of a validation strategy as its ability to select hyperparameters that perform consistently between validation and test sets. The goal is three-fold. First, we investigate routinely used validation strategies. Second, we propose the Recycle Validation, and the chaotic versions of existing validation strategies, to specifically tackle the forecasting of chaotic systems. Third, we compare Bayesian optimization with the traditional grid search for optimal hyperparameter selection. Numerical tests are performed on prototypical nonlinear systems that have chaotic and quasiperiodic solutions, such as the Lorenz and Lorenz-96 systems, and the Kuznetsov oscillator. Both model-free and model-informed Echo State Networks are analysed. By comparing the networks' performance in learning chaotic (unpredictable) versus quasiperiodic (predictable) solutions, we highlight fundamental challenges in learning chaotic solutions. The proposed validation strategies, which are based on the dynamical systems properties of chaotic time series, are shown to outperform the state-of-the-art validation strategies. Because the strategies are principled - they are based on chaos theory such as the Lyapunov time - they can be applied to other Recurrent Neural Networks architectures with little modification. This work opens up new possibilities for the robust design and application of Echo State Networks, and Recurrent Neural Networks, to the time-accurate prediction of chaotic systems.

Effective models and predictability of chaotic multiscale systems via machine learning

Understanding and modeling the dynamics of multiscale systems is a problem of considerable interest both for theory and applications. For unavoidable practical reasons, in multiscale systems, there is the need to eliminate from the description the fast and small-scale degrees of freedom and thus build effective models for only the slow and large-scale degrees of freedom. When there is a wide scale separation between the degrees of freedom, asymptotic techniques, such as the adiabatic approximation, can be used for devising such effective models, while away from this limit there exist no systematic techniques. Here, we scrutinize the use of machine learning, based on reservoir computing, to build data-driven effective models of multiscale chaotic systems. We show that, for a wide scale separation, machine learning generates effective models akin to those obtained using multiscale asymptotic techniques and, remarkably, remains effective in predictability also when the scale separation is reduced. We also show that predictability can be improved by hybridizing the reservoir with an imperfect model.

Calibrated reservoir computers

We observe the presence of infinitely fine-scaled alternations within the performance landscape of reservoir computers aimed for chaotic data forecasting. We investigate the emergence of the observed structures by means of variations of the transversal stability of the synchronization manifold relating the observational and internal dynamical states. Finally, we deduce a simple calibration method in order to attenuate the thus evidenced performance uncertainty.

Cancer: A turbulence problem

•

Application of nonlinear dynamics to cancer ecosystems.

•

Chemical turbulence and strange attractor models in tumor growth, invasion and pattern formation are investigated.

•

Computational algorithms for detecting such structures are proposed.

•

Complex systems applications to cancer dynamics.

Cancers are complex, adaptive ecosystems. They remain the leading cause of disease-related death among children in North America. As we approach computational oncology and Deep Learning Healthcare, our mathematical models of cancer dynamics must be revised. Recent findings support the perspective that cancer-microenvironment interactions may consist of chaotic gene expressions and turbulent protein flows during pattern formation. As such, cancer pattern formation, protein-folding and metastatic invasion are discussed herein as processes driven by chemical turbulence within the framework of complex systems theory. To conclude, cancer stem cells are presented as strange attractors of the Waddington landscape.

Transfer learning for nonlinear dynamics and its application to fluid turbulence

We introduce transfer learning for nonlinear dynamics, which enables efficient predictions of chaotic dynamics by utilizing a small amount of data. For the Lorenz chaos, by optimizing the transfer rate, we accomplish more accurate inference than the conventional method by an order of magnitude. Moreover, a surprisingly small amount of learning is enough to infer the energy dissipation rate of the Navier-Stokes turbulence because we can, thanks to the small-scale universality of turbulence, transfer a large amount of the knowledge learned from turbulence data at lower Reynolds number.

Predicting phase and sensing phase coherence in chaotic systems with machine learning

Recent interest in exploiting machine learning for model-free prediction of chaotic systems focused on the time evolution of the dynamical variables of the system as a whole, which include both amplitude and phase. In particular, in the framework based on reservoir computing, the prediction horizon as determined by the largest Lyapunov exponent is often short, typically about five or six Lyapunov times that contain approximately equal number of oscillation cycles of the system. There are situations in the real world where the phase information is important, such as the ups and downs of species populations in ecology, the polarity of a voltage variable in an electronic circuit, and the concentration of certain chemical above or below the average. Using classic chaotic oscillators and a chaotic food-web system from ecology as examples, we demonstrate that reservoir computing can be exploited for long-term prediction of the phase of chaotic oscillators. The typical prediction horizon can be orders of magnitude longer than that with predicting the entire variable, for which we provide a physical understanding. We also demonstrate that a properly designed reservoir computing machine can reliably sense phase synchronization between a pair of coupled chaotic oscillators with implications to the design of the parallel reservoir scheme for predicting large chaotic systems.