The reservoir's perspective on generalized synchronization

Reservoir-computing based associative memory and itinerancy for complex dynamical attractors

Traditional neural network models of associative memories were used to store and retrieve static patterns. We develop reservoir-computing based memories for complex dynamical attractors, under two common recalling scenarios in neuropsychology: location-addressable with an index channel and content-addressable without such a channel. We demonstrate that, for location-addressable retrieval, a single reservoir computing machine can memorize a large number of periodic and chaotic attractors, each retrievable with a specific index value. We articulate control strategies to achieve successful switching among the attractors, unveil the mechanism behind failed switching, and uncover various scaling behaviors between the number of stored attractors and the reservoir network size. For content-addressable retrieval, we exploit multistability with cue signals, where the stored attractors coexist in the high-dimensional phase space of the reservoir network. As the length of the cue signal increases through a critical value, a high success rate can be achieved. The work provides foundational insights into developing long-term memories and itinerancy for complex dynamical patterns.

Attractor reconstruction with reservoir computers: The effect of the reservoir's conditional Lyapunov exponents on faithful attractor reconstruction

Reservoir computing is a machine learning framework that has been shown to be able to replicate the chaotic attractor, including the fractal dimension and the entire Lyapunov spectrum, of the dynamical system on which it is trained. We quantitatively relate the generalized synchronization dynamics of a driven reservoir during the training stage to the performance of the trained reservoir computer at the attractor reconstruction task. We show that, in order to obtain successful attractor reconstruction and Lyapunov spectrum estimation, the maximal conditional Lyapunov exponent of the driven reservoir must be significantly more negative than the most negative Lyapunov exponent of the target system. We also find that the maximal conditional Lyapunov exponent of the reservoir depends strongly on the spectral radius of the reservoir adjacency matrix; therefore, for attractor reconstruction and Lyapunov spectrum estimation, small spectral radius reservoir computers perform better in general. Our arguments are supported by numerical examples on well-known chaotic systems.

Reducing echo state network size with controllability matrices

Echo state networks are a fast training variant of recurrent neural networks excelling at approximating nonlinear dynamical systems and time series prediction. These machine learning models act as nonlinear fading memory filters. While these models benefit from quick training and low complexity, computation demands from a large reservoir matrix are a bottleneck. Using control theory, a reduced size replacement reservoir matrix is found. Starting from a large, task-effective reservoir matrix, we form a controllability matrix whose rank indicates the active sub-manifold and candidate replacement reservoir size. Resulting time speed-ups and reduced memory usage come with minimal error increase to chaotic climate reconstruction or short term prediction. Experiments are performed on simple time series signals and the Lorenz-1963 and Mackey-Glass complex chaotic signals. Observing low error models shows variation of active rank and memory along a sequence of predictions.

Role of assortativity in predicting burst synchronization using echo state network

In this study, we use a reservoir computing based echo state network (ESN) to predict the collective burst synchronization of neurons. Specifically, we investigate the ability of ESN in predicting the burst synchronization of an ensemble of Rulkov neurons placed on a scale-free network. We have shown that a limited number of nodal dynamics used as input in the machine can capture the real trend of burst synchronization in this network. Further, we investigate the proper selection of nodal inputs of degree-degree (positive and negative) correlated networks. We show that for a disassortative network, selection of different input nodes based on degree has no significant role in the machine's prediction. However, in the case of assortative network, training the machine with the information (i.e., time series) of low degree nodes gives better results in predicting the burst synchronization. The results are found to be consistent with the investigation carried out with a continuous time Hindmarsh-Rose neuron model. Furthermore, the role of hyperparameters like spectral radius and leaking parameter of ESN on the prediction process has been examined. Finally, we explain the underlying mechanism responsible for observing these differences in the prediction in a degree correlated network.

Robust forecasting using predictive generalized synchronization in reservoir computing

Reservoir computers (RCs) are a class of recurrent neural networks (RNNs) that can be used for forecasting the future of observed time series data. As with all RNNs, selecting the hyperparameters in the network to yield excellent forecasting presents a challenge when training on new inputs. We analyze a method based on predictive generalized synchronization (PGS) that gives direction in designing and evaluating the architecture and hyperparameters of an RC. To determine the occurrences of PGS, we rely on the auxiliary method to provide a computationally efficient pre-training test that guides hyperparameter selection. We provide a metric for evaluating the RC using the reproduction of the input system's Lyapunov exponents that demonstrates robustness in prediction.

Optimized ensemble deep learning framework for scalable forecasting of dynamics containing extreme events

The remarkable flexibility and adaptability of both deep learning models and ensemble methods have led to the proliferation for their application in understanding many physical phenomena. Traditionally, these two techniques have largely been treated as independent methodologies in practical applications. This study develops an optimized ensemble deep learning framework wherein these two machine learning techniques are jointly used to achieve synergistic improvements in model accuracy, stability, scalability, and reproducibility, prompting a new wave of applications in the forecasting of dynamics. Unpredictability is considered one of the key features of chaotic dynamics; therefore, forecasting such dynamics of nonlinear systems is a relevant issue in the scientific community. It becomes more challenging when the prediction of extreme events is the focus issue for us. In this circumstance, the proposed optimized ensemble deep learning (OEDL) model based on a best convex combination of feed-forward neural networks, reservoir computing, and long short-term memory can play a key role in advancing predictions of dynamics consisting of extreme events. The combined framework can generate the best out-of-sample performance than the individual deep learners and standard ensemble framework for both numerically simulated and real-world data sets. We exhibit the outstanding performance of the OEDL framework for forecasting extreme events generated from a Liénard-type system, prediction of COVID-19 cases in Brazil, dengue cases in San Juan, and sea surface temperature in the Niño 3.4 region.

Reservoir computing on epidemic spreading: A case study on COVID-19 cases

A reservoir computing based echo state network (ESN) is used here for the purpose of predicting the spread of a disease. The current infection trends of a disease in some targeted locations are efficiently captured by the ESN when it is fed with the infection data for other locations. The performance of the ESN is first tested with synthetic data generated by numerical simulations of independent uncoupled patches, each governed by the classical susceptible-infected-recovery model for a choice of distributed infection parameters. From a large pool of synthetic data, the ESN predicts the current trend of infection in 5% patches by exploiting the uncorrelated infection trend of 95% patches. The prediction remains consistent for most of the patches for approximately 4 to 5 weeks. The machine's performance is further tested with real data on the current COVID-19 pandemic collected for different countries. We show that our proposed scheme is able to predict the trend of the disease for up to 3 weeks for some targeted locations. An important point is that no detailed information on the epidemiological rate parameters is needed; the success of the machine rather depends on the history of the disease progress represented by the time-evolving data sets of a large number of locations. Finally, we apply a modified version of our proposed scheme for the purpose of future forecasting.

Learn to synchronize, synchronize to learn

In recent years, the artificial intelligence community has seen a continuous interest in research aimed at investigating dynamical aspects of both training procedures and machine learning models. Of particular interest among recurrent neural networks, we have the Reservoir Computing (RC) paradigm characterized by conceptual simplicity and a fast training scheme. Yet, the guiding principles under which RC operates are only partially understood. In this work, we analyze the role played by Generalized Synchronization (GS) when training a RC to solve a generic task. In particular, we show how GS allows the reservoir to correctly encode the system generating the input signal into its dynamics. We also discuss necessary and sufficient conditions for the learning to be feasible in this approach. Moreover, we explore the role that ergodicity plays in this process, showing how its presence allows the learning outcome to apply to multiple input trajectories. Finally, we show that satisfaction of the GS can be measured by means of the mutual false nearest neighbors index, which makes effective to practitioners theoretical derivations.

Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem

This paper shows that a large class of fading memory state-space systems driven by discrete-time observations of dynamical systems defined on compact manifolds always yields continuously differentiable synchronizations. This general result provides a powerful tool for the representation, reconstruction, and forecasting of chaotic attractors. It also improves previous statements in the literature for differentiable generalized synchronizations, whose existence was so far guaranteed for a restricted family of systems and was detected using Hölder exponent-based criteria.

Optimizing Reservoir Computers for Signal Classification

Reservoir computers are a type of recurrent neural network for which the network connections are not changed. To train the reservoir computer, a set of output signals from the network are fit to a training signal by a linear fit. As a result, training of a reservoir computer is fast, and reservoir computers may be built from analog hardware, resulting in high speed and low power consumption. To get the best performance from a reservoir computer, the hyperparameters of the reservoir computer must be optimized. In signal classification problems, parameter optimization may be computationally difficult; it is necessary to compare many realizations of the test signals to get good statistics on the classification probability. In this work, it is shown in both a spiking reservoir computer and a reservoir computer using continuous variables that the optimum classification performance occurs for the hyperparameters that maximize the entropy of the reservoir computer. Optimizing for entropy only requires a single realization of each signal to be classified, making the process much faster to compute.

Low dimensional manifolds in reservoir computers

A reservoir computer is a complex dynamical system, often created by coupling nonlinear nodes in a network. The nodes are all driven by a common driving signal. Reservoir computers can contain hundreds to thousands of nodes, resulting in a high dimensional dynamical system, but the reservoir computer variables evolve on a lower dimensional manifold in this high dimensional space. This paper describes how this manifold dimension depends on the parameters of the reservoir computer, and how the manifold dimension is related to the performance of the reservoir computer at a signal estimation task. It is demonstrated that increasing the coupling between nodes while controlling the largest Lyapunov exponent of the reservoir computer can optimize the reservoir computer performance. It is also noted that the sparsity of the reservoir computer network does not have any influence on performance.

Transfer learning of chaotic systems

Can a neural network trained by the time series of system A be used to predict the evolution of system B? This problem, knowing as transfer learning in a broad sense, is of great importance in machine learning and data mining yet has not been addressed for chaotic systems. Here, we investigate transfer learning of chaotic systems from the perspective of synchronization-based state inference, in which a reservoir computer trained by chaotic system A is used to infer the unmeasured variables of chaotic system B, while A is different from B in either parameter or dynamics. It is found that if systems A and B are different in parameter, the reservoir computer can be well synchronized to system B. However, if systems A and B are different in dynamics, the reservoir computer fails to synchronize with system B in general. Knowledge transfer along a chain of coupled reservoir computers is also studied, and it is found that, although the reservoir computers are trained by different systems, the unmeasured variables of the driving system can be successfully inferred by the remote reservoir computer. Finally, by an experiment of chaotic pendulum, we demonstrate that the knowledge learned from the modeling system can be transferred and used to predict the evolution of the experimental system.

Do reservoir computers work best at the edge of chaos?

It has been demonstrated that cellular automata had the highest computational capacity at the edge of chaos [N. H. Packard, in Dynamic Patterns in Complex Systems, edited by J. A. S. Kelso, A. J. Mandell, and M. F. Shlesinger (World Scientific, Singapore, 1988), pp. 293-301; C. G. Langton, Physica D 42(1), 12-37 (1990); J. P. Crutchfield and K. Young, in Complexity, Entropy, and the Physics of Information, edited by W. H. Zurek (Addison-Wesley, Redwood City, CA, 1990), pp. 223-269], the parameter at which their behavior transitioned from ordered to chaotic. This same concept has been applied to reservoir computers; a number of researchers have stated that the highest computational capacity for a reservoir computer is at the edge of chaos, although others have suggested that this rule is not universally true. Because many reservoir computers do not show chaotic behavior but merely become unstable, it is felt that a more accurate term for this instability transition is the "edge of stability." Here, I find two examples where the computational capacity of a reservoir computer decreases as the edge of stability is approached: in one case because generalized synchronization breaks down and in the other case because the reservoir computer is a poor match to the problem being solved. The edge of stability as an optimal operating point for a reservoir computer is not in general true, although it may be true in some cases.