Consistency in echo-state networks

Impact of time-history terms on reservoir dynamics and prediction accuracy in echo state networks

The echo state network (ESN) is an excellent machine learning model for processing time-series data. This model, utilising the response of a recurrent neural network, called a reservoir, to input signals, achieves high training efficiency. Introducing time-history terms into the neuron model of the reservoir is known to improve the time-series prediction performance of ESN, yet the reasons for this improvement have not been quantitatively explained in terms of reservoir dynamics characteristics. Therefore, we hypothesised that the performance enhancement brought about by time-history terms could be explained by delay capacity, a recently proposed metric for assessing the memory performance of reservoirs. To test this hypothesis, we conducted comparative experiments using ESN models with time-history terms, namely leaky integrator ESNs (LI-ESN) and chaotic echo state networks (ChESN). The results suggest that compared with ESNs without time-history terms, the reservoir dynamics of LI-ESN and ChESN can maintain diversity and stability while possessing higher delay capacity, leading to their superior performance. Explaining ESN performance through dynamical metrics are crucial for evaluating the numerous ESN architectures recently proposed from a general perspective and for the development of more sophisticated architectures, and this study contributes to such efforts.

Attractor reconstruction with reservoir computers: The effect of the reservoir's conditional Lyapunov exponents on faithful attractor reconstruction

Reservoir computing is a machine learning framework that has been shown to be able to replicate the chaotic attractor, including the fractal dimension and the entire Lyapunov spectrum, of the dynamical system on which it is trained. We quantitatively relate the generalized synchronization dynamics of a driven reservoir during the training stage to the performance of the trained reservoir computer at the attractor reconstruction task. We show that, in order to obtain successful attractor reconstruction and Lyapunov spectrum estimation, the maximal conditional Lyapunov exponent of the driven reservoir must be significantly more negative than the most negative Lyapunov exponent of the target system. We also find that the maximal conditional Lyapunov exponent of the reservoir depends strongly on the spectral radius of the reservoir adjacency matrix; therefore, for attractor reconstruction and Lyapunov spectrum estimation, small spectral radius reservoir computers perform better in general. Our arguments are supported by numerical examples on well-known chaotic systems.

Global forecasts in reservoir computers

A reservoir computer is a machine learning model that can be used to predict the future state(s) of time-dependent processes, e.g., dynamical systems. In practice, data in the form of an input-signal are fed into the reservoir. The trained reservoir is then used to predict the future state of this signal. We develop a new method for not only predicting the future dynamics of the input-signal but also the future dynamics starting at an arbitrary initial condition of a system. The systems we consider are the Lorenz, Rossler, and Thomas systems restricted to their attractors. This method, which creates a global forecast, still uses only a single input-signal to train the reservoir but breaks the signal into many smaller windowed signals. We examine how well this windowed method is able to forecast the dynamics of a system starting at an arbitrary point on a system's attractor and compare this to the standard method without windows. We find that the standard method has almost no ability to forecast anything but the original input-signal while the windowed method can capture the dynamics starting at most points on an attractor with significant accuracy.

Harnessing synthetic active particles for physical reservoir computing

The processing of information is an indispensable property of living systems realized by networks of active processes with enormous complexity. They have inspired many variants of modern machine learning, one of them being reservoir computing, in which stimulating a network of nodes with fading memory enables computations and complex predictions. Reservoirs are implemented on computer hardware, but also on unconventional physical substrates such as mechanical oscillators, spins, or bacteria often summarized as physical reservoir computing. Here we demonstrate physical reservoir computing with a synthetic active microparticle system that self-organizes from an active and passive component into inherently noisy nonlinear dynamical units. The self-organization and dynamical response of the unit are the results of a delayed propulsion of the microswimmer to a passive target. A reservoir of such units with a self-coupling via the delayed response can perform predictive tasks despite the strong noise resulting from the Brownian motion of the microswimmers. To achieve efficient noise suppression, we introduce a special architecture that uses historical reservoir states for output. Our results pave the way for the study of information processing in synthetic self-organized active particle systems.

Exploiting oscillatory dynamics of delay systems for reservoir computing

Nonlinear dynamical systems exhibiting inherent memory can process temporal information by exploiting their responses to input drives. Reservoir computing is a prominent approach to leverage this ability for time-series forecasting. The computational capabilities of analog computing systems often depend on both the dynamical regime of the system and the input drive. Most studies have focused on systems exhibiting a stable fixed-point solution in the absence of input. Here, we go beyond that limitation, investigating the computational capabilities of a paradigmatic delay system in three different dynamical regimes. The system we chose has an Ikeda-type nonlinearity and exhibits fixed point, bistable, and limit-cycle dynamics in the absence of input. When driving the system, new input-driven dynamics emerge from the autonomous ones featuring characteristic properties. Here, we show that it is feasible to attain consistent responses across all three regimes, which is an essential prerequisite for the successful execution of the tasks. Furthermore, we demonstrate that we can exploit all three regimes in two time-series forecasting tasks, showcasing the versatility of this paradigmatic delay system in an analog computing context. In all tasks, the lowest prediction errors were obtained in the regime that exhibits limit-cycle dynamics in the undriven reservoir. To gain further insights, we analyzed the diverse time-distributed node responses generated in the three regimes of the undriven system. An increase in the effective dimensionality of the reservoir response is shown to affect the prediction error, as also fine-tuning of the distribution of nonlinear responses. Finally, we demonstrate that a trade-off between prediction accuracy and computational speed is possible in our continuous delay systems. Our results not only provide valuable insights into the computational capabilities of complex dynamical systems but also open a new perspective on enhancing the potential of analog computing systems implemented on various hardware platforms.

Time shifts to reduce the size of reservoir computers

A reservoir computer is a type of dynamical system arranged to do computation. Typically, a reservoir computer is constructed by connecting a large number of nonlinear nodes in a network that includes recurrent connections. In order to achieve accurate results, the reservoir usually contains hundreds to thousands of nodes. This high dimensionality makes it difficult to analyze the reservoir computer using tools from the dynamical systems theory. Additionally, the need to create and connect large numbers of nonlinear nodes makes it difficult to design and build analog reservoir computers that can be faster and consume less power than digital reservoir computers. We demonstrate here that a reservoir computer may be divided into two parts: a small set of nonlinear nodes (the reservoir) and a separate set of time-shifted reservoir output signals. The time-shifted output signals serve to increase the rank and memory of the reservoir computer, and the set of nonlinear nodes may create an embedding of the input dynamical system. We use this time-shifting technique to obtain excellent performance from an opto-electronic delay-based reservoir computer with only a small number of virtual nodes. Because only a few nonlinear nodes are required, construction of a reservoir computer becomes much easier, and delay-based reservoir computers can operate at much higher speeds.

Role of assortativity in predicting burst synchronization using echo state network

In this study, we use a reservoir computing based echo state network (ESN) to predict the collective burst synchronization of neurons. Specifically, we investigate the ability of ESN in predicting the burst synchronization of an ensemble of Rulkov neurons placed on a scale-free network. We have shown that a limited number of nodal dynamics used as input in the machine can capture the real trend of burst synchronization in this network. Further, we investigate the proper selection of nodal inputs of degree-degree (positive and negative) correlated networks. We show that for a disassortative network, selection of different input nodes based on degree has no significant role in the machine's prediction. However, in the case of assortative network, training the machine with the information (i.e., time series) of low degree nodes gives better results in predicting the burst synchronization. The results are found to be consistent with the investigation carried out with a continuous time Hindmarsh-Rose neuron model. Furthermore, the role of hyperparameters like spectral radius and leaking parameter of ESN on the prediction process has been examined. Finally, we explain the underlying mechanism responsible for observing these differences in the prediction in a degree correlated network.

Consistency Hierarchy of Reservoir Computers

We study the propagation and distribution of information-carrying signals injected in dynamical systems serving as reservoir computers. Through different combinations of repeated input signals, a multivariate correlation analysis reveals measures known as the consistency spectrum and consistency capacity. These are high-dimensional portraits of the nonlinear functional dependence between input and reservoir state. For multiple inputs, a hierarchy of capacities characterizes the interference of signals from each source. For an individual input, the time-resolved capacities form a profile of the reservoir's nonlinear fading memory. We illustrate this methodology for a range of echo state networks.

Reservoir time series analysis: Using the response of complex dynamical systems as a universal indicator of change

We present the idea of reservoir time series analysis (RTSA), a method by which the state space representation generated by a reservoir computing (RC) model can be used for time series analysis. We discuss the motivation for this with reference to the characteristics of RC and present three ad hoc methods for generating representative features from the reservoir state space. We then develop and implement a hypothesis test to assess the capacity of these features to distinguish signals from systems with varying parameters. In comparison to a number of benchmark approaches (statistical, Fourier, phase space, and recurrence analysis), we are able to show significant, generalized accuracy across the proposed RTSA features that surpasses the benchmark methods. Finally, we briefly present an application for bearing fault distinction to motivate the use of RTSA in application.

Optimizing memory in reservoir computers

A reservoir computer is a way of using a high dimensional dynamical system for computation. One way to construct a reservoir computer is by connecting a set of nonlinear nodes into a network. Because the network creates feedback between nodes, the reservoir computer has memory. If the reservoir computer is to respond to an input signal in a consistent way (a necessary condition for computation), the memory must be fading; that is, the influence of the initial conditions fades over time. How long this memory lasts is important for determining how well the reservoir computer can solve a particular problem. In this paper, I describe ways to vary the length of the fading memory in reservoir computers. Tuning the memory can be important to achieve optimal results in some problems; too much or too little memory degrades the accuracy of the computation.

Learn to synchronize, synchronize to learn

In recent years, the artificial intelligence community has seen a continuous interest in research aimed at investigating dynamical aspects of both training procedures and machine learning models. Of particular interest among recurrent neural networks, we have the Reservoir Computing (RC) paradigm characterized by conceptual simplicity and a fast training scheme. Yet, the guiding principles under which RC operates are only partially understood. In this work, we analyze the role played by Generalized Synchronization (GS) when training a RC to solve a generic task. In particular, we show how GS allows the reservoir to correctly encode the system generating the input signal into its dynamics. We also discuss necessary and sufficient conditions for the learning to be feasible in this approach. Moreover, we explore the role that ergodicity plays in this process, showing how its presence allows the learning outcome to apply to multiple input trajectories. Finally, we show that satisfaction of the GS can be measured by means of the mutual false nearest neighbors index, which makes effective to practitioners theoretical derivations.

Maximum Information Exploitation Using Broad Learning System for Large-Scale Chaotic Time-Series Prediction

How to make full use of the evolution information of chaotic systems for time-series prediction is a difficult issue in dynamical system modeling. In this article, we propose a maximum information exploitation broad learning system (MIE-BLS) for extreme information utilization of large-scale chaotic time-series modeling. An improved leaky integrator dynamical reservoir is introduced in order to capture the linear information of chaotic systems effectively. It can not only capture the information of the current state but also achieve the compromise with historical states in the dynamical system. Furthermore, the feature is mapped to the enhancement layer by nonlinear random mapping to exploit nonlinear information. The cascading mechanism promotes the information propagation and achieves feature reactivation in dynamical modeling. Discussions about maximum information exploration and the comparisons with ResNet, DenseNet, and HighwayNet are presented in this article. Simulation results on four large-scale data sets illustrate that MIE-BLS could achieve better performance of information exploration in large-scale dynamical system modeling.

Parameter extraction with reservoir computing: Nonlinear time series analysis and application to industrial maintenance

We study the task of determining parameters of dynamical systems from their time series using variations of reservoir computing. Averages of reservoir activations yield a static set of random features that allows us to separate different parameter values. We study such random feature models in the time and frequency domain. For the Lorenz and Rössler systems throughout stable and chaotic regimes, we achieve accurate and robust parameter extraction. For vibration data of centrifugal pumps, we find a significant ability to recover the operating regime. While the time domain models achieve higher performance for the numerical systems, the frequency domain models are superior in the application context.

Reservoir computing with swarms

We study swarms as dynamical systems for reservoir computing (RC). By example of a modified Reynolds boids model, the specific symmetries and dynamical properties of a swarm are explored with respect to a nonlinear time-series prediction task. Specifically, we seek to extract meaningful information about a predator-like driving signal from the swarm's response to that signal. We find that the naïve implementation of a swarm for computation is very inefficient, as permutation symmetry of the individual agents reduces the computational capacity. To circumvent this, we distinguish between the computational substrate of the swarm and a separate observation layer, in which the swarm's response is measured for use in the task. We demonstrate the implementation of a radial basis-localized observation layer for this task. The behavior of the swarm is characterized by order parameters and measures of consistency and related to the performance of the swarm as a reservoir. The relationship between RC performance and swarm behavior demonstrates that optimal computational properties are obtained near a phase transition regime.

Revisiting the memory capacity in reservoir computing of directed acyclic network

Reservoir computing (RC) is an attractive area of research by virtue of its potential for hardware implementation and low training cost. An intriguing research direction in this field is to interpret the underlying dynamics of an RC model by analyzing its short-term memory property, which can be quantified by the global index: memory capacity (MC). In this paper, the global MC of the RC whose reservoir network is specified as a directed acyclic network (DAN) is examined, and first we give that its global MC is theoretically bounded by the length of the longest path of the reservoir DAN. Since the global MC is technically influenced by the model hyperparameters, the dependency of the MC on the hyperparameters of this RC is then explored in detail. In the further study, we employ the improved conventional network embedding method (i.e., struc2vec) to mine the underlying memory community in the reservoir DAN, which can be regarded as the cluster of reservoir nodes with the same memory profile. Experimental results demonstrate that such a memory community structure can provide a concrete interpretation of the global MC of this RC. Finally, the clustered RC is proposed by exploiting the detected memory community structure of DAN, where its prediction performance is verified to be enhanced with lower training cost compared with other RC models on several chaotic time series benchmarks.

Transfer learning of chaotic systems

Can a neural network trained by the time series of system A be used to predict the evolution of system B? This problem, knowing as transfer learning in a broad sense, is of great importance in machine learning and data mining yet has not been addressed for chaotic systems. Here, we investigate transfer learning of chaotic systems from the perspective of synchronization-based state inference, in which a reservoir computer trained by chaotic system A is used to infer the unmeasured variables of chaotic system B, while A is different from B in either parameter or dynamics. It is found that if systems A and B are different in parameter, the reservoir computer can be well synchronized to system B. However, if systems A and B are different in dynamics, the reservoir computer fails to synchronize with system B in general. Knowledge transfer along a chain of coupled reservoir computers is also studied, and it is found that, although the reservoir computers are trained by different systems, the unmeasured variables of the driving system can be successfully inferred by the remote reservoir computer. Finally, by an experiment of chaotic pendulum, we demonstrate that the knowledge learned from the modeling system can be transferred and used to predict the evolution of the experimental system.

Path length statistics in reservoir computers

Because reservoir computers are high dimensional dynamical systems, designing a good reservoir computer is difficult. In many cases, the designer must search a large nonlinear parameter space, and each step of the search requires simulating the full reservoir computer. In this work, I show that a simple statistic based on the mean path length between nodes in the reservoir computer is correlated with better reservoir computer performance. The statistic predicts the diversity of signals produced by the reservoir computer, as measured by the covariance matrix of the reservoir computer. This statistic by itself is not sufficient to predict reservoir computer performance because not only must the reservoir computer produce a diverse set of signals, it must be well matched to the training signals. Nevertheless, this path length statistic allows the designer to eliminate some network configurations from consideration without having to actually simulate the reservoir computer, reducing the complexity of the design process.

Performance boost of time-delay reservoir computing by non-resonant clock cycle

The time-delay-based reservoir computing setup has seen tremendous success in both experiment and simulation. It allows for the construction of large neuromorphic computing systems with only few components. However, until now the interplay of the different timescales has not been investigated thoroughly. In this manuscript, we investigate the effects of a mismatch between the time-delay and the clock cycle for a general model. Typically, these two time scales are considered to be equal. Here we show that the case of equal or resonant time-delay and clock cycle could be actively detrimental and leads to an increase of the approximation error of the reservoir. In particular, we can show that non-resonant ratios of these time scales have maximal memory capacities. We achieve this by translating the periodically driven delay-dynamical system into an equivalent network. Networks that originate from a system with resonant delay-times and clock cycles fail to utilize all of their degrees of freedom, which causes the degradation of their performance.

Learned emergence in selfish collective motion

To understand the collective motion of many individuals, we often rely on agent-based models with rules that may be computationally complex and involved. For biologically inspired systems in particular, this raises questions about whether the imposed rules are necessarily an accurate reflection of what is being followed. The basic premise of updating one's state according to some underlying motivation is well suited to the realm of reservoir computing; however, entire swarms of individuals are yet to be tasked with learning movement in this framework. This work focuses on the specific case of many selfish individuals simultaneously optimizing their domains in a manner conducive to reducing their personal risk of predation. Using an echo state network and data generated from the agent-based model, we show that, with an appropriate representation of input and output states, this selfish movement can be learned. This suggests that a more sophisticated neural network, such as a brain, could also learn this behavior and provides an avenue to further the search for realistic movement rules in systems of autonomous individuals.

The reservoir's perspective on generalized synchronization

We employ reservoir computing for a reconstruction task in coupled chaotic systems, across a range of dynamical relationships including generalized synchronization. For a drive-response setup, a temporal representation of the synchronized state is discussed as an alternative to the known instantaneous form. The reservoir has access to both representations through its fading memory property, each with advantages in different dynamical regimes. We also extract signatures of the maximal conditional Lyapunov exponent in the performance of variations of the reservoir topology. Moreover, the reservoir model reproduces different levels of consistency where there is no synchronization. In a bidirectional coupling setup, high reconstruction accuracy is achieved despite poor observability and independent of generalized synchronization.

Network structure effects in reservoir computers

A reservoir computer is a complex nonlinear dynamical system that has been shown to be useful for solving certain problems, such as prediction of chaotic signals, speech recognition, or control of robotic systems. Typically, a reservoir computer is constructed by connecting a large number of nonlinear nodes in a network, driving the nodes with an input signal and using the node outputs to fit a training signal. In this work, we set up reservoirs where the edges (or connections) between all the network nodes are either +1 or 0 and proceed to alter the network structure by flipping some of these edges from +1 to -1. We use this simple network because it turns out to be easy to characterize; we may use the fraction of edges flipped as a measure of how much we have altered the network. In some cases, the network can be rearranged in a finite number of ways without changing its structure; these rearrangements are symmetries of the network, and the number of symmetries is also useful for characterizing the network. We find that changing the number of edges flipped in the network changes the rank of the covariance of a matrix consisting of the time series from the different nodes in the network and speculate that this rank is important for understanding the reservoir computer performance.