1
|
Shi L, Wang H, Wang S, Du R, Qu SX. Predicting nonsmooth chaotic dynamics by reservoir computing. Phys Rev E 2024; 109:014214. [PMID: 38366462 DOI: 10.1103/physreve.109.014214] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Accepted: 12/07/2023] [Indexed: 02/18/2024]
Abstract
Reservoir computing (RC) has been widely applied to predict the chaotic dynamics in many systems. Yet much broader areas related to nonsmooth dynamics have seldom been touched by the RC community which have great theoretical and practical importance. The generalization of RC to this kind of system is reported in this paper. The numerical work shows that the conventional RC with a hyperbolic tangent activation function is not able to predict the dynamics of nonsmooth systems very well, especially when reconstructing attractors (long-term prediction). A nonsmooth activation function with a piecewise nature is proposed. A kind of physics-informed RC scheme is established based on this activation function. The feasibility of this scheme has been proven by its successful application to the predictions of the short- and long-term (reconstructing chaotic attractor) dynamics of four nonsmooth systems with different complexity, including the tent map, piecewise linear map with a gap, both noninvertible and discontinuous compound circle maps, and Lozi map. The results show that RC with the new activation function is efficient and easy to run. It can make perfectly both short- and long-term predictions. The precision of reconstructing attractors depends on their complexity. This work reveals that, to make efficient predictions, the activation function of an RC approach should match the smooth or nonsmooth nature of the dynamical systems.
Collapse
Affiliation(s)
- Lufa Shi
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an 710119, China
| | - Hengtong Wang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an 710119, China
| | - Shengjun Wang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an 710119, China
| | - Ruhai Du
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an 710119, China
| | - Shi-Xian Qu
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an 710119, China
| |
Collapse
|
2
|
Xia J, Chu J, Leng S, Ma H. Reservoir computing decoupling memory-nonlinearity trade-off. CHAOS (WOODBURY, N.Y.) 2023; 33:113120. [PMID: 37967262 DOI: 10.1063/5.0156224] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 10/17/2023] [Indexed: 11/17/2023]
Abstract
Reservoir computing (RC), a variant recurrent neural network, has very compact architecture and ability to efficiently reconstruct nonlinear dynamics by combining both memory capacity and nonlinear transformations. However, in the standard RC framework, there is a trade-off between memory capacity and nonlinear mapping, which limits its ability to handle complex tasks with long-term dependencies. To overcome this limitation, this paper proposes a new RC framework called neural delayed reservoir computing (ND-RC) with a chain structure reservoir that can decouple the memory capacity and nonlinearity, allowing for independent tuning of them, respectively. The proposed ND-RC model offers a promising solution to the memory-nonlinearity trade-off problem in RC and provides a more flexible and effective approach for modeling complex nonlinear systems with long-term dependencies. The proposed ND-RC framework is validated with typical benchmark nonlinear systems and is particularly successful in reconstructing and predicting the Mackey-Glass system with high time delays. The memory-nonlinearity decoupling ability is further confirmed by several standard tests.
Collapse
Affiliation(s)
- Ji Xia
- School of Mathematical Sciences, Soochow University, Suzhou 215001, China
| | - Junyu Chu
- School of Mathematical Sciences, Soochow University, Suzhou 215001, China
| | - Siyang Leng
- Academy for Engineering and Technology and Research Institute of Intelligent Complex Systems, Fudan University, Shanghai 200433, China
| | - Huanfei Ma
- School of Mathematical Sciences, Soochow University, Suzhou 215001, China
| |
Collapse
|
3
|
Calvet E, Rouat J, Reulet B. Excitatory/inhibitory balance emerges as a key factor for RBN performance, overriding attractor dynamics. Front Comput Neurosci 2023; 17:1223258. [PMID: 37621962 PMCID: PMC10445160 DOI: 10.3389/fncom.2023.1223258] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 07/17/2023] [Indexed: 08/26/2023] Open
Abstract
Reservoir computing provides a time and cost-efficient alternative to traditional learning methods. Critical regimes, known as the "edge of chaos," have been found to optimize computational performance in binary neural networks. However, little attention has been devoted to studying reservoir-to-reservoir variability when investigating the link between connectivity, dynamics, and performance. As physical reservoir computers become more prevalent, developing a systematic approach to network design is crucial. In this article, we examine Random Boolean Networks (RBNs) and demonstrate that specific distribution parameters can lead to diverse dynamics near critical points. We identify distinct dynamical attractors and quantify their statistics, revealing that most reservoirs possess a dominant attractor. We then evaluate performance in two challenging tasks, memorization and prediction, and find that a positive excitatory balance produces a critical point with higher memory performance. In comparison, a negative inhibitory balance delivers another critical point with better prediction performance. Interestingly, we show that the intrinsic attractor dynamics have little influence on performance in either case.
Collapse
Affiliation(s)
- Emmanuel Calvet
- Neurosciences Computationelles et Traitement Intelligent des Signaux (NECOTIS), Faculté de Génie, Génie Électrique et Génie Informatique (GEGI), Université de Sherbrooke, Sherbrooke, QC, Canada
| | - Jean Rouat
- Neurosciences Computationelles et Traitement Intelligent des Signaux (NECOTIS), Faculté de Génie, Génie Électrique et Génie Informatique (GEGI), Université de Sherbrooke, Sherbrooke, QC, Canada
| | - Bertrand Reulet
- Département de Physique, Faculté des Sciences, Institut Quantique, Université de Sherbrooke, Sherbrooke, QC, Canada
| |
Collapse
|
4
|
Nathe C, Pappu C, Mecholsky NA, Hart J, Carroll T, Sorrentino F. Reservoir computing with noise. CHAOS (WOODBURY, N.Y.) 2023; 33:041101. [PMID: 37097967 PMCID: PMC10132850 DOI: 10.1063/5.0130278] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/10/2022] [Accepted: 03/09/2023] [Indexed: 06/19/2023]
Abstract
This paper investigates in detail the effects of measurement noise on the performance of reservoir computing. We focus on an application in which reservoir computers are used to learn the relationship between different state variables of a chaotic system. We recognize that noise can affect the training and testing phases differently. We find that the best performance of the reservoir is achieved when the strength of the noise that affects the input signal in the training phase equals the strength of the noise that affects the input signal in the testing phase. For all the cases we examined, we found that a good remedy to noise is to low-pass filter the input and the training/testing signals; this typically preserves the performance of the reservoir, while reducing the undesired effects of noise.
Collapse
Affiliation(s)
- Chad Nathe
- Mechanical Engineering Department, University of New Mexico, Albuquerque, New Mexico 87131, USA
| | - Chandra Pappu
- Electrical, Computer and Biomedical Engineering Department, Union College, Schenectady, New York 12309, USA
| | - Nicholas A. Mecholsky
- Department of Physics and Vitreous State Laboratory, The Catholic University of America, Washington, DC 20064, USA
| | - Joe Hart
- US Naval Research Laboratory, Washington, DC 20375, USA
| | | | - Francesco Sorrentino
- Mechanical Engineering Department, University of New Mexico, Albuquerque, New Mexico 87131, USA
| |
Collapse
|
5
|
Patel D, Ott E. Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. CHAOS (WOODBURY, N.Y.) 2023; 33:023143. [PMID: 36859201 DOI: 10.1063/5.0131787] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/24/2022] [Accepted: 01/31/2023] [Indexed: 06/18/2023]
Abstract
The ability of machine learning (ML) models to "extrapolate" to situations outside of the range spanned by their training data is crucial for predicting the long-term behavior of non-stationary dynamical systems (e.g., prediction of terrestrial climate change), since the future trajectories of such systems may (perhaps after crossing a tipping point) explore regions of state space which were not explored in past time-series measurements used as training data. We investigate the extent to which ML methods can yield useful results by extrapolation of such training data in the task of forecasting non-stationary dynamics, as well as conditions under which such methods fail. In general, we find that ML can be surprisingly effective even in situations that might appear to be extremely challenging, but do (as one would expect) fail when "too much" extrapolation is required. For the latter case, we show that good results can potentially be obtained by combining the ML approach with an available inaccurate conventional model based on scientific knowledge.
Collapse
Affiliation(s)
- Dhruvit Patel
- The Institute for Research in Electronics and Applied Physics, University of Maryland, College Park, Maryland 26742, USA
| | - Edward Ott
- The Institute for Research in Electronics and Applied Physics, University of Maryland, College Park, Maryland 26742, USA
| |
Collapse
|
6
|
Boriskov P, Velichko A, Shilovsky N, Belyaev M. Bifurcation and Entropy Analysis of a Chaotic Spike Oscillator Circuit Based on the S-Switch. ENTROPY (BASEL, SWITZERLAND) 2022; 24:1693. [PMID: 36421548 PMCID: PMC9689857 DOI: 10.3390/e24111693] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/28/2022] [Revised: 11/13/2022] [Accepted: 11/17/2022] [Indexed: 06/16/2023]
Abstract
This paper presents a model and experimental study of a chaotic spike oscillator based on a leaky integrate-and-fire (LIF) neuron, which has a switching element with an S-type current-voltage characteristic (S-switch). The oscillator generates spikes of the S-switch in the form of chaotic pulse position modulation driven by the feedback with rate coding instability of LIF neuron. The oscillator model with piecewise function of the S-switch has resistive feedback using a second order filter. The oscillator circuit is built on four operational amplifiers and two field-effect transistors (MOSFETs) that form an S-switch based on a Schmitt trigger, an active RC filter and a matching amplifier. We investigate the bifurcation diagrams of the model and the circuit and calculate the entropy of oscillations. For the analog circuit, the "regular oscillation-chaos" transition is analysed in a series of tests initiated by a step voltage in the matching amplifier. Entropy values are used to estimate the average time for the transition of oscillations to chaos and the degree of signal correlation of the transition mode of different tests. Study results can be applied in various reservoir computing applications, for example, in choosing and configuring the LogNNet network reservoir circuits.
Collapse
|
7
|
Gauthier DJ, Fischer I, Röhm A. Learning unseen coexisting attractors. CHAOS (WOODBURY, N.Y.) 2022; 32:113107. [PMID: 36456323 DOI: 10.1063/5.0116784] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/29/2022] [Accepted: 10/10/2022] [Indexed: 06/17/2023]
Abstract
Reservoir computing is a machine learning approach that can generate a surrogate model of a dynamical system. It can learn the underlying dynamical system using fewer trainable parameters and, hence, smaller training data sets than competing approaches. Recently, a simpler formulation, known as next-generation reservoir computing, removed many algorithm metaparameters and identified a well-performing traditional reservoir computer, thus simplifying training even further. Here, we study a particularly challenging problem of learning a dynamical system that has both disparate time scales and multiple co-existing dynamical states (attractors). We compare the next-generation and traditional reservoir computer using metrics quantifying the geometry of the ground-truth and forecasted attractors. For the studied four-dimensional system, the next-generation reservoir computing approach uses ∼ 1.7 × less training data, requires 10 × shorter "warmup" time, has fewer metaparameters, and has an ∼ 100 × higher accuracy in predicting the co-existing attractor characteristics in comparison to a traditional reservoir computer. Furthermore, we demonstrate that it predicts the basin of attraction with high accuracy. This work lends further support to the superior learning ability of this new machine learning algorithm for dynamical systems.
Collapse
Affiliation(s)
- Daniel J Gauthier
- Department of Physics, The Ohio State University, 191 West Woodruff Ave., Columbus, Ohio 43210, USA
| | - Ingo Fischer
- Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (CSIC-UIB), Campus Universitat Illes Balears, E-07122 Palma de Mallorca, Spain
| | - André Röhm
- Department of Information Physics and Computing, Graduate School of Information Science and Technology, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan
| |
Collapse
|
8
|
Zhong D, Zhao K, Xu Z, Hu Y, Deng W, Hou P, Zhang J, Zhang J. Deep optical reservoir computing and chaotic synchronization predictions based on the cascade coupled optically pumped spin-VCSELs. OPTICS EXPRESS 2022; 30:36209-36233. [PMID: 36258555 DOI: 10.1364/oe.464804] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/26/2022] [Accepted: 09/09/2022] [Indexed: 06/16/2023]
Abstract
In this work, we utilize two cascade coupling modes (unidirectional coupling and bidirectional coupling) to construct a four-layer deep reservoir computing (RC) system based on the cascade coupled optically-pumped spin-VCSEL. In such a system, there are double sub-reservoirs in each layer, which are formed by the chaotic x-PC and y-PC emitted by the reservoir spin-VCSEL in each layer. Under these two coupling modes, the chaotic x-PC and y-PC emitted by the driving optically-pumped spin-VCSEL (D-Spin-VCSEL), as two learning targets, are predicted by utilizing the four-layer reservoirs. In different parameter spaces, it is further explored that the outputs of the double sub-reservoirs in each layer are respectively synchronized with the chaotic x-PC and y-PC emitted by the D-Spin-VCSEL. The memory capacities (MCs) for the double sub-reservoirs in each layer are even further investigated. The results show that under two coupling modes, the predictions of the double sub-reservoirs with higher-layer for these two targets have smaller errors, denoting that the higher-layer double sub-reservoirs possess better predictive learning ability. Under the same system parameters, the outputs of the higher-layer dual parallel reservoirs are better synchronized with two chaotic PCs emitted by the D-Spin-VCSEL, respectively. The larger MCs can also be obtained by the higher-layer double reservoirs. In particular, compared with the four-layer reservoir computing system under unidirectional coupling, the four-layer reservoir computing system under bidirectional coupling shows better predictive ability in the same parameter space. The chaotic synchronizations predicted by each layer double sub-reservoirs under bidirectional coupling can be obtained higher qualities than those under unidirectional coupling. By the optimization of the system parameters, the outputs of the fourth-layer double sub-reservoirs are almost completely synchronized with the chaotic x-PC and y-PC emitted by the D-Spin-VCSEL, respectively, due to their correlation coefficient used to measure synchronization quality can be obtained as 0.99. These results have potential applications in chaotic computation, chaotic secure communication and accurate prediction of time series.
Collapse
|
9
|
Platt JA, Penny SG, Smith TA, Chen TC, Abarbanel HDI. A systematic exploration of reservoir computing for forecasting complex spatiotemporal dynamics. Neural Netw 2022; 153:530-552. [PMID: 35839598 DOI: 10.1016/j.neunet.2022.06.025] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2022] [Revised: 06/14/2022] [Accepted: 06/20/2022] [Indexed: 11/16/2022]
Abstract
A reservoir computer (RC) is a type of recurrent neural network architecture with demonstrated success in the prediction of spatiotemporally chaotic dynamical systems. A further advantage of RC is that it reproduces intrinsic dynamical quantities essential for its incorporation into numerical forecasting routines such as the ensemble Kalman filter-used in numerical weather prediction to compensate for sparse and noisy data. We explore here the architecture and design choices for a "best in class" RC for a number of characteristic dynamical systems. Our analysis points to the importance of large scale parameter optimization. We also note in particular the importance of including input bias in the RC design, which has a significant impact on the forecast skill of the trained RC model. In our tests, the use of a nonlinear readout operator does not affect the forecast time or the stability of the forecast. The effects of the reservoir dimension, spinup time, amount of training data, normalization, noise, and the RC time step are also investigated. Finally, we detail how our investigation leads to optimal design choices for a parallel RC scheme applied to the 40 dimensional spatiotemporally chaotic Lorenz 1996 dynamics.
Collapse
Affiliation(s)
- Jason A Platt
- Department of Physics, University of California San Diego, United States of America.
| | - Stephen G Penny
- Cooperative Institute for Research in Environmental Sciences, University of Colorado Boulder, Boulder, CO 80309, United States of America; Physical Sciences Laboratory, National Oceanic and Atmospheric Administration, Boulder, CO 80305, United States of America
| | - Timothy A Smith
- Cooperative Institute for Research in Environmental Sciences, University of Colorado Boulder, Boulder, CO 80309, United States of America; Physical Sciences Laboratory, National Oceanic and Atmospheric Administration, Boulder, CO 80305, United States of America
| | - Tse-Chun Chen
- Cooperative Institute for Research in Environmental Sciences, University of Colorado Boulder, Boulder, CO 80309, United States of America; Physical Sciences Laboratory, National Oceanic and Atmospheric Administration, Boulder, CO 80305, United States of America
| | - Henry D I Abarbanel
- Department of Physics, University of California San Diego, United States of America; Marine Physical Laboratory, Scripps Institution of Oceanography, University of California San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA
| |
Collapse
|
10
|
Carroll TL, Hart JD. Time shifts to reduce the size of reservoir computers. CHAOS (WOODBURY, N.Y.) 2022; 32:083122. [PMID: 36049918 DOI: 10.1063/5.0097850] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Accepted: 07/22/2022] [Indexed: 06/15/2023]
Abstract
A reservoir computer is a type of dynamical system arranged to do computation. Typically, a reservoir computer is constructed by connecting a large number of nonlinear nodes in a network that includes recurrent connections. In order to achieve accurate results, the reservoir usually contains hundreds to thousands of nodes. This high dimensionality makes it difficult to analyze the reservoir computer using tools from the dynamical systems theory. Additionally, the need to create and connect large numbers of nonlinear nodes makes it difficult to design and build analog reservoir computers that can be faster and consume less power than digital reservoir computers. We demonstrate here that a reservoir computer may be divided into two parts: a small set of nonlinear nodes (the reservoir) and a separate set of time-shifted reservoir output signals. The time-shifted output signals serve to increase the rank and memory of the reservoir computer, and the set of nonlinear nodes may create an embedding of the input dynamical system. We use this time-shifting technique to obtain excellent performance from an opto-electronic delay-based reservoir computer with only a small number of virtual nodes. Because only a few nonlinear nodes are required, construction of a reservoir computer becomes much easier, and delay-based reservoir computers can operate at much higher speeds.
Collapse
Affiliation(s)
| | - Joseph D Hart
- U.S. Naval Research Laboratory, Washington D.C. 20375, USA
| |
Collapse
|
11
|
Linot AJ, Graham MD. Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations. CHAOS (WOODBURY, N.Y.) 2022; 32:073110. [PMID: 35907719 DOI: 10.1063/5.0069536] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/31/2021] [Accepted: 06/15/2022] [Indexed: 06/15/2023]
Abstract
Dissipative partial differential equations that exhibit chaotic dynamics tend to evolve to attractors that exist on finite-dimensional manifolds. We present a data-driven reduced-order modeling method that capitalizes on this fact by finding a coordinate representation for this manifold and then a system of ordinary differential equations (ODEs) describing the dynamics in this coordinate system. The manifold coordinates are discovered using an undercomplete autoencoder-a neural network (NN) that reduces and then expands dimension. Then, the ODE, in these coordinates, is determined by a NN using the neural ODE framework. Both of these steps only require snapshots of data to learn a model, and the data can be widely and/or unevenly spaced. Time-derivative information is not needed. We apply this framework to the Kuramoto-Sivashinsky equation for domain sizes that exhibit chaotic dynamics with again estimated manifold dimensions ranging from 8 to 28. With this system, we find that dimension reduction improves performance relative to predictions in the ambient space, where artifacts arise. Then, with the low-dimensional model, we vary the training data spacing and find excellent short- and long-time statistical recreation of the true dynamics for widely spaced data (spacing of ∼ 0.7 Lyapunov times). We end by comparing performance with various degrees of dimension reduction and find a "sweet spot" in terms of performance vs dimension.
Collapse
Affiliation(s)
- Alec J Linot
- Department of Chemical and Biological Engineering, University of Wisconsin-Madison, Madison, Wisconsin 53706, USA
| | - Michael D Graham
- Department of Chemical and Biological Engineering, University of Wisconsin-Madison, Madison, Wisconsin 53706, USA
| |
Collapse
|
12
|
Vettelschoss B, Rohm A, Soriano MC. Information Processing Capacity of a Single-Node Reservoir Computer: An Experimental Evaluation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2022; 33:2714-2725. [PMID: 34662281 DOI: 10.1109/tnnls.2021.3116709] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Physical dynamical systems are able to process information in a nontrivial manner. The machine learning paradigm of reservoir computing (RC) provides a suitable framework for information processing in (analog) dynamical systems. The potential of dynamical systems for RC can be quantitatively characterized by the information processing capacity (IPC) measure. Here, we evaluate the IPC measure of a reservoir computer based on a single-analog nonlinear node coupled with delay. We link the extracted IPC measures to the dynamical regime of the reservoir, reporting an experimentally measured nonlinear memory of up to seventh order. In addition, we find a nonhomogeneous distribution of the linear and nonlinear contributions to the IPC as a function of the system operating conditions. Finally, we unveil the role of noise in the IPC of the analog implementation by performing ad hoc numerical simulations. In this manner, we identify the so-called edge of stability as being the most promising operating condition of the experimental implementation for RC purposes in terms of computational power and noise robustness. Similarly, a strong input drive is shown to have beneficial properties, albeit with a reduced memory depth.
Collapse
|
13
|
Srinivasan K, Coble N, Hamlin J, Antonsen T, Ott E, Girvan M. Parallel Machine Learning for Forecasting the Dynamics of Complex Networks. PHYSICAL REVIEW LETTERS 2022; 128:164101. [PMID: 35522516 DOI: 10.1103/physrevlett.128.164101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/27/2021] [Accepted: 03/28/2022] [Indexed: 06/14/2023]
Abstract
Forecasting the dynamics of large, complex, sparse networks from previous time series data is important in a wide range of contexts. Here we present a machine learning scheme for this task using a parallel architecture that mimics the topology of the network of interest. We demonstrate the utility and scalability of our method implemented using reservoir computing on a chaotic network of oscillators. Two levels of prior knowledge are considered: (i) the network links are known, and (ii) the network links are unknown and inferred via a data-driven approach to approximately optimize prediction.
Collapse
Affiliation(s)
| | - Nolan Coble
- University of Maryland, College Park, Maryland 20742, USA
- SUNY Brockport, Brockport, New York 14420, USA
| | - Joy Hamlin
- Stony Brook University, Long Island, New York 11794, USA
| | | | - Edward Ott
- University of Maryland, College Park, Maryland 20742, USA
| | | |
Collapse
|
14
|
Bauwens I, Harkhoe K, Bienstman P, Verschaffelt G, Van der Sande G. Influence of the input signal's phase modulation on the performance of optical delay-based reservoir computing using semiconductor lasers. OPTICS EXPRESS 2022; 30:13434-13446. [PMID: 35472955 DOI: 10.1364/oe.449508] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/24/2021] [Accepted: 03/06/2022] [Indexed: 06/14/2023]
Abstract
In photonic reservoir computing, semiconductor lasers with delayed feedback have shown to be suited to efficiently solve difficult and time-consuming problems. The input data in this system is often optically injected into the reservoir. Based on numerical simulations, we show that the performance depends heavily on the way that information is encoded in this optical injection signal. In our simulations we compare different input configurations consisting of Mach-Zehnder modulators and phase modulators for injecting the signal. We observe far better performance on a one-step ahead time-series prediction task when modulating the phase of the injected signal rather than only modulating its amplitude.
Collapse
|
15
|
Carroll TL. Optimizing memory in reservoir computers. CHAOS (WOODBURY, N.Y.) 2022; 32:023123. [PMID: 35232031 DOI: 10.1063/5.0078151] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/10/2021] [Accepted: 01/31/2022] [Indexed: 06/14/2023]
Abstract
A reservoir computer is a way of using a high dimensional dynamical system for computation. One way to construct a reservoir computer is by connecting a set of nonlinear nodes into a network. Because the network creates feedback between nodes, the reservoir computer has memory. If the reservoir computer is to respond to an input signal in a consistent way (a necessary condition for computation), the memory must be fading; that is, the influence of the initial conditions fades over time. How long this memory lasts is important for determining how well the reservoir computer can solve a particular problem. In this paper, I describe ways to vary the length of the fading memory in reservoir computers. Tuning the memory can be important to achieve optimal results in some problems; too much or too little memory degrades the accuracy of the computation.
Collapse
Affiliation(s)
- T L Carroll
- US Naval Research Lab, Washington DC 20375, USA
| |
Collapse
|
16
|
Platt JA, Wong A, Clark R, Penny SG, Abarbanel HDI. Robust forecasting using predictive generalized synchronization in reservoir computing. CHAOS (WOODBURY, N.Y.) 2021; 31:123118. [PMID: 34972341 DOI: 10.1063/5.0066013] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/07/2021] [Accepted: 11/15/2021] [Indexed: 06/14/2023]
Abstract
Reservoir computers (RCs) are a class of recurrent neural networks (RNNs) that can be used for forecasting the future of observed time series data. As with all RNNs, selecting the hyperparameters in the network to yield excellent forecasting presents a challenge when training on new inputs. We analyze a method based on predictive generalized synchronization (PGS) that gives direction in designing and evaluating the architecture and hyperparameters of an RC. To determine the occurrences of PGS, we rely on the auxiliary method to provide a computationally efficient pre-training test that guides hyperparameter selection. We provide a metric for evaluating the RC using the reproduction of the input system's Lyapunov exponents that demonstrates robustness in prediction.
Collapse
Affiliation(s)
- Jason A Platt
- Department of Physics, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
| | - Adrian Wong
- Department of Physics, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
| | - Randall Clark
- Department of Physics, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
| | - Stephen G Penny
- Cooperative Institute for Research in Environmental Sciences, University of Colorado Boulder, Boulder, Colorado 80305-3328, USA
| | - Henry D I Abarbanel
- Department of Physics, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
| |
Collapse
|
17
|
Jaurigue L, Robertson E, Wolters J, Lüdge K. Reservoir Computing with Delayed Input for Fast and Easy Optimisation. ENTROPY 2021; 23:e23121560. [PMID: 34945866 PMCID: PMC8700644 DOI: 10.3390/e23121560] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Revised: 11/19/2021] [Accepted: 11/21/2021] [Indexed: 01/30/2023]
Abstract
Reservoir computing is a machine learning method that solves tasks using the response of a dynamical system to a certain input. As the training scheme only involves optimising the weights of the responses of the dynamical system, this method is particularly suited for hardware implementation. Furthermore, the inherent memory of dynamical systems which are suitable for use as reservoirs mean that this method has the potential to perform well on time series prediction tasks, as well as other tasks with time dependence. However, reservoir computing still requires extensive task-dependent parameter optimisation in order to achieve good performance. We demonstrate that by including a time-delayed version of the input for various time series prediction tasks, good performance can be achieved with an unoptimised reservoir. Furthermore, we show that by including the appropriate time-delayed input, one unaltered reservoir can perform well on six different time series prediction tasks at a very low computational expense. Our approach is of particular relevance to hardware implemented reservoirs, as one does not necessarily have access to pertinent optimisation parameters in physical systems but the inclusion of an additional input is generally possible.
Collapse
Affiliation(s)
- Lina Jaurigue
- Institute of Theoretical Physics, Technische Universität Berlin, Hardenbergstr. 36, 10623 Berlin, Germany
- Correspondence:
| | - Elizabeth Robertson
- Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Institut fur Optische Sensorsysteme, Rutherfordstr. 2, 12489 Berlin, Germany; (E.R.); (J.W.)
- Institut für Optik und Atomare Physik, Technische Universität Berlin, 10623 Berlin, Germany
| | - Janik Wolters
- Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Institut fur Optische Sensorsysteme, Rutherfordstr. 2, 12489 Berlin, Germany; (E.R.); (J.W.)
- Institut für Optik und Atomare Physik, Technische Universität Berlin, 10623 Berlin, Germany
| | - Kathy Lüdge
- Institute of Physics, Technische Universität Ilmenau, Weimarer Str. 25, 98693 Ilmenau, Germany;
| |
Collapse
|
18
|
Barbosa WAS, Griffith A, Rowlands GE, Govia LCG, Ribeill GJ, Nguyen MH, Ohki TA, Gauthier DJ. Symmetry-aware reservoir computing. Phys Rev E 2021; 104:045307. [PMID: 34781436 DOI: 10.1103/physreve.104.045307] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2021] [Accepted: 09/22/2021] [Indexed: 06/13/2023]
Abstract
We demonstrate that matching the symmetry properties of a reservoir computer (RC) to the data being processed dramatically increases its processing power. We apply our method to the parity task, a challenging benchmark problem that highlights inversion and permutation symmetries, and to a chaotic system inference task that presents an inversion symmetry rule. For the parity task, our symmetry-aware RC obtains zero error using an exponentially reduced neural network and training data, greatly speeding up the time to result and outperforming artificial neural networks. When both symmetries are respected, we find that the network size N necessary to obtain zero error for 50 different RC instances scales linearly with the parity-order n. Moreover, some symmetry-aware RC instances perform a zero error classification with only N=1 for n≤7. Furthermore, we show that a symmetry-aware RC only needs a training data set with size on the order of (n+n/2) to obtain such a performance, an exponential reduction in comparison to a regular RC which requires a training data set with size on the order of n2^{n} to contain all 2^{n} possible n-bit-long sequences. For the inference task, we show that a symmetry-aware RC presents a normalized root-mean-square error three orders-of-magnitude smaller than regular RCs. For both tasks, our RC approach respects the symmetries by adjusting only the input and the output layers, and not by problem-based modifications to the neural network. We anticipate that the generalizations of our procedure can be applied in information processing for problems with known symmetries.
Collapse
Affiliation(s)
- Wendson A S Barbosa
- Department of Physics, The Ohio State University, 191 W. Woodruff Ave., Columbus, Ohio 43210, USA
| | - Aaron Griffith
- Department of Physics, The Ohio State University, 191 W. Woodruff Ave., Columbus, Ohio 43210, USA
| | - Graham E Rowlands
- Quantum Engineering and Computing, Raytheon BBN Technologies, Cambridge, Massachusetts 02138, USA
| | - Luke C G Govia
- Quantum Engineering and Computing, Raytheon BBN Technologies, Cambridge, Massachusetts 02138, USA
| | - Guilhem J Ribeill
- Quantum Engineering and Computing, Raytheon BBN Technologies, Cambridge, Massachusetts 02138, USA
| | - Minh-Hai Nguyen
- Quantum Engineering and Computing, Raytheon BBN Technologies, Cambridge, Massachusetts 02138, USA
| | - Thomas A Ohki
- Quantum Engineering and Computing, Raytheon BBN Technologies, Cambridge, Massachusetts 02138, USA
| | - Daniel J Gauthier
- Department of Physics, The Ohio State University, 191 W. Woodruff Ave., Columbus, Ohio 43210, USA
| |
Collapse
|
19
|
Carroll TL. Optimizing Reservoir Computers for Signal Classification. Front Physiol 2021; 12:685121. [PMID: 34220549 PMCID: PMC8249854 DOI: 10.3389/fphys.2021.685121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2021] [Accepted: 05/24/2021] [Indexed: 11/13/2022] Open
Abstract
Reservoir computers are a type of recurrent neural network for which the network connections are not changed. To train the reservoir computer, a set of output signals from the network are fit to a training signal by a linear fit. As a result, training of a reservoir computer is fast, and reservoir computers may be built from analog hardware, resulting in high speed and low power consumption. To get the best performance from a reservoir computer, the hyperparameters of the reservoir computer must be optimized. In signal classification problems, parameter optimization may be computationally difficult; it is necessary to compare many realizations of the test signals to get good statistics on the classification probability. In this work, it is shown in both a spiking reservoir computer and a reservoir computer using continuous variables that the optimum classification performance occurs for the hyperparameters that maximize the entropy of the reservoir computer. Optimizing for entropy only requires a single realization of each signal to be classified, making the process much faster to compute.
Collapse
|
20
|
Carroll TL. Low dimensional manifolds in reservoir computers. CHAOS (WOODBURY, N.Y.) 2021; 31:043113. [PMID: 34251231 DOI: 10.1063/5.0047006] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/09/2021] [Accepted: 03/26/2021] [Indexed: 06/13/2023]
Abstract
A reservoir computer is a complex dynamical system, often created by coupling nonlinear nodes in a network. The nodes are all driven by a common driving signal. Reservoir computers can contain hundreds to thousands of nodes, resulting in a high dimensional dynamical system, but the reservoir computer variables evolve on a lower dimensional manifold in this high dimensional space. This paper describes how this manifold dimension depends on the parameters of the reservoir computer, and how the manifold dimension is related to the performance of the reservoir computer at a signal estimation task. It is demonstrated that increasing the coupling between nodes while controlling the largest Lyapunov exponent of the reservoir computer can optimize the reservoir computer performance. It is also noted that the sparsity of the reservoir computer network does not have any influence on performance.
Collapse
Affiliation(s)
- T L Carroll
- U.S. Naval Research Lab, Washington, DC 20375, USA
| |
Collapse
|
21
|
Bollt E. On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD. CHAOS (WOODBURY, N.Y.) 2021; 31:013108. [PMID: 33754755 DOI: 10.1063/5.0024890] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/14/2020] [Accepted: 11/30/2020] [Indexed: 06/12/2023]
Abstract
Machine learning has become a widely popular and successful paradigm, especially in data-driven science and engineering. A major application problem is data-driven forecasting of future states from a complex dynamical system. Artificial neural networks have evolved as a clear leader among many machine learning approaches, and recurrent neural networks are considered to be particularly well suited for forecasting dynamical systems. In this setting, the echo-state networks or reservoir computers (RCs) have emerged for their simplicity and computational complexity advantages. Instead of a fully trained network, an RC trains only readout weights by a simple, efficient least squares method. What is perhaps quite surprising is that nonetheless, an RC succeeds in making high quality forecasts, competitively with more intensively trained methods, even if not the leader. There remains an unanswered question as to why and how an RC works at all despite randomly selected weights. To this end, this work analyzes a further simplified RC, where the internal activation function is an identity function. Our simplification is not presented for the sake of tuning or improving an RC, but rather for the sake of analysis of what we take to be the surprise being not that it does not work better, but that such random methods work at all. We explicitly connect the RC with linear activation and linear readout to well developed time-series literature on vector autoregressive (VAR) averages that includes theorems on representability through the Wold theorem, which already performs reasonably for short-term forecasts. In the case of a linear activation and now popular quadratic readout RC, we explicitly connect to a nonlinear VAR, which performs quite well. Furthermore, we associate this paradigm to the now widely popular dynamic mode decomposition; thus, these three are in a sense different faces of the same thing. We illustrate our observations in terms of popular benchmark examples including Mackey-Glass differential delay equations and the Lorenz63 system.
Collapse
Affiliation(s)
- Erik Bollt
- Department of Electrical and Computer Engineering, Clarkson University, Potsdam, New York 13699, USA and Clarkson Center for Complex Systems Science (C3S2), Potsdam, New York 13699, USA
| |
Collapse
|
22
|
Carroll TL. Do reservoir computers work best at the edge of chaos? CHAOS (WOODBURY, N.Y.) 2020; 30:121109. [PMID: 33380041 DOI: 10.1063/5.0038163] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/19/2020] [Accepted: 12/02/2020] [Indexed: 06/12/2023]
Abstract
It has been demonstrated that cellular automata had the highest computational capacity at the edge of chaos [N. H. Packard, in Dynamic Patterns in Complex Systems, edited by J. A. S. Kelso, A. J. Mandell, and M. F. Shlesinger (World Scientific, Singapore, 1988), pp. 293-301; C. G. Langton, Physica D 42(1), 12-37 (1990); J. P. Crutchfield and K. Young, in Complexity, Entropy, and the Physics of Information, edited by W. H. Zurek (Addison-Wesley, Redwood City, CA, 1990), pp. 223-269], the parameter at which their behavior transitioned from ordered to chaotic. This same concept has been applied to reservoir computers; a number of researchers have stated that the highest computational capacity for a reservoir computer is at the edge of chaos, although others have suggested that this rule is not universally true. Because many reservoir computers do not show chaotic behavior but merely become unstable, it is felt that a more accurate term for this instability transition is the "edge of stability." Here, I find two examples where the computational capacity of a reservoir computer decreases as the edge of stability is approached: in one case because generalized synchronization breaks down and in the other case because the reservoir computer is a poor match to the problem being solved. The edge of stability as an optimal operating point for a reservoir computer is not in general true, although it may be true in some cases.
Collapse
Affiliation(s)
- T L Carroll
- U.S. Naval Research Lab, Washington, DC 20375, USA
| |
Collapse
|
23
|
Carroll TL. Path length statistics in reservoir computers. CHAOS (WOODBURY, N.Y.) 2020; 30:083130. [PMID: 32872832 DOI: 10.1063/5.0014643] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/20/2020] [Accepted: 08/03/2020] [Indexed: 06/11/2023]
Abstract
Because reservoir computers are high dimensional dynamical systems, designing a good reservoir computer is difficult. In many cases, the designer must search a large nonlinear parameter space, and each step of the search requires simulating the full reservoir computer. In this work, I show that a simple statistic based on the mean path length between nodes in the reservoir computer is correlated with better reservoir computer performance. The statistic predicts the diversity of signals produced by the reservoir computer, as measured by the covariance matrix of the reservoir computer. This statistic by itself is not sufficient to predict reservoir computer performance because not only must the reservoir computer produce a diverse set of signals, it must be well matched to the training signals. Nevertheless, this path length statistic allows the designer to eliminate some network configurations from consideration without having to actually simulate the reservoir computer, reducing the complexity of the design process.
Collapse
Affiliation(s)
- T L Carroll
- U.S. Naval Research Lab, Washington, DC 20375, USA
| |
Collapse
|
24
|
Carroll TL. Dimension of reservoir computers. CHAOS (WOODBURY, N.Y.) 2020; 30:013102. [PMID: 32013466 DOI: 10.1063/1.5128898] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/23/2019] [Accepted: 12/11/2019] [Indexed: 06/10/2023]
Abstract
A reservoir computer is a complex dynamical system, often created by coupling nonlinear nodes in a network. The nodes are all driven by a common driving signal. In this work, three dimension estimation methods, false nearest neighbor, covariance dimension, and Kaplan-Yorke dimension, are used to estimate the dimension of the reservoir dynamical system. It is shown that the signals in the reservoir system exist on a relatively low dimensional surface. Changing the spectral radius of the reservoir network can increase the fractal dimension of the reservoir signals, leading to an increase in a testing error.
Collapse
Affiliation(s)
- T L Carroll
- US Naval Research Lab, Washington, DC 20375, USA
| |
Collapse
|
25
|
Chembo YK. Machine learning based on reservoir computing with time-delayed optoelectronic and photonic systems. CHAOS (WOODBURY, N.Y.) 2020; 30:013111. [PMID: 32013503 DOI: 10.1063/1.5120788] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/21/2019] [Accepted: 11/26/2019] [Indexed: 06/10/2023]
Abstract
The concept of reservoir computing emerged from a specific machine learning paradigm characterized by a three-layered architecture (input, reservoir, and output), where only the output layer is trained and optimized for a particular task. In recent years, this approach has been successfully implemented using various hardware platforms based on optoelectronic and photonic systems with time-delayed feedback. In this review, we provide a survey of the latest advances in this field, with some perspectives related to the relationship between reservoir computing, nonlinear dynamics, and network theory.
Collapse
Affiliation(s)
- Yanne K Chembo
- Department of Electrical and Computer Engineering, Institute for Research in Electronics and Applied Physics (IREAP), University of Maryland, 8279 Paint Branch Dr., College Park, Maryland 20742, USA
| |
Collapse
|
26
|
Griffith A, Pomerance A, Gauthier DJ. Forecasting chaotic systems with very low connectivity reservoir computers. CHAOS (WOODBURY, N.Y.) 2019; 29:123108. [PMID: 31893676 DOI: 10.1063/1.5120710] [Citation(s) in RCA: 48] [Impact Index Per Article: 9.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/19/2019] [Accepted: 11/20/2019] [Indexed: 06/10/2023]
Abstract
We explore the hyperparameter space of reservoir computers used for forecasting of the chaotic Lorenz '63 attractor with Bayesian optimization. We use a new measure of reservoir performance, designed to emphasize learning the global climate of the forecasted system rather than short-term prediction. We find that optimizing over this measure more quickly excludes reservoirs that fail to reproduce the climate. The results of optimization are surprising: the optimized parameters often specify a reservoir network with very low connectivity. Inspired by this observation, we explore reservoir designs with even simpler structure and find well-performing reservoirs that have zero spectral radius and no recurrence. These simple reservoirs provide counterexamples to widely used heuristics in the field and may be useful for hardware implementations of reservoir computers.
Collapse
Affiliation(s)
- Aaron Griffith
- Department of Physics, The Ohio State University, 191 W. Woodruff Ave., Columbus, Ohio 43210, USA
| | - Andrew Pomerance
- Potomac Research LLC, 801 N. Pitt St. # 117, Alexandria, Virginia 22314, USA
| | - Daniel J Gauthier
- Department of Physics, The Ohio State University, 191 W. Woodruff Ave., Columbus, Ohio 43210, USA
| |
Collapse
|
27
|
Carroll TL, Pecora LM. Network structure effects in reservoir computers. CHAOS (WOODBURY, N.Y.) 2019; 29:083130. [PMID: 31472504 DOI: 10.1063/1.5097686] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2019] [Accepted: 08/06/2019] [Indexed: 06/10/2023]
Abstract
A reservoir computer is a complex nonlinear dynamical system that has been shown to be useful for solving certain problems, such as prediction of chaotic signals, speech recognition, or control of robotic systems. Typically, a reservoir computer is constructed by connecting a large number of nonlinear nodes in a network, driving the nodes with an input signal and using the node outputs to fit a training signal. In this work, we set up reservoirs where the edges (or connections) between all the network nodes are either +1 or 0 and proceed to alter the network structure by flipping some of these edges from +1 to -1. We use this simple network because it turns out to be easy to characterize; we may use the fraction of edges flipped as a measure of how much we have altered the network. In some cases, the network can be rearranged in a finite number of ways without changing its structure; these rearrangements are symmetries of the network, and the number of symmetries is also useful for characterizing the network. We find that changing the number of edges flipped in the network changes the rank of the covariance of a matrix consisting of the time series from the different nodes in the network and speculate that this rank is important for understanding the reservoir computer performance.
Collapse
Affiliation(s)
- T L Carroll
- US Naval Research Laboratory, Washington, DC 20375, USA
| | - L M Pecora
- US Naval Research Laboratory, Washington, DC 20375, USA
| |
Collapse
|