Reservoir Computing Universality With Stochastic Inputs

The road to commercial success for neuromorphic technologies

Neuromorphic technologies adapt biological neural principles to synthesise high-efficiency computational devices, characterised by continuous real-time operation and sparse event-based communication. After several false starts, a confluence of advances now promises widespread commercial adoption. Gradient-based training of deep spiking neural networks is now an off-the-shelf technique for building general-purpose Neuromorphic applications, with open-source tools underwritten by theoretical results. Analog and mixed-signal Neuromorphic circuit designs are being replaced by digital equivalents in newer devices, simplifying application deployment while maintaining computational benefits. Designs for in-memory computing are also approaching commercial maturity. Solving two key problems-how to program general Neuromorphic applications; and how to deploy them at scale-clears the way to commercial success of Neuromorphic processors. Ultra-low-power Neuromorphic technology will find a home in battery-powered systems, local compute for internet-of-things devices, and consumer wearables. Inspiration from uptake of tensor processors and GPUs can help the field overcome remaining hurdles.

Chaotic recurrent neural networks for brain modelling: A review

Even in the absence of external stimuli, the brain is spontaneously active. Indeed, most cortical activity is internally generated by recurrence. Both theoretical and experimental studies suggest that chaotic dynamics characterize this spontaneous activity. While the precise function of brain chaotic activity is still puzzling, we know that chaos confers many advantages. From a computational perspective, chaos enhances the complexity of network dynamics. From a behavioural point of view, chaotic activity could generate the variability required for exploration. Furthermore, information storage and transfer are maximized at the critical border between order and chaos. Despite these benefits, many computational brain models avoid incorporating spontaneous chaotic activity due to the challenges it poses for learning algorithms. In recent years, however, multiple approaches have been proposed to overcome this limitation. As a result, many different algorithms have been developed, initially within the reservoir computing paradigm. Over time, the field has evolved to increase the biological plausibility and performance of the algorithms, sometimes going beyond the reservoir computing framework. In this review article, we examine the computational benefits of chaos and the unique properties of chaotic recurrent neural networks, with a particular focus on those typically utilized in reservoir computing. We also provide a detailed analysis of the algorithms designed to train chaotic RNNs, tracing their historical evolution and highlighting key milestones in their development. Finally, we explore the applications and limitations of chaotic RNNs for brain modelling, consider their potential broader impacts beyond neuroscience, and outline promising directions for future research.

Improving the performance of echo state networks through state feedback

Reservoir computing, using nonlinear dynamical systems, offers a cost-effective alternative to neural networks for complex tasks involving processing of sequential data, time series modeling, and system identification. Echo state networks (ESNs), a type of reservoir computer, mirror neural networks but simplify training. They apply fixed, random linear transformations to the internal state, followed by nonlinear changes. This process, guided by input signals and linear regression, adapts the system to match target characteristics, reducing computational demands. A potential drawback of ESNs is that the fixed reservoir may not offer the complexity needed for specific problems. While directly altering (training) the internal ESN would reintroduce the computational burden, an indirect modification can be achieved by redirecting some output as input. This feedback can influence the internal reservoir state, yielding ESNs with enhanced complexity suitable for broader challenges. In this paper, we demonstrate that by feeding some component of the reservoir state back into the network through the input, we can drastically improve upon the performance of a given ESN. We rigorously prove that, for any given ESN, feedback will almost always improve the accuracy of the output. For a set of three tasks, each representing different problem classes, we find that with feedback the average error measures are reduced by 30%-60%. Remarkably, feedback provides at least an equivalent performance boost to doubling the initial number of computational nodes, a computationally expensive and technologically challenging alternative. These results demonstrate the broad applicability and substantial usefulness of this feedback scheme.

Stochastic reservoir computers

Reservoir computing is a form of machine learning that utilizes nonlinear dynamical systems to perform complex tasks in a cost-effective manner when compared to typical neural networks. Recent advancements in reservoir computing, in particular quantum reservoir computing, use reservoirs that are inherently stochastic. In this paper, we investigate the universality of stochastic reservoir computers which use the probabilities of each stochastic reservoir state as the readout instead of the states themselves. This allows the number of readouts to scale exponentially with the size of the reservoir hardware, offering the advantage of compact device size. We prove that classes of stochastic echo state networks form universal approximating classes. We also investigate the performance of two practical examples in classification and chaotic time series prediction. While shot noise is a limiting factor, we show significantly improved performance compared to a deterministic reservoir computer with similar hardware when noise effects are small.

Universality of reservoir systems with recurrent neural networks

Approximation capability of reservoir systems whose reservoir is a recurrent neural network (RNN) is discussed. We show what we call uniform strong universality of RNN reservoir systems for a certain class of dynamical systems. This means that, given an approximation error to be achieved, one can construct an RNN reservoir system that approximates each target dynamical system in the class just via adjusting its linear readout. To show the universality, we construct an RNN reservoir system via parallel concatenation that has an upper bound of approximation error independent of each target in the class.

Infinite-dimensional next-generation reservoir computing

Next-generation reservoir computing (NG-RC) has attracted much attention due to its excellent performance in spatiotemporal forecasting of complex systems and its ease of implementation. This paper shows that NG-RC can be encoded as a kernel ridge regression that makes training efficient and feasible even when the space of chosen polynomial features is very large. Additionally, an extension to an infinite number of covariates is possible, which makes the methodology agnostic with respect to the lags into the past that are considered as explanatory factors, as well as with respect to the number of polynomial covariates, an important hyperparameter in traditional NG-RC. We show that this approach has solid theoretical backing and good behavior based on kernel universality properties previously established in the literature. Various numerical illustrations show that these generalizations of NG-RC outperform the traditional approach in several forecasting applications.

Universality and Approximation Bounds for Echo State Networks With Random Weights

We study the uniform approximation of echo state networks (ESNs) with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that ESNs with ReLU activation are universal. In this article, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the ESN can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU ESNs for sufficiently regular operators.

Ultrafast silicon photonic reservoir computing engine delivering over 200 TOPS

Reservoir computing (RC) is a powerful machine learning algorithm for information processing. Despite numerous optical implementations, its speed and scalability remain limited by the need to establish recurrent connections and achieve efficient optical nonlinearities. This work proposes a streamlined photonic RC design based on a new paradigm, called next-generation RC, which overcomes these limitations. Our design leads to a compact silicon photonic computing engine with an experimentally demonstrated processing speed of over 60 GHz. Experimental results demonstrate state-of-the-art performance in prediction, emulation, and classification tasks across various machine learning applications. Compared to traditional RC systems, our silicon photonic RC engine offers several key advantages, including no speed limitations, a compact footprint, and a high tolerance to fabrication errors. This work lays the foundation for ultrafast on-chip photonic RC, representing significant progress toward developing next-generation high-speed photonic computing and signal processing.

Infinite-dimensional reservoir computing

Reservoir computing approximation and generalization bounds are proved for a new concept class of input/output systems that extends the so-called generalized Barron functionals to a dynamic context. This new class is characterized by the readouts with a certain integral representation built on infinite-dimensional state-space systems. It is shown that this class is very rich and possesses useful features and universal approximation properties. The reservoir architectures used for the approximation and estimation of elements in the new class are randomly generated echo state networks with either linear or ReLU activation functions. Their readouts are built using randomly generated neural networks in which only the output layer is trained (extreme learning machines or random feature neural networks). The results in the paper yield a recurrent neural network-based learning algorithm with provable convergence guarantees that do not suffer from the curse of dimensionality when learning input/output systems in the class of generalized Barron functionals and measuring the error in a mean-squared sense.

Overcoming the coherence time barrier in quantum machine learning on temporal data

The practical implementation of many quantum algorithms known today is limited by the coherence time of the executing quantum hardware and quantum sampling noise. Here we present a machine learning algorithm, NISQRC, for qubit-based quantum systems that enables inference on temporal data over durations unconstrained by decoherence. NISQRC leverages mid-circuit measurements and deterministic reset operations to reduce circuit executions, while still maintaining an appropriate length persistent temporal memory in the quantum system, confirmed through the proposed Volterra Series analysis. This enables NISQRC to overcome not only limitations imposed by finite coherence, but also information scrambling in monitored circuits and sampling noise, problems that persist even in hypothetical fault-tolerant quantum computers that have yet to be realized. To validate our approach, we consider the channel equalization task to recover test signal symbols that are subject to a distorting channel. Through simulations and experiments on a 7-qubit quantum processor we demonstrate that NISQRC can recover arbitrarily long test signals, not limited by coherence time.

Master Memory Function for Delay-Based Reservoir Computers With Single-Variable Dynamics

We show that many delay-based reservoir computers considered in the literature can be characterized by a universal master memory function (MMF). Once computed for two independent parameters, this function provides linear memory capacity for any delay-based single-variable reservoir with small inputs. Moreover, we propose an analytical description of the MMF that enables its efficient and fast computation. Our approach can be applied not only to single-variable delay-based reservoirs governed by known dynamical rules, such as the Mackey-Glass or Stuart-Landau-like systems, but also to reservoirs whose dynamical model is not available.

Application of next-generation reservoir computing for predicting chaotic systems from partial observations

Next-generation reservoir computing is a machine-learning approach that has been recently proposed as an effective method for predicting the dynamics of chaotic systems. So far, this approach has been applied mainly under the assumption that all components of the state vector of dynamical systems are observable. Here we study the effectiveness of this method when only a scalar time series is available for observation. As illustrations, we use the time series of Rössler and Lorenz systems, as well as the chaotic time series generated by an electronic circuit. We found that prediction is only effective if the feature vector of a nonlinear autoregression algorithm contains monomials of a sufficiently high degree. Moreover, the prediction can be improved by replacing monomials with Chebyshev polynomials. Next-generation models, built on the basis of partial observations, are suitable not only for short-term forecasting, but are also capable of reproducing the long-term climate of chaotic systems. We demonstrate the reconstruction of the bifurcation diagram of the Rössler system and the return maps of the Lorenz and electronic circuit systems.

Controlling chaos using edge computing hardware

Machine learning provides a data-driven approach for creating a digital twin of a system - a digital model used to predict the system behavior. Having an accurate digital twin can drive many applications, such as controlling autonomous systems. Often, the size, weight, and power consumption of the digital twin or related controller must be minimized, ideally realized on embedded computing hardware that can operate without a cloud-computing connection. Here, we show that a nonlinear controller based on next-generation reservoir computing can tackle a difficult control problem: controlling a chaotic system to an arbitrary time-dependent state. The model is accurate, yet it is small enough to be evaluated on a field-programmable gate array typically found in embedded devices. Furthermore, the model only requires 25.0 ± 7.0 nJ per evaluation, well below other algorithms, even without systematic power optimization. Our work represents the first step in deploying efficient machine learning algorithms to the computing "edge."

Existence of reservoir with finite-dimensional output for universal reservoir computing

In this paper, we prove the existence of a reservoir that has a finite-dimensional output and makes the reservoir computing model universal. Reservoir computing is a method for dynamical system approximation that trains the static part of a model but fixes the dynamical part called the reservoir. Hence, reservoir computing has the advantage of training models with a low computational cost. Moreover, fixed reservoirs can be implemented as physical systems. Such reservoirs have attracted attention in terms of computation speed and energy consumption. The universality of a reservoir computing model is its ability to approximate an arbitrary system with arbitrary accuracy. Two sufficient reservoir conditions to make the model universal have been proposed. The first is the combination of fading memory and the separation property. The second is the neighborhood separation property, which we proposed recently. To date, it has been unknown whether a reservoir with a finite-dimensional output can satisfy these conditions. In this study, we prove that no reservoir with a finite-dimensional output satisfies the former condition. By contrast, we propose a single output reservoir that satisfies the latter condition. This implies that, for any dimension, a reservoir making the model universal exists with the output of that specified dimension. These results clarify the practical importance of our proposed conditions.

Emerging opportunities and challenges for the future of reservoir computing

Reservoir computing originates in the early 2000s, the core idea being to utilize dynamical systems as reservoirs (nonlinear generalizations of standard bases) to adaptively learn spatiotemporal features and hidden patterns in complex time series. Shown to have the potential of achieving higher-precision prediction in chaotic systems, those pioneering works led to a great amount of interest and follow-ups in the community of nonlinear dynamics and complex systems. To unlock the full capabilities of reservoir computing towards a fast, lightweight, and significantly more interpretable learning framework for temporal dynamical systems, substantially more research is needed. This Perspective intends to elucidate the parallel progress of mathematical theory, algorithm design and experimental realizations of reservoir computing, and identify emerging opportunities as well as existing challenges for large-scale industrial adoption of reservoir computing, together with a few ideas and viewpoints on how some of those challenges might be resolved with joint efforts by academic and industrial researchers across multiple disciplines.

Power-Efficient Multisensory Reservoir Computing Based on Zr-Doped HfO2 Memcapacitive Synapse Arrays

Hardware implementation tailored to requirements in reservoir computing would facilitate lightweight and powerful temporal processing. Capacitive reservoirs would boost power efficiency due to their ultralow static power consumption but have not been experimentally exploited yet. Here, this work reports an oxide-based memcapacitive synapse (OMC) based on Zr-doped HfO2 (HZO) for a power-efficient and multisensory processing reservoir computing system. The nonlinearity and state richness required for reservoir computing could originate from the capacitively coupled polarization switching and charge trapping of hafnium-oxide-based devices. The power consumption (≈113.4 fJ per spike) and temporal processing versatility outperform most resistive reservoirs. This system is verified by common benchmark tasks, and it exhibits high accuracy (>94%) in recognizing multisensory information, including acoustic, electrophysiological, and mechanic modalities. As a proof-of-concept, a touchless user interface for virtual shopping based on the OMC-based reservoir computing system is demonstrated, benefiting from its interference-robust acoustic and electrophysiological perception. These results shed light on the development of highly power-efficient human-machine interfaces and machine-learning platforms.

A novel approach to minimal reservoir computing

Reservoir computers are powerful machine learning algorithms for predicting nonlinear systems. Unlike traditional feedforward neural networks, they work on small training data sets, operate with linear optimization, and therefore require minimal computational resources. However, the traditional reservoir computer uses random matrices to define the underlying recurrent neural network and has a large number of hyperparameters that need to be optimized. Recent approaches show that randomness can be taken out by running regressions on a large library of linear and nonlinear combinations constructed from the input data and their time lags and polynomials thereof. However, for high-dimensional and nonlinear data, the number of these combinations explodes. Here, we show that a few simple changes to the traditional reservoir computer architecture further minimizing computational resources lead to significant and robust improvements in short- and long-term predictive performances compared to similar models while requiring minimal sizes of training data sets.

Effect of temporal resolution on the reproduction of chaotic dynamics via reservoir computing

Reservoir computing is a machine learning paradigm that uses a structure called a reservoir, which has nonlinearities and short-term memory. In recent years, reservoir computing has expanded to new functions such as the autonomous generation of chaotic time series, as well as time series prediction and classification. Furthermore, novel possibilities have been demonstrated, such as inferring the existence of previously unseen attractors. Sampling, in contrast, has a strong influence on such functions. Sampling is indispensable in a physical reservoir computer that uses an existing physical system as a reservoir because the use of an external digital system for the data input is usually inevitable. This study analyzes the effect of sampling on the ability of reservoir computing to autonomously regenerate chaotic time series. We found, as expected, that excessively coarse sampling degrades the system performance, but also that excessively dense sampling is unsuitable. Based on quantitative indicators that capture the local and global characteristics of attractors, we identify a suitable window of the sampling frequency and discuss its underlying mechanisms.

Deriving task specific performance from the information processing capacity of a reservoir computer

In the reservoir computing literature, the information processing capacity is frequently used to characterize the computing capabilities of a reservoir. However, it remains unclear how the information processing capacity connects to the performance on specific tasks. We demonstrate on a set of standard benchmark tasks that the total information processing capacity correlates poorly with task specific performance. Further, we derive an expression for the normalized mean square error of a task as a weighted function of the individual information processing capacities. Mathematically, the derivation requires the task to have the same input distribution as used to calculate the information processing capacities. We test our method on a range of tasks that violate this requirement and find good qualitative agreement between the predicted and the actual errors as long as the task input sequences do not have long autocorrelation times. Our method offers deeper insight into the principles governing reservoir computing performance. It also increases the utility of the evaluation of information processing capacities, which are typically defined on i.i.d. input, even if specific tasks deliver inputs stemming from different distributions. Moreover, it offers the possibility of reducing the experimental cost of optimizing physical reservoirs, such as those implemented in photonic systems.

Quantum reservoir computing in finite dimensions

Most existing results in the analysis of quantum reservoir computing (QRC) systems with classical inputs have been obtained using the density matrix formalism. This paper shows that alternative representations can provide better insight when dealing with design and assessment questions. More explicitly, system isomorphisms are established that unify the density matrix approach to QRC with the representation in the space of observables using Bloch vectors associated with Gell-Mann bases. It is shown that these vector representations yield state-affine systems previously introduced in the classical reservoir computing literature and for which numerous theoretical results have been established. This connection is used to show that various statements in relation to the fading memory property (FMP) and the echo state property (ESP) are independent of the representation and also to shed some light on fundamental questions in QRC theory in finite dimensions. In particular, a necessary and sufficient condition for the ESP and FMP to hold is formulated using standard hypotheses, and contractive quantum channels that have exclusively trivial semi-infinite solutions are characterized in terms of the existence of input-independent fixed points.

DAFA-BiLSTM: Deep Autoregression Feature Augmented Bidirectional LSTM network for time series prediction

Time series forecasting models that use the past information of exogenous or endogenous sequences to forecast future series play an important role in the real world because most real-world time series datasets are rich in time-dependent information. Most conventional prediction models for time series datasets are time-consuming and fraught with complex limitations because they usually fail to adequately exploit the latent spatial dependence between pairs of variables. As a successful variant of recurrent neural networks, the long short-term memory network (LSTM) has been demonstrated to have stronger nonlinear dynamics to store sequential data than traditional machine learning models. Nevertheless, the common shallow LSTM architecture has limited capacity to fully extract the transient characteristics of long interval sequential datasets. In this study, a novel deep autoregression feature augmented bidirectional LSTM network (DAFA-BiLSTM) is proposed as a new deep BiLSTM architecture for time series prediction. Initially, the input vectors are fed into a vector autoregression (VA) transformation module to represent the time-delayed linear and nonlinear properties of the input signals in an unsupervised way. Then, the learned nonlinear combination vectors of VA are progressively fed into different layers of BiLSTM and the output of the previous BiLSTM module is also concatenated with the time-delayed linear vectors of the VA as an augmented feature to form new additional input signals for the next adjacent BiLSTM layer. Extensive real-world time series applications are addressed to demonstrate the superiority and robustness of the proposed DAFA-BiLSTM. Comparative experimental results and statistical analysis show that the proposed DAFA-BiLSTM has good adaptive performance as well as robustness even in noisy environment.

Discrete-Time Signatures and Randomness in Reservoir Computing

A new explanation of the geometric nature of the reservoir computing (RC) phenomenon is presented. RC is understood in the literature as the possibility of approximating input-output systems with randomly chosen recurrent neural systems and a trained linear readout layer. Light is shed on this phenomenon by constructing what is called strongly universal reservoir systems as random projections of a family of state-space systems that generate Volterra series expansions. This procedure yields a state-affine reservoir system with randomly generated coefficients in a dimension that is logarithmically reduced with respect to the original system. This reservoir system is able to approximate any element in the fading memory filters class just by training a different linear readout for each different filter. Explicit expressions for the probability distributions needed in the generation of the projected reservoir system are stated, and bounds for the committed approximation error are provided.

Learning unseen coexisting attractors

Reservoir computing is a machine learning approach that can generate a surrogate model of a dynamical system. It can learn the underlying dynamical system using fewer trainable parameters and, hence, smaller training data sets than competing approaches. Recently, a simpler formulation, known as next-generation reservoir computing, removed many algorithm metaparameters and identified a well-performing traditional reservoir computer, thus simplifying training even further. Here, we study a particularly challenging problem of learning a dynamical system that has both disparate time scales and multiple co-existing dynamical states (attractors). We compare the next-generation and traditional reservoir computer using metrics quantifying the geometry of the ground-truth and forecasted attractors. For the studied four-dimensional system, the next-generation reservoir computing approach uses ∼ 1.7 × less training data, requires ¹⁰ × shorter "warmup" time, has fewer metaparameters, and has an ∼ 100 × higher accuracy in predicting the co-existing attractor characteristics in comparison to a traditional reservoir computer. Furthermore, we demonstrate that it predicts the basin of attraction with high accuracy. This work lends further support to the superior learning ability of this new machine learning algorithm for dynamical systems.

High-speed photonic neuromorphic computing using recurrent optical spectrum slicing neural networks

Neuromorphic computing using photonic hardware is a promising route towards ultrafast processing while maintaining low power consumption. Here we present and numerically evaluate a hardware concept for realizing photonic recurrent neural networks and reservoir computing architectures. Our method, called Recurrent Optical Spectrum Slicing Neural Networks (ROSS-NNs), uses simple optical filters placed in a loop, where each filter processes a specific spectral slice of the incoming optical signal. The synaptic weights in our scheme are equivalent to the filters’ central frequencies and bandwidths. Numerical application to high baud rate optical signal equalization (>100 Gbaud) reveals that ROSS-NN extends optical signal transmission reach to > 60 km, more than four times that of two state-of-the-art digital equalizers. Furthermore, ROSS-NN relaxes complexity, requiring less than 100 multiplications/bit in the digital domain, offering tenfold reduction in power consumption with respect to these digital counterparts. ROSS-NNs hold promise for efficient photonic hardware accelerators tailored for processing high-bandwidth (>100 GHz) optical signals in optical communication and high-speed imaging applications.

Sozos and co-workers present and numerically evaluate photonic neuromorphic hardware using recurrent optical spectrum slicing for use in ultra-fast optical applications. The approach extends optical signal transmission reach to more than four-fold that of two state-of-the-art digital equalizers and reduces power consumption tenfold.

Time series reconstructing using calibrated reservoir computing

Reservoir computing, a new method of machine learning, has recently been used to predict the state evolution of various chaotic dynamic systems. It has significant advantages in terms of training cost and adjusted parameters; however, the prediction length is limited. For classic reservoir computing, the prediction length can only reach five to six Lyapunov times. Here, we modified the method of reservoir computing by adding feedback, continuous or discrete, to “calibrate” the input of the reservoir and then reconstruct the entire dynamic systems. The reconstruction length appreciably increased and the training length obviously decreased. The reconstructing of dynamical systems is studied in detail under this method. The reconstruction can be significantly improved both in length and accuracy. Additionally, we summarized the effect of different kinds of input feedback. The more it interacts with others in dynamical equations, the better the reconstructions. Nonlinear terms can reveal more information than linear terms once the interaction terms are equal. This method has proven effective via several classical chaotic systems. It can be superior to traditional reservoir computing in reconstruction, provides new hints in computing promotion, and may be used in some real applications.

Adaptive Practical Nonlinear Model Predictive Control for Echo State Network Models

This article proposes an adaptive practical nonlinear model predictive (NMPC) control algorithm which uses an echo state network (ESN) estimated online as a process model. In the proposed control algorithm, the ESN readout parameters are estimated online using a recursive least-squares method that considers an adaptive directional forgetting factor. The ESN model is used to obtain online a nonlinear prediction of the system free response, and a linearized version of the neural model is obtained at each sampling time to get a local approximation of the system step response, which is used to build the dynamic matrix of the system. The proposed controller was evaluated in a benchmark conical tank level control problem, and the results were compared with three baseline controllers. The proposed approach achieved similar results as the ones obtained by its nonadaptive baseline version in a scenario with the process operating with the nominal parameters, and outperformed all baseline algorithms in a scenario with process parameter changes. Additionally, the computational time required by the proposed algorithm was one-tenth of that required by the baseline NMPC, which shows that the proposed algorithm is suitable to implement state-of-the-art adaptive NMPC in a computationally affordable manner.

Reservoir computing with random and optimized time-shifts

We investigate the effects of application of random time-shifts to the readouts of a reservoir computer in terms of both accuracy (training error) and performance (testing error). For different choices of the reservoir parameters and different "tasks," we observe a substantial improvement in both accuracy and performance. We then develop a simple but effective technique to optimize the choice of the time-shifts, which we successfully test in numerical experiments.

Model-free inference of unseen attractors: Reconstructing phase space features from a single noisy trajectory using reservoir computing

Reservoir computers are powerful tools for chaotic time series prediction. They can be trained to approximate phase space flows and can thus both predict future values to a high accuracy and reconstruct the general properties of a chaotic attractor without requiring a model. In this work, we show that the ability to learn the dynamics of a complex system can be extended to systems with multiple co-existing attractors, here a four-dimensional extension of the well-known Lorenz chaotic system. We demonstrate that a reservoir computer can infer entirely unexplored parts of the phase space; a properly trained reservoir computer can predict the existence of attractors that were never approached during training and, therefore, are labeled as unseen. We provide examples where attractor inference is achieved after training solely on a single noisy trajectory.

Next generation reservoir computing

Reservoir computing is a best-in-class machine learning algorithm for processing information generated by dynamical systems using observed time-series data. Importantly, it requires very small training data sets, uses linear optimization, and thus requires minimal computing resources. However, the algorithm uses randomly sampled matrices to define the underlying recurrent neural network and has a multitude of metaparameters that must be optimized. Recent results demonstrate the equivalence of reservoir computing to nonlinear vector autoregression, which requires no random matrices, fewer metaparameters, and provides interpretable results. Here, we demonstrate that nonlinear vector autoregression excels at reservoir computing benchmark tasks and requires even shorter training data sets and training time, heralding the next generation of reservoir computing.

Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem

This paper shows that a large class of fading memory state-space systems driven by discrete-time observations of dynamical systems defined on compact manifolds always yields continuously differentiable synchronizations. This general result provides a powerful tool for the representation, reconstruction, and forecasting of chaotic attractors. It also improves previous statements in the literature for differentiable generalized synchronizations, whose existence was so far guaranteed for a restricted family of systems and was detected using Hölder exponent-based criteria.

Towards Embedded Computation with Building Materials

We present results showing the capability of concrete-based information processing substrate in the signal classification task in accordance with in materio computing paradigm. As the Reservoir Computing is a suitable model for describing embedded in materio computation, we propose that this type of presented basic construction unit can be used as a source for “reservoir of states” necessary for simple tuning of the readout layer. We present an electrical characterization of the set of samples with different additive concentrations followed by a dynamical analysis of selected specimens showing fingerprints of memfractive properties. As part of dynamic analysis, several fractal dimensions and entropy parameters for the output signal were analyzed to explore the richness of the reservoir configuration space. In addition, to investigate the chaotic nature and self-affinity of the signal, Lyapunov exponents and Detrended Fluctuation Analysis exponents were calculated. Moreover, on the basis of obtained parameters, classification of the signal waveform shapes can be performed in scenarios explicitly tuned for a given device terminal.

Fading memory echo state networks are universal

Echo state networks (ESNs) have been recently proved to be universal approximants for input/output systems with respect to various L^p-type criteria. When 1≤p<∞, only p-integrability hypotheses need to be imposed, while in the case p=∞ a uniform boundedness hypotheses on the inputs is required. This note shows that, in the last case, a universal family of ESNs can be constructed that contains exclusively elements that have the echo state and the fading memory properties. This conclusion could not be drawn with the results and methods available so far in the literature.

On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD

Machine learning has become a widely popular and successful paradigm, especially in data-driven science and engineering. A major application problem is data-driven forecasting of future states from a complex dynamical system. Artificial neural networks have evolved as a clear leader among many machine learning approaches, and recurrent neural networks are considered to be particularly well suited for forecasting dynamical systems. In this setting, the echo-state networks or reservoir computers (RCs) have emerged for their simplicity and computational complexity advantages. Instead of a fully trained network, an RC trains only readout weights by a simple, efficient least squares method. What is perhaps quite surprising is that nonetheless, an RC succeeds in making high quality forecasts, competitively with more intensively trained methods, even if not the leader. There remains an unanswered question as to why and how an RC works at all despite randomly selected weights. To this end, this work analyzes a further simplified RC, where the internal activation function is an identity function. Our simplification is not presented for the sake of tuning or improving an RC, but rather for the sake of analysis of what we take to be the surprise being not that it does not work better, but that such random methods work at all. We explicitly connect the RC with linear activation and linear readout to well developed time-series literature on vector autoregressive (VAR) averages that includes theorems on representability through the Wold theorem, which already performs reasonably for short-term forecasts. In the case of a linear activation and now popular quadratic readout RC, we explicitly connect to a nonlinear VAR, which performs quite well. Furthermore, we associate this paradigm to the now widely popular dynamic mode decomposition; thus, these three are in a sense different faces of the same thing. We illustrate our observations in terms of popular benchmark examples including Mackey-Glass differential delay equations and the Lorenz63 system.

Using machine learning to assess short term causal dependence and infer network links

We introduce and test a general machine-learning-based technique for the inference of short term causal dependence between state variables of an unknown dynamical system from time-series measurements of its state variables. Our technique leverages the results of a machine learning process for short time prediction to achieve our goal. The basic idea is to use the machine learning to estimate the elements of the Jacobian matrix of the dynamical flow along an orbit. The type of machine learning that we employ is reservoir computing. We present numerical tests on link inference of a network of interacting dynamical nodes. It is seen that dynamical noise can greatly enhance the effectiveness of our technique, while observational noise degrades the effectiveness. We believe that the competition between these two opposing types of noise will be the key factor determining the success of causal inference in many of the most important application situations.