Chaotic time series prediction based on a novel robust echo state network

Complex network-based multistep forecasting model for hyperchaotic time series

We present a method for predicting hyperchaotic time series using a complex network-based forecasting model. We first construct a network from a given time series, which serves as a coarse-grained representation of the underlying attractor. This network facilitates multistep forecasting by capturing the local nonlinearity of the dynamics and offers superior accuracy over more extended periods than traditional methods. The network is formed by converting the patterns of local oscillations into sequences of numerical symbols, which are then used to create nodes and edges in a network, capturing the system's dynamical behavior at a reduced resolution. The network allows predictions up to several steps ahead without the exponential error increase usually associated with linear first-order methods. The improved predictions result from the unique ability of the network to collect identical pattern transitions in the orbit dynamics into a system of neighborhoods in the network. The effectiveness of this approach is demonstrated through its application to several high-dimensional hyperchaotic systems, where it outperforms both the linear first-order and other network-based methods in terms of prediction accuracy and horizon. Besides enhancing the predictability of chaotic systems, this methodology also outlines a procedure to develop a discrete model flow within an attractor.

High-Dimensional Physical Reservoir with Back-End-of-Line-Compatible Tin Monoxide Thin-Film Transistor

This work demonstrates a physical reservoir using a back-end-of-line compatible thin-film transistor (TFT) with tin monoxide (SnO) as the channel material for neuromorphic computing. The electron trapping and time-dependent detrapping at the channel interface induce the SnO·TFT to exhibit fading memory and nonlinearity characteristics, the critical assets for physical reservoir computing. The three-terminal configuration of the TFT allows the generation of higher-dimensional reservoir states by simultaneously adjusting the bias conditions of the gate and drain terminals, surpassing the performances of typical two-terminal-based reservoirs such as memristors. The high-dimensional SnO TFT reservoir performs exceptionally in two benchmark tests, achieving a 94.1% accuracy in Modified National Institute of Standards and Technology handwritten number recognition and a normalized root-mean-square error of 0.089 in Mackey-Glass time-series prediction. Furthermore, it is suitable for vertical integration because its fabrication temperature is <250 °C, providing the benefit of achieving a high integration density.

Evolving Dual-Threshold Bienenstock-Cooper-Munro Learning Rules in Echo State Networks

The strengthening and the weakening of synaptic strength in existing Bienenstock-Cooper-Munro (BCM) learning rule are determined by a long-term potentiation (LTP) sliding modification threshold and the afferent synaptic activities. However, synaptic long-term depression (LTD) even affects low-active synapses during the induction of synaptic plasticity, which may lead to information loss. Biological experiments have found another LTD threshold that can induce either potentiation or depression or no change, even at the activated synapses. In addition, existing BCM learning rules can only select a set of fixed rule parameters, which is biologically implausible and practically inflexible to learn the structural information of input signals. In this article, an evolved dual-threshold BCM learning rule is proposed to regulate the reservoir internal connection weights of the echo-state-network (ESN), which can contribute to alleviating information loss and enhancing learning performance by introducing different optimal LTD thresholds for different postsynaptic neurons. Our experimental results show that the evolved dual-threshold BCM learning rule can result in the synergistic learning of different plasticity rules, effectively improving the learning performance of an ESN in comparison with existing neural plasticity learning rules and some state-of-the-art ESN variants on three widely used benchmark tasks and the prediction of an esterification process.

Spatiotemporal Data Processing with Memristor Crossbar-Array-Based Graph Reservoir

Memristor-based physical reservoir computing (RC) is a robust framework for processing complex spatiotemporal data parallelly. However, conventional memristor-based reservoirs cannot capture the spatial relationship between the time-varying inputs due to the specific mapping scheme assigning one input signal to one memristor conductance. Here, a physical "graph reservoir" is introduced using a metal cell at the diagonal-crossbar array (mCBA) with dynamic self-rectifying memristors. Input and inverted input signals are applied to the word and bit lines of the mCBA, respectively, storing the correlation information between input signals in the memristors. In this way, the mCBA graph reservoirs can map the spatiotemporal correlation of the input data in a high-dimensional feature space. The high-dimensional mapping characteristics of the graph reservoir achieve notable results, including a normalized root-mean-square error of 0.09 in Mackey-Glass time series prediction, a 97.21% accuracy in MNIST recognition, and an 80.0% diagnostic accuracy in human connectome classification.

fastESN: Fast Echo State Network

Echo state networks (ESNs) are reservoir computing-based recurrent neural networks widely used in pattern analysis and machine intelligence applications. In order to achieve high accuracy with large model capacity, ESNs usually contain a large-sized internal layer (reservoir), making the evaluation process too slow for some applications. In this work, we speed up the evaluation of ESN by building a reduced network called the fast ESN (fastESN) and achieve an ESN evaluation complexity independent of the original ESN size for the first time. FastESN is generated using three techniques. First, the high-dimensional state of the original ESN is approximated by a low-dimensional state through proper orthogonal decomposition (POD)-based projection. Second, the activation function evaluation number is reduced through the discrete empirical interpolation method (DEIM). Third, we show the directly generated fastESN has instability problems and provide a stabilization scheme as a solution. Through experiments on four popular benchmarks, we show that fastESN is able to accelerate the sparse storage-based ESN evaluation with a high parameter compression ratio and a fast evaluation speed.

Hierarchical Echo State Network With Sparse Learning: A Method for Multidimensional Chaotic Time Series Prediction

Echo state network (ESN), a type of special recurrent neural network with a large-scale randomly fixed hidden layer (called a reservoir) and an adaptable linear output layer, has been widely employed in the field of time series analysis and modeling. However, when tackling the problem of multidimensional chaotic time series prediction, due to the randomly generated rules for input and reservoir weights, not only the representation of valuable variables is enriched but also redundant and irrelevant information is accumulated inevitably. To remove the redundant components, reduce the approximate collinearity among echo-state information, and improve the generalization and stability, a new method called hierarchical ESN with sparse learning (HESN-SL) is proposed. The HESN-SL mines and captures the latent evolution patterns hidden from the dynamic system by means of layer-by-layer processing in stacked reservoirs, and leverage monotone accelerated proximal gradient algorithm to train a sparse output layer with variable selection capability. Meanwhile, we further prove that the HESN-SL satisfies the echo state property, which guarantees the stability and convergence of the proposed model when applied to time series prediction. Experimental results on two synthetic chaotic systems and a real-world meteorological dataset illustrate the proposed HESN-SL outperforms both original ESN and existing hierarchical ESN-based models for multidimensional chaotic time series prediction.

Configured quantum reservoir computing for multi-task machine learning

Amidst the rapid advancements in experimental technology, noise-intermediate-scale quantum (NISQ) devices have become increasingly programmable, offering versatile opportunities to leverage quantum computational advantage. Here we explore the intricate dynamics of programmable NISQ devices for quantum reservoir computing. Using a genetic algorithm to configure the quantum reservoir dynamics, we systematically enhance the learning performance. Remarkably, a single configured quantum reservoir can simultaneously learn multiple tasks, including a synthetic oscillatory network of transcriptional regulators, chaotic motifs in gene regulatory networks, and the fractional-order Chua's circuit. Our configured quantum reservoir computing yields highly precise predictions for these learning tasks, outperforming classical reservoir computing. We also test the configured quantum reservoir computing in foreign exchange (FX) market applications and demonstrate its capability to capture the stochastic evolution of the exchange rates with significantly greater accuracy than classical reservoir computing approaches. Through comparison with classical reservoir computing, we highlight the unique role of quantum coherence in the quantum reservoir, which underpins its exceptional learning performance. Our findings suggest the exciting potential of configured quantum reservoir computing for exploiting the quantum computation power of NISQ devices in developing artificial general intelligence.

Matrix completion under complex survey sampling

Multivariate nonresponse is often encountered in complex survey sampling, and simply ignoring it leads to erroneous inference. In this paper, we propose a new matrix completion method for complex survey sampling. Different from existing works either conducting row-wise or column-wise imputation, the data matrix is treated as a whole which allows for exploiting both row and column patterns simultaneously. A column-space-decomposition model is adopted incorporating a low-rank structured matrix for the finite population with easy-to-obtain demographic information as covariates. Besides, we propose a computationally efficient projection strategy to identify the model parameters under complex survey sampling. Then, an augmented inverse probability weighting estimator is used to estimate the parameter of interest, and the corresponding asymptotic upper bound of the estimation error is derived. Simulation studies show that the proposed estimator has a smaller mean squared error than other competitors, and the corresponding variance estimator performs well. The proposed method is applied to assess the health status of the U.S. population.

A novel time series prediction method based on pooling compressed sensing echo state network and its application in stock market

In the prediction of time series, the echo state network (ESN) exhibits exclusive strengths and a unique training structure. Based on ESN model, a pooling activation algorithm consisting noise value and adjusted pooling algorithm is proposed to enrich the update strategy of the reservoir layer in ESN. The algorithm optimizes the distribution of reservoir layer nodes. And the nodes set will be more matched to the characteristics of the data. In addition, we introduce a more efficient and accurate compressed sensing technique based on the existing research. The novel compressed sensing technique reduces the amount of spatial computation of methods. The ESN model based on the above two techniques overcomes the limitations in traditional prediction. In the experimental part, the model is validated with different chaotic time series as well as multiple stocks, and the method shows its efficiency and accuracy in prediction.

Minimum complexity integrated photonic architecture for delay-based reservoir computing

Reservoir computing is an analog bio-inspired computation scheme for efficiently processing time-dependent signals, the photonic implementations of which promise a combination of massive parallel information processing, low power consumption, and high-speed operation. However, most of these implementations, especially for the case of time-delay reservoir computing, require extensive multi-dimensional parameter optimization to find the optimal combination of parameters for a given task. We propose a novel, largely passive integrated photonic TDRC scheme based on an asymmetric Mach-Zehnder interferometer in a self-feedback configuration, where the nonlinearity is provided by the photodetector, and with only one tunable parameter in the form of a phase shifting element that, as a result of our configuration, allows also to tune the feedback strength, consequently tuning the memory capacity in a lossless manner. Through numerical simulations, we show that the proposed scheme achieves good performance -when compared to other integrated photonic architectures- on the temporal bitwise XOR task and various time series prediction tasks, while greatly reducing hardware and operational complexity.

Path integral approach to universal dynamics of reservoir computers

In this work, we give a characterization of the reservoir computer (RC) by the network structure, especially the probability distribution of random coupling constants. First, based on the path integral method, we clarify the universal behavior of the random network dynamics in the thermodynamic limit, which depends only on the asymptotic behavior of the second cumulant generating functions of the network coupling constants. This result enables us to classify the random networks into several universality classes, according to the distribution function of coupling constants chosen for the networks. Interestingly, it is revealed that such a classification has a close relationship with the distribution of eigenvalues of the random coupling matrix. We also comment on the relation between our theory and some practical choices of random connectivity in the RC. Subsequently, we investigate the relationship between the RC's computational power and the network parameters for several universality classes. We perform several numerical simulations to evaluate the phase diagrams of the steady reservoir states, common-signal-induced synchronization, and the computational power in the chaotic time series inference tasks. As a result, we clarify the close relationship between these quantities, especially a remarkable computational performance near the phase transitions, which is realized even near a nonchaotic transition boundary. These results may provide us with a new perspective on the designing principle for the RC.

Sparse Bayesian Learning Based on Collaborative Neurodynamic Optimization

Regression in a sparse Bayesian learning (SBL) framework is usually formulated as a global optimization problem with a nonconvex objective function and solved in a majorization-minimization framework where the solution quality and consistency depend heavily on the initial values of the used algorithm. In view of the shortcomings, this article presents an SBL algorithm based on collaborative neurodynamic optimization (CNO) for searching global optimal solutions to the global optimization problem. The CNO system consists of a population of recurrent neural networks (RNNs) where each RNN is convergent to a local optimum to the global optimization problem. Reinitialized repetitively via particle swarm optimization with exchanged local optima information, the RNNs iteratively improve their searching performance until reaching global convergence. The proposed CNO-based SBL algorithm is almost surely convergent to a global optimal solution to the formulated global optimization problem. Two applications with experimental results on sparse signal reconstruction and partial differential equation identification are elaborated to substantiate the superiority and efficacy of the proposed method in terms of solution optimality and consistency.

Computational Modeling of Structural Synaptic Plasticity in Echo State Networks

Most existing studies on computational modeling of neural plasticity have focused on synaptic plasticity. However, regulation of the internal weights in the reservoir based on synaptic plasticity often results in unstable learning dynamics. In this article, a structural synaptic plasticity learning rule is proposed to train the weights and add or remove neurons within the reservoir, which is shown to be able to alleviate the instability of the synaptic plasticity, and to contribute to increase the memory capacity of the network as well. Our experimental results also reveal that a few stronger connections may last for a longer period of time in a constantly changing network structure, and are relatively resistant to decay or disruptions in the learning process. These results are consistent with the evidence observed in biological systems. Finally, we show that an echo state network (ESN) using the proposed structural plasticity rule outperforms an ESN using synaptic plasticity and three state-of-the-art ESNs on four benchmark tasks.

An Efficient Sparse Bayesian Learning Algorithm Based on Gaussian-Scale Mixtures

Sparse Bayesian learning (SBL) is a popular machine learning approach with a superior generalization capability due to the sparsity of its adopted model. However, it entails a matrix inversion at each iteration, hindering its practical applications with large-scale data sets. To overcome this bottleneck, we propose an efficient SBL algorithm with O(n²) computational complexity per iteration based on a Gaussian-scale mixture prior model. By specifying two different hyperpriors, the proposed efficient SBL algorithm can meet two different requirements, such as high efficiency and high sparsity. A surrogate function is introduced herein to approximate the posterior density of model parameters and thereby to avoid matrix inversions. Using a data-dependent term, a joint cost function with separate penalty terms is reformulated in a joint space of model parameters and hyperparameters. The resulting nonconvex optimization problem is solved using a block coordinate descent method in a majorization-minimization framework. Finally, the results of extensive experiments for sparse signal recovery and sparse image reconstruction on benchmark problems are elaborated to substantiate the effectiveness and superiority of the proposed approach in terms of computational time and estimation error.

A Novel Broad Echo State Network for Time Series Prediction: Cascade of Mapping Nodes and Optimization of Enhancement Layer

Time series prediction is crucial for advanced control and management of complex systems, while the actual data are usually highly nonlinear and nonstationary. A novel broad echo state network is proposed herein for the prediction problem of complex time series data. Firstly, the framework of the broad echo state network with cascade of mapping nodes (CMBESN) is designed by embedding the echo state network units into the broad learning system. Secondly, the number of enhancement layer nodes of the CMBESN is determined by proposing an incremental algorithm. It can obtain the optimal network structure parameters. Meanwhile, an optimization method is proposed based on the nonstationary statistic metrics to determine the enhancement layer. Finally, experiments are conducted both on the simulated and actual datasets. The results show that the proposed CMBESN and its optimization have good prediction capability for nonstationary time series data.

Learning Both Dynamic-Shared and Dynamic-Specific Patterns for Chaotic Time-Series Prediction

In the real world, multivariate time series from the dynamical system are correlated with deterministic relationships. Analyzing them dividedly instead of utilizing the shared-pattern of the dynamical system is time consuming and cumbersome. Multitask learning (MTL) is an effective inductive bias method to utilize latent shared features and discover the structural relationships from related tasks. Base on this concept, we propose a novel MTL model for multivariate chaotic time-series prediction, which could learn both dynamic-shared and dynamic-specific patterns. We implement the dynamic analysis of multiple time series through a special network structure design. The model could disentangle the complex relationships among multivariate chaotic time series and derive the common evolutionary trend of the multivariate chaotic dynamical system by inductive bias. We also develop an efficient Crank-Nicolson-like curvilinear update algorithm based on the alternating direction method of multipliers (ADMM) for the nonconvex nonsmooth Stiefel optimization problem. Simulation results and analysis demonstrate the effectiveness on dynamic-shared pattern discovery and prediction performance.

Mutual Information Variational Autoencoders and Its Application to Feature Extraction of Multivariate Time Series

The application of deep learning in time-series prediction has developed gradually. In this paper, we propose a deep generative network model for feature extraction of multivariate time series, namely, mutual information variational autoencoders (MI-VAE). In the architecture of the proposed model, we use the latent space of VAE for feature learning, which can extract the essential features of multivariate time-series data effectively. The latent space employed directly as a feature extractor can avoid poor interpretability of model. In addition, we introduce a mutual information term into the loss function, which improves the expression capability and accuracy of model. The proposed model, combining the merits of VAE and mutual information, extracts features for multivariate time-series data from a new perspective. The Lorenz system and Beijing air quality time series are used to test performance of the proposed model and comparative models. Results show that the proposed model is superior to other similar models in terms of accuracy and expression capability of latent space.

Dual Heuristic Programming for Optimal Control of Continuous-Time Nonlinear Systems Using Single Echo State Network

This article presents an improved online adaptive dynamic programming (ADP) algorithm to solve the optimal control problem of continuous-time nonlinear systems with infinite horizon cost. The Hamilton-Jacobi-Bellman (HJB) equation is iteratively approximated by a novel critic-only structure which is constructed using the single echo state network (ESN). Inspired by the dual heuristic programming (DHP) technique, ESN is designed to approximate the costate function, then to derive the optimal controller. As the ESN is characterized by the echo state property (ESP), it is proved that the ESN can successfully approximate the solution to the HJB equation. Besides, to eliminate the requirement for the initial admissible control, a new weight tuning law is designed by adding an alternative condition. The stability of the closed-loop optimal control system and the convergence of the out weights of the ESN are guaranteed by using the Lyapunov theorem in the sense of uniformly ultimately bounded (UUB). Two simulation examples, including linear system and nonlinear system, are given to illustrate the availability and effectiveness of the proposed approach by comparing it with the polynomial neural-network scheme.

Unsupervised fNIRS feature extraction with CAE and ESN autoencoder for driver cognitive load classification

Objective. Understanding the cognitive load of drivers is crucial for road safety. Brain sensing has the potential to provide an objective measure of driver cognitive load. We aim to develop an advanced machine learning framework for classifying driver cognitive load using functional near-infrared spectroscopy (fNIRS).Approach. We conducted a study using fNIRS in a driving simulator with theN-back task used as a secondary task to impart structured cognitive load on drivers. To classify different driver cognitive load levels, we examined the application of convolutional autoencoder (CAE) and Echo State Network (ESN) autoencoder for extracting features from fNIRS.Main results. By using CAE, the accuracies for classifying two and four levels of driver cognitive load with the 30 s window were 73.25% and 47.21%, respectively. The proposed ESN autoencoder achieved state-of-art classification results for group-level models without window selection, with accuracies of 80.61% and 52.45% for classifying two and four levels of driver cognitive load.Significance. This work builds a foundation for using fNIRS to measure driver cognitive load in real-world applications. Also, the results suggest that the proposed ESN autoencoder can effectively extract temporal information from fNIRS data and can be useful for other fNIRS data classification tasks.

Convolutional Multitimescale Echo State Network

As efficient recurrent neural network (RNN) models, echo state networks (ESNs) have attracted widespread attention and been applied in many application domains in the last decade. Although they have achieved great success in modeling time series, a single ESN may have difficulty in capturing the multitimescale structures that naturally exist in temporal data. In this paper, we propose the convolutional multitimescale ESN (ConvMESN), which is a novel training-efficient model for capturing multitimescale structures and multiscale temporal dependencies of temporal data. In particular, a multitimescale memory encoder is constructed with a multireservoir structure, in which different reservoirs have recurrent connections with different skip lengths (or time spans). By collecting all past echo states in each reservoir, this multireservoir structure encodes the history of a time series as nonlinear multitimescale echo state representations (MESRs). Our visualization analysis verifies that the MESRs provide better discriminative features for time series. Finally, multiscale temporal dependencies of MESRs are learned by a convolutional layer. By leveraging the multitimescale reservoirs followed by a convolutional learner, the ConvMESN has not only efficient memory encoding ability for temporal data with multitimescale structures but also strong learning ability for complex temporal dependencies. Furthermore, the training-free reservoirs and the single convolutional layer provide high-computational efficiency for the ConvMESN to model complex temporal data. Extensive experiments on 18 multivariate time series (MTS) benchmark datasets and 3 skeleton-based action recognition datasets demonstrate that the ConvMESN captures multitimescale dynamics and outperforms existing methods.

Using machine learning to predict statistical properties of non-stationary dynamical processes: System climate,regime transitions, and the effect of stochasticity

We develop and test machine learning techniques for successfully using past state time series data and knowledge of a time-dependent system parameter to predict the evolution of the "climate" associated with the long-term behavior of a non-stationary dynamical system, where the non-stationary dynamical system is itself unknown. By the term climate, we mean the statistical properties of orbits rather than their precise trajectories in time. By the term non-stationary, we refer to systems that are, themselves, varying with time. We show that our methods perform well on test systems predicting both continuous gradual climate evolution as well as relatively sudden climate changes (which we refer to as "regime transitions"). We consider not only noiseless (i.e., deterministic) non-stationary dynamical systems, but also climate prediction for non-stationary dynamical systems subject to stochastic forcing (i.e., dynamical noise), and we develop a method for handling this latter case. The main conclusion of this paper is that machine learning has great promise as a new and highly effective approach to accomplishing data driven prediction of non-stationary systems.

An Experimental Review on Deep Learning Architectures for Time Series Forecasting

In recent years, deep learning techniques have outperformed traditional models in many machine learning tasks. Deep neural networks have successfully been applied to address time series forecasting problems, which is a very important topic in data mining. They have proved to be an effective solution given their capacity to automatically learn the temporal dependencies present in time series. However, selecting the most convenient type of deep neural network and its parametrization is a complex task that requires considerable expertise. Therefore, there is a need for deeper studies on the suitability of all existing architectures for different forecasting tasks. In this work, we face two main challenges: a comprehensive review of the latest works using deep learning for time series forecasting and an experimental study comparing the performance of the most popular architectures. The comparison involves a thorough analysis of seven types of deep learning models in terms of accuracy and efficiency. We evaluate the rankings and distribution of results obtained with the proposed models under many different architecture configurations and training hyperparameters. The datasets used comprise more than 50,000 time series divided into 12 different forecasting problems. By training more than 38,000 models on these data, we provide the most extensive deep learning study for time series forecasting. Among all studied models, the results show that long short-term memory (LSTM) and convolutional networks (CNN) are the best alternatives, with LSTMs obtaining the most accurate forecasts. CNNs achieve comparable performance with less variability of results under different parameter configurations, while also being more efficient.

A reservoir computing approach for forecasting and regenerating both dynamical and time-delay controlled financial system behavior

Significant research in reservoir computing over the past two decades has revived interest in recurrent neural networks. Owing to its ingrained capability of performing high-speed and low-cost computations this has become a panacea for multi-variate complex systems having non-linearity within their relationships. Modelling economic and financial trends has always been a challenging task owing to their volatile nature and no linear dependence on associated influencers. Prior studies aimed at effectively forecasting such financial systems, but, always left a visible room for optimization in terms of cost, speed and modelling complexities. Our work employs a reservoir computing approach complying to echo-state network principles, along with varying strengths of time-delayed feedback to model a complex financial system. The derived model is demonstrated to act robustly towards influence of trends and other fluctuating parameters by effectively forecasting long-term system behavior. Moreover, it also re-generates the financial system unknowns with a high degree of accuracy when only limited future data is available, thereby, becoming a reliable feeder for any long-term decision making or policy formulations.

Echo Memory-Augmented Network for time series classification

Echo State Networks (ESNs) are efficient recurrent neural networks (RNNs) which have been successfully applied to time series modeling tasks. However, ESNs are unable to capture the history information far from the current time step, since the echo state at the present step of ESNs mostly impacted by the previous one. Thus, ESN may have difficulty in capturing the long-term dependencies of temporal data. In this paper, we propose an end-to-end model named Echo Memory-Augmented Network (EMAN) for time series classification. An EMAN consists of an echo memory-augmented encoder and a multi-scale convolutional learner. First, the time series is fed into the reservoir of an ESN to produce the echo states, which are all collected into an echo memory matrix along with the time steps. After that, we design an echo memory-augmented mechanism employing the sparse learnable attention to the echo memory matrix to obtain the Echo Memory-Augmented Representations (EMARs). In this way, the input time series is encoded into the EMARs with enhancing the temporal memory of the ESN. We then use multi-scale convolutions with the max-over-time pooling to extract the most discriminative features from the EMARs. Finally, a fully-connected layer and a softmax layer calculate the probability distribution on categories. Experiments conducted on extensive time series datasets show that EMAN is state-of-the-art compared to existing time series classification methods. The visualization analysis also demonstrates the effectiveness of enhancing the temporal memory of the ESN.

Hybrid Model for Short-Term Water Demand Forecasting Based on Error Correction Using Chaotic Time Series

Short-term water demand forecasting plays an important role in smart management and real-time simulation of water distribution systems (WDSs). This paper proposes a hybrid model for the short-term forecasting in the horizon of one day with 15 min time steps, which improves the forecasting accuracy by adding an error correction module to the initial forecasting model. The initial forecasting model is firstly established based on the least square support vector machine (LSSVM), the errors time series obtained by comparing the observed values and the initial forecasted values is next transformed into chaotic time series, and then the error correction model is established by the LSSVM method to forecast errors at the next time step. The hybrid model is tested on three real-world district metering areas (DMAs) in Beijing, China, with different demand patterns. The results show that, with the help of the error correction module, the hybrid model reduced the mean absolute percentage error (MAPE) of forecasted demand from (5.64%, 4.06%, 5.84%) to (4.84%, 3.15%, 3.47%) for the three DMAs, compared with using LSSVM without error correction. Therefore, the proposed hybrid model provides a better solution for short-term water demand forecasting on the tested cases.

Evolving Local Plasticity Rules for Synergistic Learning in Echo State Networks

Existing synaptic plasticity rules for optimizing the connections between neurons within the reservoir of echo state networks (ESNs) remain to be global in that the same type of plasticity rule with the same parameters is applied to all neurons. However, this is biologically implausible and practically inflexible for learning the structures in the input signals, thereby limiting the learning performance of ESNs. In this paper, we propose to use local plasticity rules that allow different neurons to use different types of plasticity rules and different parameters, which are achieved by optimizing the parameters of the local plasticity rules using the evolution strategy (ES) with covariance matrix adaptation (CMA-ES). We show that evolving neural plasticity will result in a synergistic learning of different plasticity rules, which plays an important role in improving the learning performance. Meanwhile, we show that the local plasticity rules can effectively alleviate synaptic interferences in learning the structure in sensory inputs. The proposed local plasticity rules are compared with a number of the state-of-the-art ESN models and the canonical ESN using a global plasticity rule on a set of widely used prediction and classification benchmark problems to demonstrate its competitive learning performance.

Feature Extraction of Surface Electromyography Based on Improved Small-World Leaky Echo State Network

Surface electromyography (sEMG) is an electrophysiological reflection of skeletal muscle contractile activity that can directly reflect neuromuscular activity. It has been a matter of research to investigate feature extraction methods of sEMG signals. In this letter, we propose a feature extraction method of sEMG signals based on the improved small-world leaky echo state network (ISWLESN). The reservoir of leaky echo state network (LESN) is connected by a random network. First, we improved the reservoir of the echo state network (ESN) by these networks and used edge-added probability to improve these networks. That idea enhances the adaptability of the reservoir, the generalization ability, and the stability of ESN. Then we obtained the output weight of the network through training and used it as features. We recorded the sEMG signals during different activities: falling, walking, sitting, squatting, going upstairs, and going downstairs. Afterward, we extracted corresponding features by ISWLESN and used principal component analysis for dimension reduction. At the end, scatter plot, the class separability index, and the Davies-Bouldin index were used to assess the performance of features. The results showed that the ISWLESN clustering performance was better than those of LESN and ESN. By support vector machine, it was also revealed that the performance of ISWLESN for classifying the activities was better than those of ESN and LESN.

Modeling the Nonlinearity of Sea Level Oscillations in the Malaysian Coastal Areas Using Machine Learning Algorithms

The estimation of an increase in sea level with sufficient warning time is important in low-lying regions, especially in the east coast of Peninsular Malaysia (ECPM). This study primarily aims to investigate the validity and effectiveness of the support vector machine (SVM) and genetic programming (GP) models for predicting the monthly mean sea level variations and comparing their prediction accuracies in terms of the model performances. The input dataset was obtained from Kerteh, Tioman Island, and Tanjung Sedili in Malaysia from January 2007 to December 2017 to predict the sea levels for five different time periods (1, 5, 10, 20, and 40 years). Further, the SVM and GP models are subjected to preprocessing to obtain optimal performance. The tuning parameters are generalized for the optimal input designs (SVM2 and GP2), and the results denote that SVM2 outperforms GP with R of 0.81 and 0.86 during the training and testing periods, respectively, at the study locations. However, GP can provide values of 0.71 and 0.79 for training and testing, respectively, at the study locations. The results show precise predictions of the monthly mean sea level, denoting the promising potential of the used models for performing sea level data analysis.

Interval Type-2 Fuzzy Neural Networks for Chaotic Time Series Prediction: A Concise Overview

Chaotic time series widely exists in nature and society (e.g., meteorology, physics, economics, etc.), which usually exhibits seemingly unpredictable features due to its inherent nonstationary and high complexity. Thankfully, multifarious advanced approaches have been developed to tackle the prediction issues, such as statistical methods, artificial neural networks (ANNs), and support vector machines. Among them, the interval type-2 fuzzy neural network (IT2FNN), which is a synergistic integration of fuzzy logic systems and ANNs, has received wide attention in the field of chaotic time series prediction. This paper begins with the structural features and superiorities of IT2FNN. Moreover, chaotic characters identification and phase-space reconstruction matters for prediction are presented. In addition, we also offer a comprehensive review of state-of-the-art applications of IT2FNN, with an emphasis on chaotic time series prediction and summarize their main contributions as well as some hardware implementations for computation speedup. Finally, this paper trends and extensions of this field, along with an outlook of future challenges are revealed. The primary objective of this paper is to serve as a tutorial or referee for interested researchers to have an overall picture on the current developments and identify their potential research direction to further investigation.

Cholesky Factorization Based Online Sequential Extreme Learning Machines with Persistent Regularization and Forgetting Factor

The online sequential extreme learning machine with persistent regularization and forgetting factor (OSELM-PRFF) can avoid potential singularities or ill-posed problems of online sequential regularized extreme learning machines with forgetting factors (FR-OSELM), and is particularly suitable for modelling in non-stationary environments. However, existing algorithms for OSELM-PRFF are time-consuming or unstable in certain paradigms or parameters setups. This paper presents a novel algorithm for OSELM-PRFF, named “Cholesky factorization based” OSELM-PRFF (CF-OSELM-PRFF), which recurrently constructs an equation for extreme learning machine and efficiently solves the equation via Cholesky factorization during every cycle. CF-OSELM-PRFF deals with timeliness of samples by forgetting factor, and the regularization term in its cost function works persistently. CF-OSELM-PRFF can learn data one-by-one or chunk-by-chunk with a fixed or varying chunk size. Detailed performance comparisons between CF-OSELM-PRFF and relevant approaches are carried out on several regression problems. The numerical simulation results show that CF-OSELM-PRFF demonstrates higher computational efficiency than its counterparts, and can yield stable predictions.

Time series classification with Echo Memory Networks

Echo state networks (ESNs) are randomly connected recurrent neural networks (RNNs) that can be used as a temporal kernel for modeling time series data, and have been successfully applied on time series prediction tasks. Recently, ESNs have been applied to time series classification (TSC) tasks. However, previous ESN-based classifiers involve either training the model by predicting the next item of a sequence, or predicting the class label at each time step. The former is essentially a predictive model adapted from time series prediction work, rather than a model designed specifically for the classification task. The latter approach only considers local patterns at each time step and then averages over the classifications. Hence, rather than selecting the most discriminating sections of the time series, this approach will incorporate non-discriminative information into the classification, reducing accuracy. In this paper, we propose a novel end-to-end framework called the Echo Memory Network (EMN) in which the time series dynamics and multi-scale discriminative features are efficiently learned from an unrolled echo memory using multi-scale convolution and max-over-time pooling. First, the time series data are projected into the high dimensional nonlinear space of the reservoir and the echo states are collected into the echo memory matrix, followed by a single multi-scale convolutional layer to extract multi-scale features from the echo memory matrix. Max-over-time pooling is used to maintain temporal invariance and select the most important local patterns. Finally, a fully-connected hidden layer feeds into a softmax layer for classification. This architecture is applied to both time series classification and human action recognition datasets. For the human action recognition datasets, we divide the action data into five different components of the human body, and propose two spatial information fusion strategies to integrate the spatial information over them. With one training-free recurrent layer and only one layer of convolution, the EMN is a very efficient end-to-end model, and ranks first in overall classification ability on 55 TSC benchmark datasets and four 3D skeleton-based human action recognition tasks.

Robust manifold broad learning system for large-scale noisy chaotic time series prediction: A perturbation perspective

Noises and outliers commonly exist in dynamical systems because of sensor disturbations or extreme dynamics. Thus, the robustness and generalization capacity are of vital importance for system modeling. In this paper, the robust manifold broad learning system(RM-BLS) is proposed for system modeling and large-scale noisy chaotic time series prediction. Manifold embedding is utilized for chaotic system evolution discovery. The manifold representation is randomly corrupted by perturbations while the features not related to low-dimensional manifold embedding are discarded by feature selection. It leads to a robust learning paradigm and achieves better generalization performance. We also develop an efficient solution for Stiefel manifold optimization, in which the orthogonal constraints are maintained by Cayley transformation and curvilinear search algorithm. Furthermore, we discuss the common thoughts between random perturbation approximation and other mainstream regularization methods. We also prove the equivalence between perturbations to manifold embedding and Tikhonov regularization. Simulation results on large-scale noisy chaotic time series prediction illustrates the robustness and generalization performance of our method.

Consensus Maneuvering for a Class of Nonlinear Multivehicle Systems in Strict-Feedback Form

In this paper, a consensus maneuvering problem for nonlinear multivehicle systems in strict-feedback form is investigated. The consensus maneuvering problem includes a geometric task and a dynamic task. The geometric task means that all trajectories of follower vehicles converge to a parameterized path. The dynamic task is to drive the system to satisfy a desired dynamic assignment. A consensus maneuvering controller is developed for each vehicle based on a modular design approach. First, an estimator module is designed based on an echo state network, which is used to estimate uncertain nonlinearities. Then, a controller module is designed based on a modified dynamic surface control method through the use of a second-order nonlinear tracking differentiator. Finally, a path update law is designed based on a distributed maneuvering error feedback and a filtering scheme. The proposed controller is distributed in the sense that the path information is accessed by a small number of follower vehicles only. The stability of the closed-loop system cascaded by the estimator module and the controller module is analyzed based on input-to-state stability theory and cascade theory. Simulation results are provided to demonstrate the efficacy of the proposed consensus maneuvering controllers for uncertain nonlinear strict-feedback systems.