Time series prediction using deep echo state networks

Novel efficient reservoir computing methodologies for regular and irregular time series classification

Time series is a data structure prevalent in a wide range of fields such as healthcare, finance and meteorology. It goes without saying that analyzing time series data holds the key to gaining insight into our day-to-day observations. Among the vast spectrum of time series analysis, time series classification offers the unique opportunity to classify the sequences into their respective categories for the sake of automated detection. To this end, two types of mainstream approaches, recurrent neural networks and distance-based methods, have been commonly employed to address this specific problem. Despite their enormous success, methods like Long Short-Term Memory networks typically require high computational resources. It is largely as a consequence of the nature of backpropagation, driving the search for some backpropagation-free alternatives. Reservoir computing is an instance of recurrent neural networks that is known for its efficiency in processing time series sequences. Therefore, in this article, we will develop two reservoir computing based methods that can effectively deal with regular and irregular time series with minimal computational cost, both while achieving a desirable level of classification accuracy.

Variability of echo state network prediction horizon for partially observed dynamical systems

Study of dynamical systems using partial state observation is an important problem due to its applicability to many real-world systems. We address the problem by studying an echo state network (ESN) framework with partial state input with partial or full state output. Application to the Lorenz system and Chua's oscillator (both numerically simulated and experimental systems) demonstrate the effectiveness of our method. We show that the ESN, as an autonomous dynamical system, is capable of making short-term predictions up to a few Lyapunov times. However, the prediction horizon has high variability depending on the initial condition-an aspect that we explore in detail using the distribution of the prediction horizon. Further, using a variety of statistical metrics to compare the long-term dynamics of the ESN predictions with numerically simulated or experimental dynamics and observed similar results, we show that the ESN can effectively learn the system's dynamics even when trained with noisy numerical or experimental data sets. Thus, we demonstrate the potential of ESNs to serve as cheap surrogate models for simulating the dynamics of systems where complete observations are unavailable.

Time-series benchmarks based on frequency features for fair comparative evaluation

Time-series prediction and imputation receive lots of attention in academic and industrial areas. Machine learning methods have been developed for specific time-series scenarios; however, it is difficult to evaluate the effectiveness of a certain method on other new cases. In the perspective of frequency features, a comprehensive benchmark for time-series prediction is designed for fair evaluation. A prediction problem generation process, composed of the finite impulse response filter-based approach and problem setting module, is adopted to generate the NCAA2022 dataset, which includes 16 prediction problems. To reduce the computational burden, the filter parameters matrix is divided into sub-matrices. The discrete Fourier transform is introduced to analyze the frequency distribution of transformed results. In addition, a baseline experiment further reflects the benchmarking capability of NCAA2022 dataset.

An octonion-based nonlinear echo state network for speech emotion recognition in Metaverse

While the Metaverse is becoming a popular trend and drawing much attention from academia, society, and businesses, processing cores used in its infrastructures need to be improved, particularly in terms of signal processing and pattern recognition. Accordingly, the speech emotion recognition (SER) method plays a crucial role in creating the Metaverse platforms more usable​ and enjoyable for its users. However, existing SER methods continue to be plagued by two significant problems in the online environment. The shortage of adequate engagement and customization between avatars and users is recognized as the first issue and the second problem is related to the complexity of SER problems in the Metaverse as we face people and their digital twins or avatars. This is why developing efficient machine learning (ML) techniques specified for hypercomplex signal processing is essential to enhance the impressiveness and tangibility of the Metaverse platforms. As a solution, echo state networks (ESNs), which are an ML powerful tool for SER, can be an appropriate technique to enhance the Metaverse's foundations in this area. Nevertheless, ESNs have some technical issues restricting them from a precise and reliable analysis, especially in the aspect of high-dimensional data. The most significant limitation of these networks is the high memory consumption caused by their reservoir structure in face of high-dimensional signals. To solve all problems associated with ESNs and their application in the Metaverse, we have come up with a novel structure for ESNs empowered by octonion algebra called NO2GESNet. Octonion numbers have eight dimensions, compactly display high-dimensional data, and improve the network precision and performance in comparison to conventional ESNs. The proposed network also solves the weaknesses of the ESNs in the presentation of the higher-order statistics to the output layer by equipping it with a multidimensional bilinear filter. Three comprehensive scenarios to use the proposed network in the Metaverse have been designed and analyzed, not only do they show the accuracy and performance of the proposed approach, but also the ways how SER can be employed in the Metaverse platforms.

The Prediction of Enterprise Stock Change Trend by Deep Neural Network Model

This study aims to accurately predict the changing trend of stocks in stock trading so that company investors can obtain higher returns. In building a financial forecasting model, historical data and learned parameters are used to predict future stock prices. Firstly, the relevant theories of stock forecasting are discussed, and problems in stock forecasting are raised. Secondly, the inadequacies of deep neural network (DNN) models are discussed. A prediction trend model of enterprise stock is established based on long short-term memory (LSTM). The uniqueness and innovation lie in using the stock returns of Bank of China securities in 2022 as the training data set. LSTM prediction models are used to perform error analysis on company data training. The 20-day change trend of the company's stock returns under different models is predicted and analyzed. The results show that as the number of iterations increases, the loss rate of the LSTM training curve keeps decreasing until 0. The average return price of the LSTM prediction model is 14.01. This figure is closest to the average real return price of 13.89. Through the forecast trend analysis under different models, LSTM predicts that the stock change trend of the enterprise model is closest to the changing trend of the actual earnings price. The prediction accuracy is better than other prediction models. In addition, this study explores the characteristics of high noise and complexity of corporate stock time series, designs a DNN prediction model, and verifies the feasibility of the LSTM model to predict corporate stock changes with high accuracy.

Broad Echo State Network with Reservoir Pruning for Nonstationary Time Series Prediction

The nonstationary time series is generated in various natural and man-made systems, of which the prediction is vital for advanced control and management. The neural networks have been explored in the time series prediction, but the problem remains in modeling the data’s nonstationary and nonlinear features. Referring to the time series feature and network property, a novel network is designed with dynamic optimization of the model structure. Firstly, the echo state network (ESN) is introduced into the broad learning system (BLS). The broad echo state network (BESN) can increase the training efficiency with the incremental learning algorithm by removing the error backpropagation. Secondly, an optimization algorithm is proposed to reduce the redundant information in the training process of BESN units. The number of neurons in BESN with a fixed step size is pruned according to the contribution degree. Finally, the improved network is applied in the different datasets. The tests in the time series of natural and man-made systems prove that the proposed network performs better on the nonstationary time series prediction than the typical methods, including the ESN, BLS, and recurrent neural network.

A Deep Learning-based approach for forecasting off-gas production and consumption in the blast furnace

This article presents the application of a recent neural network topology known as the deep echo state network to the prediction and modeling of strongly nonlinear systems typical of the process industry. The article analyzes the results by introducing a comparison with one of the most common and efficient topologies, the long short-term memories, in order to highlight the strengths and weaknesses of a reservoir computing approach compared to one currently considered as a standard of recurrent neural network. As benchmark application, two specific processes common in the integrated steelworks are selected, with the purpose of forecasting the future energy exchanges and transformations. The procedures of training, validation and test are based on data analysis, outlier detection and reconciliation and variable selection starting from real field industrial data. The analysis of results shows the effectiveness of deep echo state networks and their strong forecasting capabilities with respect to standard recurrent methodologies both in terms of training procedures and accuracy.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00521-021-05984-x.