Adaptive Elastic Echo State Network for Multivariate Time Series Prediction

A Dynamic Systems Approach to Modeling Human-Machine Rhythm Interaction

Rhythm is an inherent aspect of human behavior, present from infancy and embedded in cultural practices. At the core of rhythm perception lies meter anticipation, a spontaneous process in the human brain that typically occurs before actual beats. This anticipation can be framed as a time series prediction problem. From the perspective of human embodied system behavior, although many models have been developed for time series prediction, most prioritize accuracy over biological realism, contrasting with the natural imprecision of human internal clocks. Neuroscientific evidence, such as infants' natural meter synchronization, underscores the need for biologically plausible models. Therefore, we propose a neuron oscillator-based dynamic system that simulates human behavior during meter perception. The model introduces two tunable parameters for local and global adjustments, fine-tuning the oscillation combinations to emulate human-like rhythmic behavior. The experiments are conducted under three common scenarios encountered during human-machine interaction, demonstrating that the proposed model can exhibit human-like reactions. Additionally, experiments involving human-machine and interhuman interactions show that the model successfully replicates real-world rhythmic behavior, advancing toward more natural and synchronized human-machine rhythm interaction.

Machine Learning-Driven Metabolic Syndrome Prediction: An International Cohort Validation Study

Background/Objectives: This study aimed to develop and validate a machine learning (ML)-based metabolic syndrome (MetS) risk prediction model. Methods: We examined data from 6155 participants of the China Health and Retirement Longitudinal Study (CHARLS) in 2011. The LASSO regression feature selection identified the best MetS predictors. Nine ML-based algorithms were adopted to build predictive models. The model performance was validated using cohort data from the Korea National Health and Nutrition Examination Survey (KNHANES) (n = 5297), the United Kingdom (UK) Biobank (n = 218,781), and the National Health and Nutrition Examination Survey (NHANES) (n = 2549). Results: The multilayer perceptron (MLP)-based model performed best in the CHARLS cohort (AUC = 0.8908; PRAUC = 0.8073), the logistic model in the KNHANES cohort (AUC = 0.9101, PRAUC = 0.8116), the xgboost model in the UK Biobank cohort (AUC = 0.8556, PRAUC = 0.6246), and the MLP model in the NHANES cohort (AUC = 0.9055, PRAUC = 0.8264). Conclusions: Our MLP-based model has the potential to serve as a clinical application for detecting MetS in different populations.

A lightweight method of integrated local load forecasting and control of edge computing in active distribution networks

The strong resource constraints of edge-computing devices and the dynamic evolution of load characteristics put forward higher requirements for forecasting methods of active distribution networks. This paper proposes a lightweight adaptive ensemble learning method for local load forecasting and predictive control of active distribution networks based on edge computing in resource constrained scenarios. First, the adaptive sparse integration method is proposed to reduce the model scale. Then, the auto-encoder is introduced to downscale the model variables to further reduce computation time and storage overhead. An adaptive correction method is proposed to maintain the adaptability. Finally, a multi-timescale predictive control method for the edge side is established, which realizes the collaboration of local load forecasting and control. All cases can be deployed on an actual edge-computing device. Compared to other benchmark methods and the existing researches, the proposed method can minimize the model complexity without reducing the forecasting accuracy.

fastESN: Fast Echo State Network

Echo state networks (ESNs) are reservoir computing-based recurrent neural networks widely used in pattern analysis and machine intelligence applications. In order to achieve high accuracy with large model capacity, ESNs usually contain a large-sized internal layer (reservoir), making the evaluation process too slow for some applications. In this work, we speed up the evaluation of ESN by building a reduced network called the fast ESN (fastESN) and achieve an ESN evaluation complexity independent of the original ESN size for the first time. FastESN is generated using three techniques. First, the high-dimensional state of the original ESN is approximated by a low-dimensional state through proper orthogonal decomposition (POD)-based projection. Second, the activation function evaluation number is reduced through the discrete empirical interpolation method (DEIM). Third, we show the directly generated fastESN has instability problems and provide a stabilization scheme as a solution. Through experiments on four popular benchmarks, we show that fastESN is able to accelerate the sparse storage-based ESN evaluation with a high parameter compression ratio and a fast evaluation speed.

Hierarchical Echo State Network With Sparse Learning: A Method for Multidimensional Chaotic Time Series Prediction

Echo state network (ESN), a type of special recurrent neural network with a large-scale randomly fixed hidden layer (called a reservoir) and an adaptable linear output layer, has been widely employed in the field of time series analysis and modeling. However, when tackling the problem of multidimensional chaotic time series prediction, due to the randomly generated rules for input and reservoir weights, not only the representation of valuable variables is enriched but also redundant and irrelevant information is accumulated inevitably. To remove the redundant components, reduce the approximate collinearity among echo-state information, and improve the generalization and stability, a new method called hierarchical ESN with sparse learning (HESN-SL) is proposed. The HESN-SL mines and captures the latent evolution patterns hidden from the dynamic system by means of layer-by-layer processing in stacked reservoirs, and leverage monotone accelerated proximal gradient algorithm to train a sparse output layer with variable selection capability. Meanwhile, we further prove that the HESN-SL satisfies the echo state property, which guarantees the stability and convergence of the proposed model when applied to time series prediction. Experimental results on two synthetic chaotic systems and a real-world meteorological dataset illustrate the proposed HESN-SL outperforms both original ESN and existing hierarchical ESN-based models for multidimensional chaotic time series prediction.

Reservoir Dynamic Interpretability for Time Series Prediction: A Permutation Entropy View

An echo state network (ESN) is an efficient recurrent neural network (RNN) that is widely used in time series prediction tasks due to its simplicity and low training cost. However, the "black-box" nature of reservoirs hinders the development of ESN. Although a large number of studies have concentrated on reservoir interpretability, the perspective of reservoir modeling is relatively single, and the relationship between reservoir richness and reservoir projection capacity has not been effectively established. To tackle this problem, a novel reservoir interpretability framework based on permutation entropy (PE) theory is proposed in this paper. In structure, this framework consists of reservoir state extraction, PE modeling, and PE analysis. Based on these, the instantaneous reservoir states and neuronal time-varying states are extracted, which are followed by phase space reconstruction, sorting, and entropy calculation. Firstly, the obtained instantaneous state entropy (ISE) and global state entropy (GSE) can measure reservoir richness for interpreting good reservoir projection capacity. On the other hand, the multiscale complexity-entropy analysis of global and neuron-level reservoir states is performed to reveal more detailed dynamics. Finally, the relationships between ESN performance and reservoir dynamic are investigated via Pearson correlation, considering different prediction steps and time scales. Experimental evaluations on several benchmarks and real-world datasets demonstrate the effectiveness and superiority of the proposed reservoir interpretability framework.

Autonomous Tracking and State Estimation With Generalized Group Lasso

We address the problem of autonomous tracking and state estimation for marine vessels, autonomous vehicles, and other dynamic signals under a (structured) sparsity assumption. The aim is to improve the tracking and estimation accuracy with respect to the classical Bayesian filters and smoothers. We formulate the estimation problem as a dynamic generalized group Lasso problem and develop a class of smoothing-and-splitting methods to solve it. The Levenberg-Marquardt iterated extended Kalman smoother-based multiblock alternating direction method of multipliers (LM-IEKS-mADMMs) algorithms are based on the alternating direction method of multipliers (ADMMs) framework. This leads to minimization subproblems with an inherent structure to which three new augmented recursive smoothers are applied. Our methods can deal with large-scale problems without preprocessing for dimensionality reduction. Moreover, the methods allow one to solve nonsmooth nonconvex optimization problems. We then prove that under mild conditions, the proposed methods converge to a stationary point of the optimization problem. By simulated and real-data experiments, including multisensor range measurement problems, marine vessel tracking, autonomous vehicle tracking, and audio signal restoration, we show the practical effectiveness of the proposed methods.

Design of PID Controller Based on Echo State Network With Time-Varying Reservoir Parameter

In this article, a new design method based on the echo state network with time-varying reservoir parameter (TVRP-ESN) is proposed to optimize the proportional-integral-derivative (PID) controller parameters for a class of discrete-time systems with time delay. The TVRP-ESN can quickly obtain the PID controller parameters to meet the control performance of the system. According to the network learning and approximation ability of TVRP-ESN, the output weights and the reservoir parameters of TVRP-ESN can be synchronously updated, and then the TVRP-ESN can improve the convergence speed of determining the PID controller parameters. In order to update the output weights and the reservoir parameters of TVRP-ESN, the partial derivative of the system output error is used. Three simulation examples are used to show the effectiveness of the proposed method.

Flexible, non-parametric modeling using regularized neural networks

Non-parametric, additive models are able to capture complex data dependencies in a flexible, yet interpretable way. However, choosing the format of the additive components often requires non-trivial data exploration. Here, as an alternative, we propose PrAda-net, a one-hidden-layer neural network, trained with proximal gradient descent and adaptive lasso. PrAda-net automatically adjusts the size and architecture of the neural network to reflect the complexity and structure of the data. The compact network obtained by PrAda-net can be translated to additive model components, making it suitable for non-parametric statistical modelling with automatic model selection. We demonstrate PrAda-net on simulated data, where we compare the test error performance, variable importance and variable subset identification properties of PrAda-net to other lasso-based regularization approaches for neural networks. We also apply PrAda-net to the massive U.K. black smoke data set, to demonstrate how PrAda-net can be used to model complex and heterogeneous data with spatial and temporal components. In contrast to classical, statistical non-parametric approaches, PrAda-net requires no preliminary modeling to select the functional forms of the additive components, yet still results in an interpretable model representation.

Incorporation of causality structures to complex network analysis of time-varying behaviour of multivariate time series

This paper presents a new methodology for characterising the evolving behaviour of the time-varying causality between multivariate time series, from the perspective of change in the structure of the causality pattern. We propose that such evolutionary behaviour should be tracked by means of a complex network whose nodes are causality patterns and edges are transitions between those patterns of causality. In our new methodology each edge has a weight that includes the frequency of the given transition and two metrics relating to the gross and net structural change in causality pattern, which we call [Formula: see text] and [Formula: see text]. To characterise aspects of the behaviour within this network, five approaches are presented and motivated. To act as a demonstration of this methodology an application of sample data from the international oil market is presented. This example illustrates how our new methodology is able to extract information about evolving causality behaviour. For example, it reveals non-random time-varying behaviour that favours transitions resulting in predominantly similar causality patterns, and it discovers clustering of similar causality patterns and some transitional behaviour between these clusters. The example illustrates how our new methodology supports the inference that the evolution of causality in the system is related to the addition or removal of a few causality links, primarily keeping a similar causality pattern, and that the evolution is not related to some other measure such as the overall number of causality links.

Convolutional Multitimescale Echo State Network

As efficient recurrent neural network (RNN) models, echo state networks (ESNs) have attracted widespread attention and been applied in many application domains in the last decade. Although they have achieved great success in modeling time series, a single ESN may have difficulty in capturing the multitimescale structures that naturally exist in temporal data. In this paper, we propose the convolutional multitimescale ESN (ConvMESN), which is a novel training-efficient model for capturing multitimescale structures and multiscale temporal dependencies of temporal data. In particular, a multitimescale memory encoder is constructed with a multireservoir structure, in which different reservoirs have recurrent connections with different skip lengths (or time spans). By collecting all past echo states in each reservoir, this multireservoir structure encodes the history of a time series as nonlinear multitimescale echo state representations (MESRs). Our visualization analysis verifies that the MESRs provide better discriminative features for time series. Finally, multiscale temporal dependencies of MESRs are learned by a convolutional layer. By leveraging the multitimescale reservoirs followed by a convolutional learner, the ConvMESN has not only efficient memory encoding ability for temporal data with multitimescale structures but also strong learning ability for complex temporal dependencies. Furthermore, the training-free reservoirs and the single convolutional layer provide high-computational efficiency for the ConvMESN to model complex temporal data. Extensive experiments on 18 multivariate time series (MTS) benchmark datasets and 3 skeleton-based action recognition datasets demonstrate that the ConvMESN captures multitimescale dynamics and outperforms existing methods.

Echo Memory-Augmented Network for time series classification

Echo State Networks (ESNs) are efficient recurrent neural networks (RNNs) which have been successfully applied to time series modeling tasks. However, ESNs are unable to capture the history information far from the current time step, since the echo state at the present step of ESNs mostly impacted by the previous one. Thus, ESN may have difficulty in capturing the long-term dependencies of temporal data. In this paper, we propose an end-to-end model named Echo Memory-Augmented Network (EMAN) for time series classification. An EMAN consists of an echo memory-augmented encoder and a multi-scale convolutional learner. First, the time series is fed into the reservoir of an ESN to produce the echo states, which are all collected into an echo memory matrix along with the time steps. After that, we design an echo memory-augmented mechanism employing the sparse learnable attention to the echo memory matrix to obtain the Echo Memory-Augmented Representations (EMARs). In this way, the input time series is encoded into the EMARs with enhancing the temporal memory of the ESN. We then use multi-scale convolutions with the max-over-time pooling to extract the most discriminative features from the EMARs. Finally, a fully-connected layer and a softmax layer calculate the probability distribution on categories. Experiments conducted on extensive time series datasets show that EMAN is state-of-the-art compared to existing time series classification methods. The visualization analysis also demonstrates the effectiveness of enhancing the temporal memory of the ESN.

Anomaly detection of bridge health monitoring data based on KNN algorithm

Pattern recognition technology is applied to bridge health monitoring to solve abnormalities in bridge health monitoring data. Testing is of great significance. For abnormal data detection, this paper proposes a single variable pattern anomaly detection method based on KNN distance and a multivariate time series anomaly detection method based on the covariance matrix and singular value decomposition. This method first performs compression and segmentation on the original data sequence based on important points to obtain multiple time subsequences, then calculates the pattern distance between each time subsequence according to the similarity measure of the time series, and finally selects the abnormal mode according to the KNN method. In this paper, the reliability of the method is verified through experiments. The experimental results in this paper show that the 5/7/9 / 11-nearest neighbors point to a specific number of nodes. Combined with the original time series diagram corresponding to the time zone view, in this paragraph in the time, the value of the temperature sensor No. 6 stays at 32.5 degrees Celsius for up to one month. The detection algorithm controls the number of MTS subsequences through sliding windows and sliding intervals. The execution time is not large, and the value of K is different. Although the calculated results are different, most of the most obvious abnormal sequences can be detected. The results of this paper provide a certain reference value for the study of abnormal detection of bridge health monitoring data.

Recurrent Broad Learning Systems for Time Series Prediction

The broad learning system (BLS) is an emerging approach for effective and efficient modeling of complex systems. The inputs are transferred and placed in the feature nodes, and then sent into the enhancement nodes for nonlinear transformation. The structure of a BLS can be extended in a wide sense. Incremental learning algorithms are designed for fast learning in broad expansion. Based on the typical BLSs, a novel recurrent BLS (RBLS) is proposed in this paper. The nodes in the enhancement units of the BLS are recurrently connected, for the purpose of capturing the dynamic characteristics of a time series. A sparse autoencoder is used to extract the features from the input instead of the randomly initialized weights. In this way, the RBLS retains the merit of fast computing and fits for processing sequential data. Motivated by the idea of "fine-tuning" in deep learning, the weights in the RBLS can be updated by conjugate gradient methods if the prediction errors are large. We exhibit the merits of our proposed model on several chaotic time series. Experimental results substantiate the effectiveness of the RBLS. For chaotic benchmark datasets, the RBLS achieves very small errors, and for the real-world dataset, the performance is satisfactory.

Interval Type-2 Fuzzy Neural Networks for Chaotic Time Series Prediction: A Concise Overview

Chaotic time series widely exists in nature and society (e.g., meteorology, physics, economics, etc.), which usually exhibits seemingly unpredictable features due to its inherent nonstationary and high complexity. Thankfully, multifarious advanced approaches have been developed to tackle the prediction issues, such as statistical methods, artificial neural networks (ANNs), and support vector machines. Among them, the interval type-2 fuzzy neural network (IT2FNN), which is a synergistic integration of fuzzy logic systems and ANNs, has received wide attention in the field of chaotic time series prediction. This paper begins with the structural features and superiorities of IT2FNN. Moreover, chaotic characters identification and phase-space reconstruction matters for prediction are presented. In addition, we also offer a comprehensive review of state-of-the-art applications of IT2FNN, with an emphasis on chaotic time series prediction and summarize their main contributions as well as some hardware implementations for computation speedup. Finally, this paper trends and extensions of this field, along with an outlook of future challenges are revealed. The primary objective of this paper is to serve as a tutorial or referee for interested researchers to have an overall picture on the current developments and identify their potential research direction to further investigation.

Hybrid Regularized Echo State Network for Multivariate Chaotic Time Series Prediction

Multivariate chaotic time series prediction is a hot research topic, the goal of which is to predict the future of the time series based on past observations. Echo state networks (ESNs) have recently been widely used in time series prediction, but there may be an ill-posed problem for a large number of unknown output weights. To solve this problem, we propose a hybrid regularized ESN, which employs a sparse regression with the L₁/2 regularization and the L₂ regularization to compute the output weights. The L₁/2 penalty shows many attractive properties, such as unbiasedness and sparsity. The L₂ penalty presents appealing ability on shrinking the amplitude of the output weights. After the output weights are calculated, the input weights, internal weights, and output weights are fine-tuning by a Hessian-free optimization method-conjugate gradient backpropagation algorithm. The fine-tuning helps to bubble up the input information toward the output layer. Besides, the largest Lyapunov exponent is used to calculate the predictable horizon of a chaotic time series. Experimental results on benchmark and real-world datasets show that our proposed method is superior to other ESN-based models, as sparser, smaller-absolute-value, and more informative output weights are obtained. All of the predictions within the predictable horizon of the proposed model are accurate.

Spatio-Temporal Interpolated Echo State Network for Meteorological Series Prediction

Spatio-temporal series prediction has attracted increasing attention in the field of meteorology in recent years. The spatial and temporal joint effect makes predictions challenging. Most of the existing spatio-temporal prediction models are computationally complicated. To develop an accurate but easy-to-implement spatio-temporal prediction model, this paper designs a novel spatio-temporal prediction model based on echo state networks. For real-world observed meteorological data with randomness and large changes, we use a cubic spline method to bridge the gaps between the neighboring points, which results in a pleasingly smooth series. The interpolated series is later input into the spatio-temporal echo state networks, in which the spatial coefficients are computed by the elastic-net algorithm. This approach offers automatic selection and continuous shrinkage of the spatial variables. The proposed model provides an intuitive but effective approach to address the interaction of spatial and temporal effects. To demonstrate the practicality of the proposed model, we apply it to predict two real-world datasets: monthly precipitation series and daily air quality index series. Experimental results demonstrate that the proposed model achieves a normalized root-mean-square error of approximately 0.250 on both datasets. Similar results are achieved on the long short-term memory model, but the computation time of our proposed model is considerably shorter. It can be inferred that our proposed neural network model has advantages on predicting meteorological series over other models.

Nonuniform State Space Reconstruction for Multivariate Chaotic Time Series

State space reconstruction is the foundation of chaotic system modeling. Selection of reconstructed variables is essential to the analysis and prediction of multivariate chaotic time series. As most existing state space reconstruction theorems deal with univariate time series, we have presented a novel nonuniform state space reconstruction method using information criterion for multivariate chaotic time series. We derived a new criterion based on low dimensional approximation of joint mutual information for time delay selection, which can be solved efficiently through the use of an intelligent optimization algorithm with low computation complexity. The embedding dimension is determined by conditional entropy, after which the reconstructed variables have relatively strong independence and low redundancy. The scheme, which integrates nonuniform embedding and feature selection, results in better reconstructions for multivariate chaotic systems. Moreover, the proposed nonuniform state space reconstruction method shows good performance in forecasting benchmark and actual multivariate chaotic time series.

Decoding the grasping intention from electromyography during reaching motions

BACKGROUND

Active upper-limb prostheses are used to restore important hand functionalities, such as grasping. In conventional approaches, a pattern recognition system is trained over a number of static grasping gestures. However, training a classifier in a static position results in lower classification accuracy when performing dynamic motions, such as reach-to-grasp. We propose an electromyography-based learning approach that decodes the grasping intention during the reaching motion, leading to a faster and more natural response of the prosthesis.

METHODS AND RESULTS

Eight able-bodied subjects and four individuals with transradial amputation gave informed consent and participated in our study. All the subjects performed reach-to-grasp motions for five grasp types, while the elecromyographic (EMG) activity and the extension of the arm were recorded. We separated the reach-to-grasp motion into three phases, with respect to the extension of the arm. A multivariate analysis of variance (MANOVA) on the muscular activity revealed significant differences among the motion phases. Additionally, we examined the classification performance on these phases. We compared the performance of three different pattern recognition methods; Linear Discriminant Analysis (LDA), Support Vector Machines (SVM) with linear and non-linear kernels, and an Echo State Network (ESN) approach. Our off-line analysis shows that it is possible to have high classification performance above 80% before the end of the motion when with three-grasp types. An on-line evaluation with an upper-limb prosthesis shows that the inclusion of the reaching motion in the training of the classifier importantly improves classification accuracy and enables the detection of grasp intention early in the reaching motion.

CONCLUSIONS

This method offers a more natural and intuitive control of prosthetic devices, as it will enable controlling grasp closure in synergy with the reaching motion. This work contributes to the decrease of delays between the user's intention and the device response and improves the coordination of the device with the motion of the arm.