Multiobjective Learning in the Model Space for Time Series Classification

Delayed self-feedback echo state network for long-term dynamics of hyperchaotic systems

Analyzing the long-term behavior of hyperchaotic systems poses formidable challenges in the field of nonlinear science. This paper proposes a data-driven model called the delayed self-feedback echo state network (self-ESN) specifically designed for the evolution behavior of hyperchaotic systems. Self-ESN incorporates a delayed self-feedback term into the dynamic equation of a reservoir to reflect the finite transmission speed of neuron signals. Delayed self-feedback establishes a connection between the current and previous m time steps of the reservoir state and provides an effective means to capture the dynamic characteristics of the system, thereby significantly improving memory performance. In addition, the concept of local echo state property (ESP) is introduced to relax the conventional ESP condition, and theoretical analysis is conducted on guiding the selection of feedback delay and gain to ensure the local ESP. The judicious selection of feedback gain and delay in self-ESN improves prediction accuracy and overcomes the challenges associated with obtaining optimal parameters of the reservoir in conventional ESN models. Numerical experiments are conducted to assess the long-term prediction capabilities of the self-ESN across various scenarios, including a 4D hyperchaotic system, a hyperchaotic network, and an infinite-dimensional delayed chaotic system. The experiments involve reconstructing bifurcation diagrams, predicting the chaotic synchronization, examining spatiotemporal evolution patterns, and uncovering the hidden attractors. The results underscore the capability of the proposed self-ESN as a strategy for long-term prediction and analysis of the complex systems.

Underground Diagnosis Based on GPR and Learning in the Model Space

Ground Penetrating Radar (GPR) has been widely used in pipeline detection and underground diagnosis. In practical applications, the characteristics of the GPR data of the detected area and the likely underground anomalous structures could be rarely acknowledged before fully analyzing the obtained GPR data, causing challenges to identify the underground structures or anomalies automatically. In this article, a GPR B-scan image diagnosis method based on learning in the model space is proposed. The idea of learning in the model space is to use models fitted on parts of data as more stable and parsimonious representations of the data. For the GPR image, 2-Direction Echo State Network (2D-ESN) is proposed to fit the image segments through the next item prediction. By building the connections between the points on the image in both the horizontal and vertical directions, the 2D-ESN regards the GPR image segment as a whole and could effectively capture the dynamic characteristics of the GPR image. And then, semi-supervised and supervised learning methods could be further implemented on the 2D-ESN models for underground diagnosis. Experiments on real-world datasets are conducted, and the results demonstrate the effectiveness of the proposed model.

Evolving Dual-Threshold Bienenstock-Cooper-Munro Learning Rules in Echo State Networks

The strengthening and the weakening of synaptic strength in existing Bienenstock-Cooper-Munro (BCM) learning rule are determined by a long-term potentiation (LTP) sliding modification threshold and the afferent synaptic activities. However, synaptic long-term depression (LTD) even affects low-active synapses during the induction of synaptic plasticity, which may lead to information loss. Biological experiments have found another LTD threshold that can induce either potentiation or depression or no change, even at the activated synapses. In addition, existing BCM learning rules can only select a set of fixed rule parameters, which is biologically implausible and practically inflexible to learn the structural information of input signals. In this article, an evolved dual-threshold BCM learning rule is proposed to regulate the reservoir internal connection weights of the echo-state-network (ESN), which can contribute to alleviating information loss and enhancing learning performance by introducing different optimal LTD thresholds for different postsynaptic neurons. Our experimental results show that the evolved dual-threshold BCM learning rule can result in the synergistic learning of different plasticity rules, effectively improving the learning performance of an ESN in comparison with existing neural plasticity learning rules and some state-of-the-art ESN variants on three widely used benchmark tasks and the prediction of an esterification process.

Fuzzy Cognitive Map-Driven Comprehensive Time-Series Classification

This article presents a comprehensive approach for time-series classification. The proposed model employs a fuzzy cognitive map (FCM) as a classification engine. Preprocessed input data feed the employed FCM. Map responses, after a postprocessing procedure, are used in the calculation of the final classification decision. The time-series data are staged using the moving-window technique to capture the time flow in the training procedure. We use a backward error propagation algorithm to compute the required model hyperparameters. Four model hyperparameters require tuning. Two are crucial for the model construction: 1) FCM size (number of concepts) and 2) window size (for the moving-window technique). Other two are important for training the model: 1) the number of epochs and 2) the learning rate (for training). Two distinguishing aspects of the proposed model are worth noting: 1) the separation of the classification engine from pre- and post-processing and 2) the time flow capture for data from concept space. The proposed classifier joins the key advantage of the FCM model, which is the interpretability of the model, with the superior classification performance attributed to the specially designed pre- and postprocessing stages. This article presents the experiments performed, demonstrating that the proposed model performs well against a wide range of state-of-the-art time-series classification algorithms.

Computational Modeling of Structural Synaptic Plasticity in Echo State Networks

Most existing studies on computational modeling of neural plasticity have focused on synaptic plasticity. However, regulation of the internal weights in the reservoir based on synaptic plasticity often results in unstable learning dynamics. In this article, a structural synaptic plasticity learning rule is proposed to train the weights and add or remove neurons within the reservoir, which is shown to be able to alleviate the instability of the synaptic plasticity, and to contribute to increase the memory capacity of the network as well. Our experimental results also reveal that a few stronger connections may last for a longer period of time in a constantly changing network structure, and are relatively resistant to decay or disruptions in the learning process. These results are consistent with the evidence observed in biological systems. Finally, we show that an echo state network (ESN) using the proposed structural plasticity rule outperforms an ESN using synaptic plasticity and three state-of-the-art ESNs on four benchmark tasks.

An adaptive particle swarm optimization-based hybrid long short-term memory model for stock price time series forecasting

In this paper, we presented a long short-term memory (LSTM) network and adaptive particle swarm optimization (PSO)-based hybrid deep learning model for forecasting the stock price of three major stock indices such as Sensex, S&P 500, and Nifty 50 for short term and long term. Although the LSTM can handle uncertain, sequential, and nonlinear data, the biggest challenge in it is optimizing its weights and bias. The back-propagation through time algorithm has a drawback to overfit the data and being stuck in local minima. Thus, we proposed PSO-based hybrid deep learning model for evolving the initial weights of LSTM and fully connected layer (FCL). Furthermore, we introduced an adaptive approach for improving the inertia coefficient of PSO using the velocity of particles. The proposed method is an aggregation of adaptive PSO and Adam optimizer for training the LSTM. The adaptive PSO attempts to evolve the initial weights in different layers of the LSTM network and FCL. This research also compares the forecasting efficacy of the proposed method to the genetic algorithm (GA)-based hybrid LSTM model, the Elman neural network (ENN), and standard LSTM. Experimental findings demonstrate that the suggested model is successful in achieving the optimum initial weights and bias of the LSTM and FC layers, as well as superior forecasting accuracy.

Difference-Guided Representation Learning Network for Multivariate Time-Series Classification

Multivariate time series (MTSs) are widely found in many important application fields, for example, medicine, multimedia, manufacturing, action recognition, and speech recognition. The accurate classification of MTS has become an important research topic. Traditional MTS classification methods do not explicitly model the temporal difference information of time series, which is, in fact, important and reflects the dynamic evolution information. In this article, the difference-guided representation learning network (DGRL-Net) is proposed to guide the representation learning of time series by dynamic evolution information. The DGRL-Net consists of a difference-guided layer and a multiscale convolutional layer. First, in the difference-guided layer, we propose a difference gating LSTM to model the time dependency and dynamic evolution of the time series to obtain feature representations of both raw and difference series. Then, these two representations are used as two input channels of the multiscale convolutional layer to extract multiscale information. Extensive experiments demonstrate that the proposed model outperforms state-of-the-art methods on 18 MTS benchmark datasets and achieves competitive results on two skeleton-based action recognition datasets. Furthermore, the ablation study and visualized analysis are designed to verify the effectiveness of the proposed model.

Self-Supervised Time Series Clustering With Model-Based Dynamics

Time series clustering is usually an essential unsupervised task in cases when category information is not available and has a wide range of applications. However, existing time series clustering methods usually either ignore temporal dynamics of time series or isolate the feature extraction from clustering tasks without considering the interaction between them. In this article, a time series clustering framework named self-supervised time series clustering network (STCN) is proposed to optimize the feature extraction and clustering simultaneously. In the feature extraction module, a recurrent neural network (RNN) conducts a one-step time series prediction that acts as the reconstruction of the input data, capturing the temporal dynamics and maintaining the local structures of the time series. The parameters of the output layer of the RNN are regarded as model-based dynamic features and then fed into a self-supervised clustering module to obtain the predicted labels. To bridge the gap between these two modules, we employ spectral analysis to constrain the similar features to have the same pseudoclass labels and align the predicted labels with pseudolabels as well. STCN is trained by iteratively updating the model parameters and the pseudoclass labels. Experiments conducted on extensive time series data sets show that STCN has state-of-the-art performance, and the visualization analysis also demonstrates the effectiveness of the proposed model.

Reservoir Computing Approaches for Representation and Classification of Multivariate Time Series

Classification of multivariate time series (MTS) has been tackled with a large variety of methodologies and applied to a wide range of scenarios. Reservoir computing (RC) provides efficient tools to generate a vectorial, fixed-size representation of the MTS that can be further processed by standard classifiers. Despite their unrivaled training speed, MTS classifiers based on a standard RC architecture fail to achieve the same accuracy of fully trainable neural networks. In this article, we introduce the reservoir model space, an unsupervised approach based on RC to learn vectorial representations of MTS. Each MTS is encoded within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics. Compared with other RC methods, our model space yields better representations and attains comparable computational performance due to an intermediate dimensionality reduction procedure. As a second contribution, we propose a modular RC framework for MTS classification, with an associated open-source Python library. The framework provides different modules to seamlessly implement advanced RC architectures. The architectures are compared with other MTS classifiers, including deep learning models and time series kernels. Results obtained on the benchmark and real-world MTS data sets show that RC classifiers are dramatically faster and, when implemented using our proposed representation, also achieve superior classification accuracy.

A Knee-Guided Evolutionary Algorithm for Compressing Deep Neural Networks

Deep neural networks (DNNs) have been regarded as fundamental tools for many disciplines. Meanwhile, they are known for their large-scale parameters, high redundancy in weights, and extensive computing resource consumptions, which pose a tremendous challenge to the deployment in real-time applications or on resource-constrained devices. To cope with this issue, compressing DNNs for accelerating its inference has drawn extensive interest recently. The basic idea is to prune parameters with little performance degradation. However, the overparameterized nature and the conflict between parameters reduction and performance maintenance make it prohibitive to manually search the pruning parameter space. In this paper, we formally establish filter pruning as a multiobjective optimization problem, and propose a knee-guided evolutionary algorithm (KGEA) that can automatically search for the solution with quality tradeoff between the scale of parameters and performance, in which both conflicting objectives can be optimized simultaneously. In particular, by incorporating a minimum Manhattan distance approach, the search effort in the proposed KGEA is explicitly guided toward the knee area, which greatly facilitates the manual search for a good tradeoff solution. Moreover, the parameter importance is directly estimated on the criterion of performance loss, which can robustly identify the redundancy. In addition to the knee solution, a performance-improved model can also be found in a fine-tuning-free fashion. The experiments on compressing fully convolutional LeNet and VGG-19 networks validate the superiority of the proposed algorithm over the state-of-the-art competing methods.

A pareto ensemble based spectral clustering framework

Similarity matrix has a significant effect on the performance of the spectral clustering, and how to determine the neighborhood in the similarity matrix effectively is one of its main difficulties. In this paper, a “divide and conquer” strategy is proposed to model the similarity matrix construction task by adopting Multiobjective evolutionary algorithm (MOEA). The whole procedure is divided into two phases, phase I aims to determine the nonzero entries of the similarity matrix, and Phase II aims to determine the value of the nonzero entries of the similarity matrix. In phase I, the main contribution is that we model the task as a biobjective dynamic optimization problem, which optimizes the diversity and the similarity at the same time. It makes each individual determine one nonzero entry for each sample, and the encoding length decreases to O(N) in contrast with the non-ensemble multiobjective spectral clustering. In addition, a specific initialization operator and diversity preservation strategy are proposed during this phase. In phase II, three ensemble strategies are designed to determine the value of the nonzero value of the similarity matrix. Furthermore, this Pareto ensemble framework is extended to semi-supervised clustering by transforming the semi-supervised information to constraints. In contrast with the previous multiobjective evolutionary-based spectral clustering algorithms, the proposed Pareto ensemble-based framework makes a balance between time cost and the clustering accuracy, which is demonstrated in the experiments section.