Optimization in Quaternion Dynamic Systems: Gradient, Hessian, and Learning Algorithms

Time series compression using quaternion valued neural networks and quaternion backpropagation

We propose a novel quaternionic time series compression methodology where we divide a long time series into segments of data, extract the min, max, mean and standard deviation of these chunks as representative features and encapsulate them in a quaternion, yielding a quaternion valued time series. This time series is processed using quaternion valued neural network layers, where we aim to preserve the relation between these features through the usage of the Hamilton product. To train this quaternion neural network, we derive quaternion backpropagation employing the GHR calculus, which is required for a valid product and chain rule in quaternion space. Furthermore, we investigate the connection between the derived update rules and automatic differentiation. We apply our proposed compression method on the Tennessee Eastman Dataset, where we perform fault classification using the compressed data in two settings: a fully supervised one and in a semi supervised, contrastive learning setting. Both times, we were able to outperform real valued counterparts as well as two baseline models: one with the uncompressed time series as the input and the other with a regular downsampling using the mean. Further, we could improve the classification benchmark set by SimCLR-TS from 81.43% to 83.90%.

A Dynamic Gain Fixed-Time Robust ZNN Model for Time-Variant Equality Constrained Quaternion Least Squares Problem With Applications to Multiagent Systems

A dynamic gain fixed-time (FXT) robust zeroing neural network (DFTRZNN) model is proposed to effectively solve time-variant equality constrained quaternion least squares problem (TV-EQLS). The proposed approach surmounts the shortcomings of conventional numerical algorithms which fail to address time-variant problems. The DFTRZNN model is constructed with a novel dynamic gain parameter and a novel activation function (NAF), which differs from previous zeroing neural network (ZNN) models. Moreover, the comprehensive theoretical derivation of the FXT stability and robustness of the DFTRZNN model is presented in detail. Simulation results further confirm the availability and superiority of the DFTRZNN model for solving TV-EQLS. Finally, the consensus protocols of multiagent systems are presented by utilizing the design scheme of the DFTRZNN model, which further demonstrates its practical application value.

Overcoming Long-Term Catastrophic Forgetting Through Adversarial Neural Pruning and Synaptic Consolidation

Enabling a neural network to sequentially learn multiple tasks is of great significance for expanding the applicability of neural networks in real-world applications. However, artificial neural networks face the well-known problem of catastrophic forgetting. What is worse, the degradation of previously learned skills becomes more severe as the task sequence increases, known as the long-term catastrophic forgetting. It is due to two facts: first, as the model learns more tasks, the intersection of the low-error parameter subspace satisfying for these tasks becomes smaller or even does not exist; second, when the model learns a new task, the cumulative error keeps increasing as the model tries to protect the parameter configuration of previous tasks from interference. Inspired by the memory consolidation mechanism in mammalian brains with synaptic plasticity, we propose a confrontation mechanism in which Adversarial Neural Pruning and synaptic Consolidation (ANPyC) is used to overcome the long-term catastrophic forgetting issue. The neural pruning acts as long-term depression to prune task-irrelevant parameters, while the novel synaptic consolidation acts as long-term potentiation to strengthen task-relevant parameters. During the training, this confrontation achieves a balance in that only crucial parameters remain, and non-significant parameters are freed to learn subsequent tasks. ANPyC avoids forgetting important information and makes the model efficient to learn a large number of tasks. Specifically, the neural pruning iteratively relaxes the current task's parameter conditions to expand the common parameter subspace of the task; the synaptic consolidation strategy, which consists of a structure-aware parameter-importance measurement and an element-wise parameter updating strategy, decreases the cumulative error when learning new tasks. Our approach encourages the synapse to be sparse and polarized, which enables long-term learning and memory. ANPyC exhibits effectiveness and generalization on both image classification and generation tasks with multiple layer perceptron, convolutional neural networks, and generative adversarial networks, and variational autoencoder. The full source code is available at https://github.com/GeoX-Lab/ANPyC.

Feedforward–Feedback Controller Based on a Trained Quaternion Neural Network Using a Generalised HR Calculus with Application to Trajectory Control of a Three-Link Robot Manipulator

This study derives a learning algorithm for a quaternion neural network using the steepest descent method extended to quaternion numbers. This applies the generalised Hamiltonian–Real calculus to obtain derivatives of a real–valued cost function concerning quaternion variables and designs a feedback–feedforward controller as a control system application using such a network. The quaternion neural network is trained in real-time by introducing a feedback error learning framework to the controller. Thus, the quaternion neural network-based controller functions as an adaptive-type controller. The designed controller is applied to the control problem of a three-link robot manipulator, with the control task of making the robot manipulator’s end effector follow a desired trajectory in the Cartesian space. Computational experiments are conducted to investigate the learning capability and the characteristics of the quaternion neural network used in the controller. The experimental results confirm the feasibility of using the derived learning algorithm based on the generalised Hamiltonian–Real calculus to train the quaternion neural network and the availability of such a network for a control systems application.

Penalty Method for Constrained Distributed Quaternion-Variable Optimization

This article studies the constrained optimization problems in the quaternion regime via a distributed fashion. We begin with presenting some differences for the generalized gradient between the real and quaternion domains. Then, an algorithm for the considered optimization problem is given, by which the desired optimization problem is transformed into an unconstrained setup. Using the tools from the Lyapunov-based technique and nonsmooth analysis, the convergence property associated with the devised algorithm is further guaranteed. In addition, the designed algorithm has the potential for solving distributed neurodynamic optimization problems as a recurrent neural network. Finally, a numerical example involving machine learning is given to illustrate the efficiency of the obtained results.

Innovative Head-Mounted System Based on Inertial Sensors and Magnetometer for Detecting Falling Movements

This work presents a fall detection system that is worn on the head, where the acceleration and posture are stable such that everyday movement can be identified without disturbing the wearer. Falling movements are recognized by comparing the acceleration and orientation of a wearer's head using prespecified thresholds. The proposed system consists of a triaxial accelerometer, gyroscope, and magnetometer; as such, a Madgwick's filter is adopted to improve the accuracy of the estimation of orientation. Moreover, with its integrated Wi-Fi module, the proposed system can notify an emergency contact in a timely manner to provide help for the falling person. Based on experimental results concerning falling movements and activities of daily living, the proposed system achieved a sensitivity of 96.67% in fall detection, with a specificity of 98.27%, and, therefore, is suitable for detecting falling movements in daily life.

Hessian with Mini-Batches for Electrical Demand Prediction

The steepest descent method is frequently used for neural network tuning. Mini-batches are commonly used to get better tuning of the steepest descent in the neural network. Nevertheless, steepest descent with mini-batches could be delayed in reaching a minimum. The Hessian could be quicker than the steepest descent in reaching a minimum, and it is easier to achieve this goal by using the Hessian with mini-batches. In this article, the Hessian is combined with mini-batches for neural network tuning. The discussed algorithm is applied for electrical demand prediction.

Constrained Quaternion-Variable Convex Optimization: A Quaternion-Valued Recurrent Neural Network Approach

This paper proposes a quaternion-valued one-layer recurrent neural network approach to resolve constrained convex function optimization problems with quaternion variables. Leveraging the novel generalized Hamilton-real (GHR) calculus, the quaternion gradient-based optimization techniques are proposed to derive the optimization algorithms in the quaternion field directly rather than the methods of decomposing the optimization problems into the complex domain or the real domain. Via chain rules and Lyapunov theorem, the rigorous analysis shows that the deliberately designed quaternion-valued one-layer recurrent neural network stabilizes the system dynamics while the states reach the feasible region in finite time and converges to the optimal solution of the considered constrained convex optimization problems finally. Numerical simulations verify the theoretical results.

Low-Density EEG for Neural Activity Reconstruction Using Multivariate Empirical Mode Decomposition

Several approaches can be used to estimate neural activity. The main differences between them concern the a priori information used and its sensitivity to high noise levels. Empirical mode decomposition (EMD) has been recently applied to electroencephalography EEG-based neural activity reconstruction to provide a priori time-frequency information to improve the estimation of neural activity. EMD has the specific ability to identify independent oscillatory modes in non-stationary signals with multiple oscillatory components. However, attempts to use EMD in EEG analysis have not yet provided optimal reconstructions, due to the intrinsic mode-mixing problem of EMD. Several studies have used single-channel analysis, whereas others have used multiple-channel analysis for other applications. Here, we present the results of multiple-channel analysis using multivariate empirical mode decomposition (MEMD) to reduce the mode-mixing problem and provide useful a priori time-frequency information for the reconstruction of neuronal activity using several low-density EEG electrode montages. The methods were evaluated using real and synthetic EEG data, in which the reconstructions were performed using the multiple sparse priors (MSP) algorithm with EEG electrode montages of 32, 16, and 8 electrodes. The quality of the source reconstruction was assessed using the Wasserstein metric. A comparison of the solutions without pre-processing and those after applying MEMD showed the source reconstructions to be improved using MEMD as a priori information for the low-density montages of 8 and 16 electrodes. The mean source reconstruction error on a real EEG dataset was reduced by 59.42 and 66.04% for the 8 and 16 electrode montages, respectively, and that on a simulated EEG with three active sources, by 87.31 and 31.45% for the same electrode montages.

Localization of Active Brain Sources From EEG Signals Using Empirical Mode Decomposition: A Comparative Study

The localization of active brain sources from Electroencephalogram (EEG) is a useful method in clinical applications, such as the study of localized epilepsy, evoked-related-potentials, and attention deficit/hyperactivity disorder. The distributed-source model is a common method to estimate neural activity in the brain. The location and amplitude of each active source are estimated by solving the inverse problem by regularization or using Bayesian methods with spatio-temporal constraints. Frequency and spatio-temporal constraints improve the quality of the reconstructed neural activity. However, separation into frequency bands is beneficial when the relevant information is in specific sub-bands. We improved frequency-band identification and preserved good temporal resolution using EEG pre-processing techniques with good frequency band separation and temporal resolution properties. The identified frequency bands were included as constraints in the solution of the inverse problem by decomposing the EEG signals into frequency bands through various methods that offer good frequency and temporal resolution, such as empirical mode decomposition (EMD) and wavelet transform (WT). We present a comparative analysis of the accuracy of brain-source reconstruction using these techniques. The accuracy of the spatial reconstruction was assessed using the Wasserstein metric for real and simulated signals. We approached the mode-mixing problem, inherent to EMD, by exploring three variants of EMD: masking EMD, Ensemble-EMD (EEMD), and multivariate EMD (MEMD). The results of the spatio-temporal brain source reconstruction using these techniques show that masking EMD and MEMD can largely mitigate the mode-mixing problem and achieve a good spatio-temporal reconstruction of the active sources. Masking EMD and EEMD achieved better reconstruction than standard EMD, Multiple Sparse Priors, or wavelet packet decomposition when EMD was used as a pre-processing tool for the spatial reconstruction (averaged over time) of the brain sources. The spatial resolution obtained using all three EMD variants was substantially better than the use of EMD alone, as the mode-mixing problem was mitigated, particularly with masking EMD and EEMD. These findings encourage further exploration into the use of EMD-based pre-processing, the mode-mixing problem, and its impact on the accuracy of brain source activity reconstruction.

Sequential Outlier Criterion for Sparsification of Online Adaptive Filtering

In this paper, we deal with the learning problem when using an adaptive filtering method. For the learning system in filtering, the knowledge is obtained and updated based on the newly acquired information that is extracted and learned from the sequential samples over time. Effective measurement on the informativeness of a sample and reasonable subsequent treatment on the sample will improve the learning performance. This paper proposes a sequential outlier criterion for sparsification of online adaptive filtering. The method is proposed to achieve effective informativeness measurement of online filtering to obtain a more accurate and more compact network in the learning process. In the proposed method, the measurement on the samples' informativeness is established based on the historical sequentially adjacent samples, and then the informative-measured samples are treated individually by the learning system based on whether the sample is informative, redundant, or abnormal. With our method, a more sensible learning process can be achieved with valid knowledge extracted, and the optimal network in the learning system can be obtained. Simulations based on static function estimation, Mackey-Glass time series prediction, and Lorenz chaotic time series prediction demonstrate that the proposed method can provide more effective classification on samples and more accurate networks in online adaptive filtering.

On the Dynamics of Hopfield Neural Networks on Unit Quaternions

In this paper, we first address the dynamics of the elegant multivalued quaternionic Hopfield neural network (MV-QHNN) proposed by Minemoto et al. Contrary to what was expected, we show that the MV-QHNN, as well as one of its variation, does not always come to rest at an equilibrium state under the usual conditions. In fact, we provide simple examples in which the network yields a periodic sequence of quaternionic state vectors. Afterward, we turn our attention to the continuous-valued quaternionic Hopfield neural network (CV-QHNN), which can be derived from the MV-QHNN by means of a limit process. The CV-QHNN can be implemented more easily than the MV-QHNN model. Furthermore, the asynchronous CV-QHNN always settles down into an equilibrium state under the usual conditions. Theoretical issues are all illustrated by examples in this paper.

Quaternion Collaborative and Sparse Representation With Application to Color Face Recognition

Collaborative representation-based classification (CRC) and sparse RC (SRC) have recently achieved great success in face recognition (FR). Previous CRC and SRC are originally designed in the real setting for grayscale image-based FR. They separately represent the color channels of a query color image and ignore the structural correlation information among the color channels. To remedy this limitation, in this paper, we propose two novel RC methods for color FR, namely, quaternion CRC (QCRC) and quaternion SRC (QSRC) using quaternion ℓ₁ minimization. By modeling each color image as a quaternionic signal, they naturally preserve the color structures of both query and gallery color images while uniformly coding the query channel images in a holistic manner. Despite the empirical success of CRC and SRC on FR, a few theoretical results are developed to guarantee their effectiveness. Another purpose of this paper is to establish the theoretical guarantee for QCRC and QSRC under mild conditions. Comparisons with competing methods on benchmark real-world databases consistently show the superiority of the proposed methods for both color FR and reconstruction.

Enabling quaternion derivatives: the generalized HR calculus

Quaternion derivatives exist only for a very restricted class of analytic (regular) functions; however, in many applications, functions of interest are real-valued and hence not analytic, a typical case being the standard real mean square error objective function. The recent HR calculus is a step forward and provides a way to calculate derivatives and gradients of both analytic and non-analytic functions of quaternion variables; however, the HR calculus can become cumbersome in complex optimization problems due to the lack of rigorous product and chain rules, a consequence of the non-commutativity of quaternion algebra. To address this issue, we introduce the generalized HR (GHR) derivatives which employ quaternion rotations in a general orthogonal system and provide the left- and right-hand versions of the quaternion derivative of general functions. The GHR calculus also solves the long-standing problems of product and chain rules, mean-value theorem and Taylor's theorem in the quaternion field. At the core of the proposed GHR calculus is quaternion rotation, which makes it possible to extend the principle to other functional calculi in non-commutative settings. Examples in statistical learning theory and adaptive signal processing support the analysis.