Feature hypergraph representation learning on spatial-temporal correlations for EEG emotion recognition

An EEG-based emotion recognition method by fusing multi-frequency-spatial features under multi-frequency bands

BACKGROUND

Recognition of emotion changes is of great significance to a person's physical and mental health. At present, EEG-based emotion recognition methods are mainly focused on time or frequency domains, but rarely on spatial information. Therefore, the goal of this study is to improve the performance of emotion recognition by integrating frequency and spatial domain information under multi-frequency bands.

NEW METHODS

Firstly, EEG signals of four frequency bands are extracted, and then three frequency-spatial features of differential entropy (DE) symmetric difference (SD) and symmetric quotient (SQ) are separately calculated. Secondly, according to the distribution of EEG electrodes, a series of brain maps are constructed by three frequency-spatial features for each frequency band. Thirdly, a Multi-Parallel-Input Convolutional Neural Network (MPICNN) uses the constructed brain maps to train and obtain the emotion recognition model. Finally, the subject-dependent experiments are conducted on DEAP and SEED-IV datasets.

RESULTS

The experimental results of DEAP dataset show that the average accuracy of four-class emotion recognition, namely, high-valence high-arousal, high-valence low-arousal, low-valence high-arousal and low-valence low-arousal, reaches 98.71 %. The results of SEED-IV dataset show the average accuracy of four-class emotion recognition, namely, happy, sad, neutral and fear reaches 92.55 %.

COMPARISON WITH EXISTING METHODS

This method has a best classification performance compared with the state-of-the-art methods on both four-class emotion recognition datasets.

CONCLUSIONS

This EEG-based emotion recognition method fused multi-frequency-spatial features under multi-frequency bands, and effectively improved the recognition performance compared with the existing methods.

A Novel and Powerful Dual-Stream Multi-Level Graph Convolution Network for Emotion Recognition

Emotion recognition enables machines to more acutely perceive and understand users' emotional states, thereby offering more personalized and natural interactive experiences. Given the regularity of the responses of brain activity to human cognitive processes, we propose a powerful and novel dual-stream multi-level graph convolution network (DMGCN) with the ability to capture the hierarchies of connectivity between cerebral cortex neurons and improve computational efficiency. This consists of a hierarchical dynamic geometric interaction neural network (HDGIL) and multi-level feature fusion classifier (M2FC). First, the HDGIL diversifies representations by learning emotion-related representations in multi-level graphs. Subsequently, M2FC integrates advantages from methods for early and late feature fusion and enables the addition of more details to final representations from EEG samples. We conducted extensive experiments to validate the superiority of our model over numerous state-of-the-art (SOTA) baselines in terms of classification accuracy, the efficiency of graph embedding and information propagation, achieving accuracies of 98.73%, 95.97%, 72.74% and 94.89% for our model as well as increases of up to 0.59%, 0.32%, 2.24% and 3.17% over baselines on the DEAP-Arousal, DEAP-Valence, DEAP and SEED datasets, respectively. Additionally, these experiments demonstrated the effectiveness of each module for emotion recognition tasks.

Multi-branch fusion graph neural network based on multi-head attention for childhood seizure detection

The most common manifestation of neurological disorders in children is the occurrence of epileptic seizures. In this study, we propose a multi-branch graph convolutional network (MGCNA) framework with a multi-head attention mechanism for detecting seizures in children. The MGCNA framework extracts effective and reliable features from high-dimensional data, particularly by exploring the relationships between EEG features and electrodes and considering the spatial and temporal dependencies in epileptic brains. This method incorporates three graph learning approaches to systematically assess the connectivity and synchronization of multi-channel EEG signals. The multi-branch graph convolutional network is employed to dynamically learn temporal correlations and spatial topological structures. Utilizing the multi-head attention mechanism to process multi-branch graph features further enhances the capability to handle local features. Experimental results demonstrate that the MGCNA exhibits superior performance on patient-specific and patient-independent experiments. Our end-to-end model for automatic detection of epileptic seizures could be employed to assist in clinical decision-making.

PSPN: Pseudo-Siamese Pyramid Network for multimodal emotion analysis

Emotion recognition plays an important role in human life and healthcare. The EEG has been extensively researched as an objective indicator of intense emotions. However, current existing methods lack sufficient analysis of shallow and deep EEG features. In addition, human emotions are complex and variable, making it difficult to comprehensively represent emotions using a single-modal signal. As a signal associated with gaze tracking and eye movement detection, Eye-related signals provide various forms of supplementary information for multimodal emotion analysis. Therefore, we propose a Pseudo-Siamese Pyramid Network (PSPN) for multimodal emotion analysis. The PSPN model employs a Depthwise Separable Convolutional Pyramid (DSCP) to extract and integrate intrinsic emotional features at various levels and scales from EEG signals. Simultaneously, we utilize a fully connected subnetwork to extract the external emotional features from eye-related signals. Finally, we introduce a Pseudo-Siamese network that integrates a flexible cross-modal dual-branch subnetwork to collaboratively utilize EEG emotional features and eye-related behavioral features, achieving consistency and complementarity in multimodal emotion recognition. For evaluation, we conducted experiments on the DEAP and SEED-IV public datasets. The experimental results demonstrate that multimodal fusion significantly improves the accuracy of emotion recognition compared to single-modal approaches. Our PSPN model achieved the best accuracy of 96.02% and 96.45% on the valence and arousal dimensions of the DEAP dataset, and 77.81% on the SEED-IV dataset, respectively. Our code link is: https://github.com/Yinyanyan003/PSPN.git.

S3T-Net: A novel electroencephalogram signals-oriented emotion recognition model

In this paper, a novel skipping spatial-spectral-temporal network (S3T-Net) is developed to handle intra-individual differences in electroencephalogram (EEG) signals for accurate, robust, and generalized emotion recognition. In particular, aiming at the 4D features extracted from the raw EEG signals, a multi-branch architecture is proposed to learn spatial-spectral cross-domain representations, which benefits enhancing the model generalization ability. Time dependency among different spatial-spectral features is further captured via a bi-directional long-short term memory module, which employs an attention mechanism to integrate context information. Moreover, a skip-change unit is designed to add another auxiliary pathway for updating model parameters, which alleviates the vanishing gradient problem in complex spatial-temporal network. Evaluation results show that the proposed S3T-Net outperforms other advanced models in terms of the emotion recognition accuracy, which yields an performance improvement of 0.23% , 0.13%, and 0.43% as compared to the sub-optimal model in three test scenes, respectively. In addition, the effectiveness and superiority of the key components of S3T-Net are demonstrated from various experiments. As a reliable and competent emotion recognition model, the proposed S3T-Net contributes to the development of intelligent sentiment analysis in human-computer interaction (HCI) realm.

Unsupervised heterogeneous domain adaptation for EEG classification

Objective.Domain adaptation has been recognized as a potent solution to the challenge of limited training data for electroencephalography (EEG) classification tasks. Existing studies primarily focus on homogeneous environments, however, the heterogeneous properties of EEG data arising from device diversity cannot be overlooked. This motivates the development of heterogeneous domain adaptation methods that can fully exploit the knowledge from an auxiliary heterogeneous domain for EEG classification.Approach.In this article, we propose a novel model named informative representation fusion (IRF) to tackle the problem of unsupervised heterogeneous domain adaptation in the context of EEG data. In IRF, we consider different perspectives of data, i.e. independent identically distributed (iid) and non-iid, to learn different representations. Specifically, from the non-iid perspective, IRF models high-order correlations among data by hypergraphs and develops hypergraph encoders to obtain data representations of each domain. From the non-iid perspective, by applying multi-layer perceptron networks to the source and target domain data, we achieve another type of representation for both domains. Subsequently, an attention mechanism is used to fuse these two types of representations to yield informative features. To learn transferable representations, the maximum mean discrepancy is utilized to align the distributions of the source and target domains based on the fused features.Main results.Experimental results on several real-world datasets demonstrate the effectiveness of the proposed model.Significance.This article handles an EEG classification situation where the source and target EEG data lie in different spaces, and what's more, under an unsupervised learning setting. This situation is practical in the real world but barely studied in the literature. The proposed model achieves high classification accuracy, and this study is important for the commercial applications of EEG-based BCIs.

Semi-supervised multi-source transfer learning for cross-subject EEG motor imagery classification

Electroencephalogram (EEG) motor imagery (MI) classification refers to the use of EEG signals to identify and classify subjects' motor imagery activities; this task has received increasing attention with the development of brain-computer interfaces (BCIs). However, the collection of EEG data is usually time-consuming and labor-intensive, which makes it difficult to obtain sufficient labeled data from the new subject to train a new model. Moreover, the EEG signals of different individuals exhibit significant differences, leading to a significant drop in the performance of a model trained on the existing subjects when directly classifying EEG signals acquired from new subjects. Therefore, it is crucial to make full use of the EEG data of the existing subjects and the unlabeled EEG data of the new target subject to improve the MI classification performance achieved for the target subject. This research study proposes a semi-supervised multi-source transfer (SSMT) learning model to address the above problems; the model learns informative and domain-invariant representations to address cross-subject MI-EEG classification tasks. In particular, a dynamic transferred weighting schema is presented to obtain the final predictions by integrating the weighted features derived from multi-source domains. The average accuracies achieved on two publicly available EEG datasets reach 83.57 % and 85.09 % , respectively, validating the effectiveness of the SSMT process. The SSMT process reveals the importance of informative and domain-invariant representations in MI classification tasks, as they make full use of the domain-invariant information acquired from each subject.

An epilepsy detection method based on multi-dimensional feature extraction and dual-branch hypergraph convolutional network

Epilepsy is a disease caused by abnormal neural discharge, which severely harms the health of patients. Its pathogenesis is complex and variable with various forms of seizures, leading to significant differences in epilepsy manifestations among different patients. The changes of brain network are strongly correlated with related pathologies. Therefore, it is crucial to effectively and deeply explore the intrinsic features of epilepsy signals to reveal the rules of epilepsy occurrence and achieve accurate detection. Existing methods have faced the following issues: 1) single approach for feature extraction, resulting in insufficient classification information due to the lack of rich dimensions in captured features; 2) inability to deeply analyze the essential commonality of epilepsy signal after feature extraction, making the model susceptible to data distribution and noise interference. Thus, we proposed a high-precision and robust model for epileptic seizure detection, which, for the first time, applies hypergraph convolution to the field of epilepsy detection. Through a hypergraph network structure constructed based on relationships between channels in electroencephalogram (EEG) signals, the model explores higher-order characteristics of epilepsy EEG data. Specifically, we use the Conv-LSTM module and Power spectral density (PSD), a two-branch parallel method, to extract channel features from space-time and frequency domains to solve the problem of insufficient feature extraction, and can adequately describe the data structure and distribution from multiple perspectives through double-branch parallel feature extraction. In addition, we construct a hypergraph on the captured features to explore the intrinsic features in the high-dimensional space in an attempt to reveal the essential commonality of epileptic signal feature extraction. Finally, using the ensemble learning concept, we accomplished epilepsy detection on the dual-branch hypergraph convolution. The model underwent leave-one-out cross-validation on the TUH dataset, achieving an average accuracy of 96.9%, F1 score of 97.3%, Pre of 98.2% and Re of 96.7%. In addition, the model was generalized performance tested on CHB-MIT scalp EEG dataset with leave-one-out cross-validation, and the average ACC, F1 score, Pre and Re were 94.4%, 95.1%, 95.8%, and 93.9% respectively. Experimental results indicate that the model outperforms related literature, providing valuable reference for the clinical application of epilepsy detection.