EEGformer: A transformer-based brain activity classification method using EEG signal

Self-Supervised Learning with Adaptive Frequency-Time Attention Transformer for Seizure Prediction and Classification

BACKGROUND

In deep learning-based epilepsy prediction and classification, enhancing the extraction of electroencephalogram (EEG) features is crucial for improving model accuracy. Traditional supervised learning methods rely on large, detailed annotated datasets, limiting the feasibility of large-scale training. Recently, self-supervised learning approaches using masking-and-reconstruction strategies have emerged, reducing dependence on labeled data. However, these methods are vulnerable to inherent noise and signal degradation in EEG data, which diminishes feature extraction robustness and overall model performance.

METHODS

In this study, we proposed a self-supervised learning Transformer network enhanced with Adaptive Frequency-Time Attention (AFTA) for learning robust EEG feature representations from unlabeled data, utilizing a masking-and-reconstruction framework. Specifically, we pretrained the Transformer network using a self-supervised learning approach, and subsequently fine-tuned the pretrained model for downstream tasks like seizure prediction and classification. To mitigate the impact of inherent noise in EEG signals and enhance feature extraction capabilities, we incorporated AFTA into the Transformer architecture. AFTA incorporates an Adaptive Frequency Filtering Module (AFFM) to perform adaptive global and local filtering in the frequency domain. This module was then integrated with temporal attention mechanisms, enhancing the model's self-supervised learning capabilities.

RESULT

Our method achieved exceptional performance in EEG analysis tasks. Our method consistently outperformed state-of-the-art approaches across TUSZ, TUAB, and TUEV datasets, achieving the highest AUROC (0.891), balanced accuracy (0.8002), weighted F1-score (0.8038), and Cohen's kappa (0.6089). These results validate its robustness, generalization, and effectiveness in seizure detection and classification tasks on diverse EEG datasets.

Tackling the possibility of extracting a brain digital fingerprint based on personal hobbies predilection

In an attempt to create a more familiar brain-machine interaction for biometric authentication applications, we investigated the efficiency of using the users' personal hobbies, interests, and memory collections. This approach creates a unique and pleasant experience that can be later utilized within an authentication protocol. This paper presents a new EEG dataset recorded while subjects watch images of popular hobbies, pictures with no point of interest and images with great personal significance. In addition, we propose several applications that can be tackled with our newly collected dataset. Namely, our study showcases 4 types of applications and we obtain state-of-the-art level results for all of them. The tackled tasks are: emotion classification, category classification, authorization process, and person identification. Our experiments show great potential for using EEG response to hobby visualization for people authentication. In our study, we show preliminary results for using predilection for personal hobbies, as measured by EEG, for identifying people. Also, we propose a novel authorization process paradigm using electroencephalograms. Code and dataset are available here.

REI-Net: A Reference Electrode Standardization Interpolation Technique Based 3D CNN for Motor Imagery Classification

High-quality scalp EEG datasets are extremely valuable for motor imagery (MI) analysis. However, due to electrode size and montage, different datasets inevitably experience channel information loss, posing a significant challenge for MI decoding. A 2D representation that focuses on the time domain may loss the spatial information in EEG. In contrast, a 3D representation based on topography may suffer from channel loss and introduce noise through different padding methods. In this paper, we propose a framework called Reference Electrode Standardization Interpolation Network (REI-Net). Through an interpolation of 3D representation, REI-Net retains the temporal information in 2D scalp EEG while improving the spatial resolution within a certain montage. Additionally, to overcome the data variability caused by individual differences, transfer learning is employed to enhance the decoding robustness. Our approach achieves promising performance on two widely-recognized MI datasets, with an accuracy of 77.99% on BCI-C IV-2a and an accuracy of 63.94% on Kaya2018. The proposed algorithm outperforms the SOTAs leading to more accurate and robust results.

Multimodal depression detection based on an attention graph convolution and transformer

Traditional depression detection methods typically rely on single-modal data, but these approaches are limited by individual differences, noise interference, and emotional fluctuations. To address the low accuracy in single-modal depression detection and the poor fusion of multimodal features from electroencephalogram (EEG) and speech signals, we have proposed a multimodal depression detection model based on EEG and speech signals, named the multi-head attention-GCN_ViT (MHA-GCN_ViT). This approach leverages deep learning techniques, including graph convolutional networks (GCN) and vision transformers (ViT), to effectively extract and fuse the frequency-domain features and spatiotemporal characteristics of EEG signals with the frequency-domain features of speech signals. First, a discrete wavelet transform (DWT) was used to extract wavelet features from 29 channels of EEG signals. These features serve as node attributes for the construction of a feature matrix, calculating the Pearson correlation coefficient between channels, from which an adjacency matrix is constructed to represent the brain network structure. This structure was then fed into a graph convolutional network (GCN) for deep feature learning. A multi-head attention mechanism was introduced to enhance the GCN's capability in representing brain networks. Using a short-time Fourier transform (STFT), we extracted 2D spectral features of EEG signals and mel spectrogram features of speech signals. Both were further processed using a vision transformer (ViT) to obtain deep features. Finally, the multiple features from EEG and speech spectrograms were fused at the decision level for depression classification. A five-fold cross-validation on the MODMA dataset demonstrated the model's accuracy, precision, recall, and F1 score of 89.03%, 90.16%, 89.04%, and 88.83%, respectively, indicating a significant improvement in the performance of multimodal depression detection. Furthermore, MHA-GCN_ViT demonstrated robust performance in depression detection and exhibited broad applicability, with potential for extension to multimodal detection tasks in other psychological and neurological disorders.

Transformers in EEG Analysis: A Review of Architectures and Applications in Motor Imagery, Seizure, and Emotion Classification

Transformers have rapidly influenced research across various domains. With their superior capability to encode long sequences, they have demonstrated exceptional performance, outperforming existing machine learning methods. There has been a rapid increase in the development of transformer-based models for EEG analysis. The high volumes of recently published papers highlight the need for further studies exploring transformer architectures, key components, and models employed particularly in EEG studies. This paper aims to explore four major transformer architectures: Time Series Transformer, Vision Transformer, Graph Attention Transformer, and hybrid models, along with their variants in recent EEG analysis. We categorize transformer-based EEG studies according to the most frequent applications in motor imagery classification, emotion recognition, and seizure detection. This paper also highlights the challenges of applying transformers to EEG datasets and reviews data augmentation and transfer learning as potential solutions explored in recent years. Finally, we provide a summarized comparison of the most recent reported results. We hope this paper serves as a roadmap for researchers interested in employing transformer architectures in EEG analysis.

ID3RSNet: cross-subject driver drowsiness detection from raw single-channel EEG with an interpretable residual shrinkage network

Accurate monitoring of drowsy driving through electroencephalography (EEG) can effectively reduce traffic accidents. Developing a calibration-free drowsiness detection system with single-channel EEG alone is very challenging due to the non-stationarity of EEG signals, the heterogeneity among different individuals, and the relatively parsimonious compared to multi-channel EEG. Although deep learning-based approaches can effectively decode EEG signals, most deep learning models lack interpretability due to their black-box nature. To address these issues, we propose a novel interpretable residual shrinkage network, namely, ID3RSNet, for cross-subject driver drowsiness detection using single-channel EEG signals. First, a base feature extractor is employed to extract the essential features of EEG frequencies; to enhance the discriminative feature learning ability, the residual shrinkage building unit with attention mechanism is adopted to perform adaptive feature recalibration and soft threshold denoising inside the residual network is further applied to achieve automatic feature extraction. In addition, a fully connected layer with weight freezing is utilized to effectively suppress the negative influence of neurons on the model classification. With the global average pooling (GAP) layer incorporated in the residual shrinkage network structure, we introduce an EEG-based Class Activation Map (ECAM) interpretable method to enable visualization analysis of sample-wise learned patterns to effectively explain the model decision. Extensive experimental results demonstrate that the proposed method achieves the superior classification performance and has found neurophysiologically reliable evidence of classification.

Leveraging deep learning for robust EEG analysis in mental health monitoring

Introduction

Mental health monitoring utilizing EEG analysis has garnered notable interest due to the non-invasive characteristics and rich temporal information encoded in EEG signals, which are indicative of cognitive and emotional conditions. Conventional methods for EEG-based mental health evaluation often depend on manually crafted features or basic machine learning approaches, like support vector classifiers or superficial neural networks. Despite the potential of these approaches, they often fall short in capturing the intricate spatiotemporal relationships within EEG data, leading to lower classification accuracy and poor adaptability across various populations and mental health scenarios.

Methods

To overcome these limitations, we introduce the EEG Mind-Transformer, an innovative deep learning architecture composed of a Dynamic Temporal Graph Attention Mechanism (DT-GAM), a Hierarchical Graph Representation and Analysis (HGRA) module, and a Spatial-Temporal Fusion Module (STFM). The DT-GAM is designed to dynamically extract temporal dependencies within EEG data, while the HGRA models the brain's hierarchical structure to capture both localized and global interactions among different brain regions. The STFM synthesizes spatial and temporal elements, generating a comprehensive representation of EEG signals.

Results and discussion

Our empirical results confirm that the EEG Mind-Transformer significantly surpasses conventional approaches, achieving an accuracy of 92.5%, a recall of 91.3%, an F1-score of 90.8%, and an AUC of 94.2% across several datasets. These findings underline the model's robustness and its generalizability to diverse mental health conditions. Moreover, the EEG Mind-Transformer not only pushes the boundaries of state-of-the-art EEG-based mental health monitoring but also offers meaningful insights into the underlying brain functions associated with mental disorders, solidifying its value for both research and clinical settings.

CIT-EmotionNet: convolution interactive transformer network for EEG emotion recognition

Emotion recognition is a significant research problem in affective computing as it has a lot of potential areas of application. One of the approaches in emotion recognition uses electroencephalogram (EEG) signals to identify the emotion of a person. However, effectively using the global and local features of EEG signals to improve the performance of emotion recognition is still a challenge. In this study, we propose a novel Convolution Interactive Transformer Network for EEG Emotion Recognition, known as CIT-EmotionNet, which efficiently integrates the global and local features of EEG signals. We convert the raw EEG signals into spatial-spectral representations, which serve as the inputs into the model. The model integrates convolutional neural network (CNN) and Transformer within a single framework in a parallel manner. We propose a Convolution Interactive Transformer module, which facilitates the interaction and fusion of local and global features extracted by CNN and Transformer respectively, thereby improving the average accuracy of emotion recognition. The proposed CIT-EmotionNet outperforms state-of-the-art methods, achieving an average recognition accuracy of 98.57% and 92.09% on two publicly available datasets, SEED and SEED-IV, respectively.

Attention-based CNN-BiLSTM for sleep state classification of spatiotemporal wide-field calcium imaging data

BACKGROUND

Wide-field calcium imaging (WFCI) with genetically encoded calcium indicators allows for spatiotemporal recordings of neuronal activity in mice. When applied to the study of sleep, WFCI data are manually scored into the sleep states of wakefulness, non-REM (NREM) and REM by use of adjunct EEG and EMG recordings. However, this process is time-consuming, invasive and often suffers from low inter- and intra-rater reliability. Therefore, an automated sleep state classification method that operates on spatiotemporal WFCI data is desired.

NEW METHOD

A hybrid network architecture consisting of a convolutional neural network (CNN) to extract spatial features of image frames and a bidirectional long short-term memory network (BiLSTM) with attention mechanism to identify temporal dependencies among different time points was proposed to classify WFCI data into states of wakefulness, NREM and REM sleep.

RESULTS

Sleep states were classified with an accuracy of 84 % and Cohen's κ of 0.64. Gradient-weighted class activation maps revealed that the frontal region of the cortex carries more importance when classifying WFCI data into NREM sleep while posterior area contributes most to the identification of wakefulness. The attention scores indicated that the proposed network focuses on short- and long-range temporal dependency in a state-specific manner.

COMPARISON WITH EXISTING METHOD

On a held out, repeated 3-hour WFCI recording, the CNN-BiLSTM achieved a κ of 0.67, comparable to a κ of 0.65 corresponding to the human EEG/EMG-based scoring.

CONCLUSIONS

The CNN-BiLSTM effectively classifies sleep states from spatiotemporal WFCI data and will enable broader application of WFCI in sleep research.

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

Understanding emotional states is pivotal for the development of next-generation human-machine interfaces. Human behaviors in social interactions have resulted in psycho-physiological processes influenced by perceptual inputs. Therefore, efforts to comprehend brain functions and human behavior could potentially catalyze the development of AI models with human-like attributes. In this study, we introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, eliciting five distinct emotions: neutral, anger, happiness, sadness, and calmness. Throughout the experiment, each participant contributed 200 interactions, which encompassed both listening and speaking. This resulted in a cumulative total of 8,400 interactions across all participants. We evaluated the baseline performance of emotion recognition for each modality using established deep neural network (DNN) methods. The Emotion in EEG-Audio-Visual (EAV) dataset represents the first public dataset to incorporate three primary modalities for emotion recognition within a conversational context. We anticipate that this dataset will make significant contributions to the modeling of the human emotional process, encompassing both fundamental neuroscience and machine learning viewpoints.

A comprehensive review of deep learning in EEG-based emotion recognition: classifications, trends, and practical implications

Emotion recognition utilizing EEG signals has emerged as a pivotal component of human-computer interaction. In recent years, with the relentless advancement of deep learning techniques, using deep learning for analyzing EEG signals has assumed a prominent role in emotion recognition. Applying deep learning in the context of EEG-based emotion recognition carries profound practical implications. Although many model approaches and some review articles have scrutinized this domain, they have yet to undergo a comprehensive and precise classification and summarization process. The existing classifications are somewhat coarse, with insufficient attention given to the potential applications within this domain. Therefore, this article systematically classifies recent developments in EEG-based emotion recognition, providing researchers with a lucid understanding of this field's various trajectories and methodologies. Additionally, it elucidates why distinct directions necessitate distinct modeling approaches. In conclusion, this article synthesizes and dissects the practical significance of EEG signals in emotion recognition, emphasizing its promising avenues for future application.

U-shaped convolutional transformer GAN with multi-resolution consistency loss for restoring brain functional time-series and dementia diagnosis

Introduction

The blood oxygen level-dependent (BOLD) signal derived from functional neuroimaging is commonly used in brain network analysis and dementia diagnosis. Missing the BOLD signal may lead to bad performance and misinterpretation of findings when analyzing neurological disease. Few studies have focused on the restoration of brain functional time-series data.

Methods

In this paper, a novel U-shaped convolutional transformer GAN (UCT-GAN) model is proposed to restore the missing brain functional time-series data. The proposed model leverages the power of generative adversarial networks (GANs) while incorporating a U-shaped architecture to effectively capture hierarchical features in the restoration process. Besides, the multi-level temporal-correlated attention and the convolutional sampling in the transformer-based generator are devised to capture the global and local temporal features for the missing time series and associate their long-range relationship with the other brain regions. Furthermore, by introducing multi-resolution consistency loss, the proposed model can promote the learning of diverse temporal patterns and maintain consistency across different temporal resolutions, thus effectively restoring complex brain functional dynamics.

Results

We theoretically tested our model on the public Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, and our experiments demonstrate that the proposed model outperforms existing methods in terms of both quantitative metrics and qualitative assessments. The model's ability to preserve the underlying topological structure of the brain functional networks during restoration is a particularly notable achievement.

Conclusion

Overall, the proposed model offers a promising solution for restoring brain functional time-series and contributes to the advancement of neuroscience research by providing enhanced tools for disease analysis and interpretation.

A novel approach for ASD recognition based on graph attention networks

Early detection and diagnosis of Autism Spectrum Disorder (ASD) can significantly improve the quality of life for affected individuals. Identifying ASD based on brain functional connectivity (FC) poses a challenge due to the high heterogeneity of subjects' fMRI data in different sites. Meanwhile, deep learning algorithms show efficacy in ASD identification but lack interpretability. In this paper, a novel approach for ASD recognition is proposed based on graph attention networks. Specifically, we treat the region of interest (ROI) of the subjects as node, conduct wavelet decomposition of the BOLD signal in each ROI, extract wavelet features, and utilize them along with the mean and variance of the BOLD signal as node features, and the optimized FC matrix as the adjacency matrix, respectively. We then employ the self-attention mechanism to capture long-range dependencies among features. To enhance interpretability, the node-selection pooling layers are designed to determine the importance of ROI for prediction. The proposed framework are applied to fMRI data of children (younger than 12 years old) from the Autism Brain Imaging Data Exchange datasets. Promising results demonstrate superior performance compared to recent similar studies. The obtained ROI detection results exhibit high correspondence with previous studies and offer good interpretability.