Linking Attention-Based Multiscale CNN With Dynamical GCN for Driving Fatigue Detection

An adaptive session-incremental broad learning system for continuous motor imagery EEG classification

Motor imagery electroencephalography (MI-EEG) is usually used as a driving signal in neuro-rehabilitation systems, and its feature space varies with the recovery progress. It is required to endow the recognition model with continuous learning and self-updating capability. Broad learning system (BLS) can be remodeled in an efficient incremental learning way. However, its architecture is intractable to change automatically to adapt to new incoming MI-EEG with time-varying and complex temporal-spatial characteristics. In this paper, an adaptive session-incremental BLS (ASiBLS) is proposed based on mutual information theory and BLS. For the initial session data, a compact temporal-spatial feature extractor (CTS) is designed to acquire the temporal-spatial features, which are input to a baseline BLS (bBLS). Furthermore, for new session data, a mutual information maximization constraint (MIMC) is introduced into the loss function of CTS to make the features' probability distribution sufficiently similar to that of the previous session, a new incremental BLS sequence (iBLS) is obtained by adding a small number of nodes to the previous model, and so on. Experiments are conducted based on the BCI Competition IV-2a dataset with two sessions and IV-2b dataset with five sessions, ASiBLS achieves average decoding accuracies of 79.89% and 87.04%, respectively. The kappa coefficient and forgetting rate are also used to evaluate the model performance. The results show that ASiBLS can adaptively generate an optimized and reduced model for each session successively, which has better plasticity in learning new knowledge and stability in retaining old knowledge as well.

Graph convolution network-based eeg signal analysis: a review

With the advancement of artificial intelligence technology, more and more effective methods are being used to identify and classify Electroencephalography (EEG) signals to address challenges in healthcare and brain-computer interface fields. The applications and major achievements of Graph Convolution Network (GCN) techniques in EEG signal analysis are reviewed in this paper. Through an exhaustive search of the published literature, a module-by-module discussion is carried out for the first time to address the current research status of GCN. An exhaustive classification of methods and a systematic analysis of key modules, such as brain map construction, node feature extraction, and GCN architecture design, are presented. In addition, we pay special attention to several key research issues related to GCN. This review enhances the understanding of the future potential of GCN in the field of EEG signal analysis. At the same time, several valuable development directions are sorted out for researchers in related fields, such as analysing the applicability of different GCN layers, building task-oriented GCN models, and improving adaptation to limited data.

Recognizing drivers' sleep onset by detecting slow eye movement using a parallel multimodal one-dimensional convolutional neural network

Slow eye movements (SEMs) are a reliable physiological marker of drivers' sleep onset, often accompanied by EEG alpha wave attenuation. A parallel multimodal 1D convolutional neural network (PM-1D-CNN) model is proposed to classify SEMs. The model uses two parallel 1D-CNN blocks to extract features from EOG and EEG signals, which are then fused and fed into fully connected layers for classification. Results show that the PM-1D-CNN outperforms the SGL-1D-CNN and Bimodal-LSTM networks in both subject-to-subject and cross-subject evaluations, confirming its effectiveness in detecting sleep onset.

A hybrid critical channels and optimal feature subset selection framework for EEG fatigue recognition

Fatigue driving is one of the potential factors threatening road safety, and monitoring drivers' mental state through electroencephalography (EEG) can effectively prevent such risks. In this paper, a new model, DE-GFRJMCMC, is proposed for selecting critical channels and optimal feature subsets from EEG data to improve the accuracy of fatigue driving recognition. The model is validated on the SEED-VIG dataset. The model first selects critical EEG channels using the Differential Evolution (DE) algorithm, extracting important electrode channel information to enhance recognition accuracy. These electrode channels are used to construct a Functional Brain Network (FBN), from which the topological feature set is extracted. Empirical Mode Decomposition (EMD) is then applied to extract the intrinsic mode components as network nodes, thereby reducing the influence of the number of electrode channels on the brain functional network. The topological features extracted from these components form the suboptimal feature set. To minimize redundant information, we propose an improved Reversible Jump Markov Chain Monte Carlo (RJMCMC) algorithm for selecting the optimal feature subset, ensuring both the efficiency and accuracy of fatigue recognition. The optimal feature subsets were input into various classifiers, and the results showed that the K-Nearest Neighbor (KNN)-based classifier achieved the highest recognition accuracy of 96.11% ± 0.43%, demonstrating the method's stability and robustness. Compared to similar studies, this model shows superior performance in fatigue driving recognition, which is of significant value for research on fatigue driving detection and prevention.

A portable EEG signal acquisition system and a limited-electrode channel classification network for SSVEP

Brain-computer interfaces (BCIs) have garnered significant research attention, yet their complexity has hindered widespread adoption in daily life. Most current electroencephalography (EEG) systems rely on wet electrodes and numerous electrodes to enhance signal quality, making them impractical for everyday use. Portable and wearable devices offer a promising solution, but the limited number of electrodes in specific regions can lead to missing channels and reduced BCI performance. To overcome these challenges and enable better integration of BCI systems with external devices, this study developed an EEG signal acquisition platform (Gaitech BCI) based on the Robot Operating System (ROS) using a 10-channel dry electrode EEG device. Additionally, a multi-scale channel attention selection network based on the Squeeze-and-Excitation (SE) module (SEMSCS) is proposed to improve the classification performance of portable BCI devices with limited channels. Steady-state visual evoked potential (SSVEP) data were collected using the developed BCI system to evaluate both the system and network performance. Offline data from ten subjects were analyzed using within-subject and cross-subject experiments, along with ablation studies. The results demonstrated that the SEMSCS model achieved better classification performance than the comparative reference model, even with a limited number of channels. Additionally, the implementation of online experiments offers a rational solution for controlling external devices via BCI.

Dynamic graph attention network based on multi-scale frequency domain features for motion imagery decoding in hemiplegic patients

Brain-computer interfaces (BCIs) establish a direct communication pathway between the brain and external devices and have been widely applied in upper limb rehabilitation for hemiplegic patients. However, significant individual variability in motor imagery electroencephalogram (MI-EEG) signals leads to poor generalization performance of MI-based BCI decoding methods to new patients. This paper proposes a Multi-scale Frequency domain Feature-based Dynamic graph Attention Network (MFF-DANet) for upper limb MI decoding in hemiplegic patients. MFF-DANet employs convolutional kernels of various scales to extract feature information across multiple frequency bands, followed by a channel attention-based average pooling operation to retain the most critical frequency domain features. Additionally, MFF-DANet integrates a graph attention convolutional network to capture spatial topological features across different electrode channels, utilizing electrode positions as prior knowledge to construct and update the graph adjacency matrix. We validated the performance of MFF-DANet on the public PhysioNet dataset, achieving optimal decoding accuracies of 61.6% for within-subject case and 52.7% for cross-subject case. t-Distributed Stochastic Neighbor Embedding (t-SNE) visualization of the features demonstrates the effectiveness of each designed module, and visualization of the adjacency matrix indicates that the extracted spatial topological features have physiological interpretability.

Real-time EEG-based detection of driving fatigue using a novel semi-dry electrode with self-replenishment of conductive fluid

A novel semi-dry electrode that can realize self-replenishment of conductive liquid is proposed in this study. Driving fatigue is detected by extracting the refined composite multiscale fluctuation dispersion entropy (RCMFDE) features in electroencephalogram (EEG) signals collected by this electrode. The results show that the new semi-dry electrode can automatically complete the conductive fluid supplement according to its own humidity conditions, which not only notably improves the effective working time, but also significantly reduces the skin impedance. By comparing with the classical entropy algorithms, the computational speed and the stability of the RCMFDE method are Substantially enhanced.

Driving Fatigue Detection Based on Hybrid Electroencephalography and Eye Tracking

EEG-based unimodal method has demonstrated significant success in the detection of driving fatigue. Nonetheless, data from a single modality might be not sufficient to optimize fatigue detection due to incomplete information. To address this limitation and enhance the performance of driving fatigue detection, a novel multimodal architecture combining hybrid electroencephalograph (EEG) and eye tracking data was proposed in this work. Specifically, the EEG and eye tracking data were separately input into encoders, generating two one-dimensional (1D) features. Subsequently, these 1D features were fed into a cross-modal predictive alignment module to improve fusion efficiency and two 1D attention modules to enhance feature representation. Furthermore, the fused features were recognized by a linear classifier. To evaluate the effectiveness of the proposed multimodal method, comprehensive validation tasks were conducted, including intra-session, cross-session, and cross-subject evaluations. In the intra-session task, the proposed architecture achieves an exceptional average accuracy of 99.93%. Moreover, in the cross-session task, our method demonstrates an average accuracy of 88.67%, surpassing the performance of EEG-only approach by 8.52%, eye tracking-only method by 5.92%, multimodal deep canonical correlation analysis (DCCA) technique by 0.42%, and multimodal deep generalized canonical correlation analysis (DGCCA) approach by 0.84%. Similarly, in the cross-subject task, the proposed approach achieves an average accuracy of 78.19%, outperforming EEG-only method by 5.87%, eye tracking-only approach by 4.21%, DCCA method by 0.55%, and DGCCA approach by 0.44%. The experimental results conclusively illustrate the superior effectiveness of the proposed method compared to both single modality approaches and canonical correlation analysis-based multimodal methods.

A Novel Multi-Feature Fusion Network With Spatial Partitioning Strategy and Cross-Attention for Armband-Based Gesture Recognition

Effectively integrating the time-space-frequency information of multi-modal signals from armband sensor, including surface electromyogram (sEMG) and accelerometer data, is critical for accurate gesture recognition. Existing approaches often neglect the abundant spatial relationships inherent in multi-channel sEMG signals obtained via armband sensors and face challenges in harnessing the correlations across multiple feature domains. To address this issue, we propose a novel multi-feature fusion network with spatial partitioning strategy and cross-attention (MFN-SPSCA) to improve the accuracy and robustness of gesture recognition. Specifically, a spatiotemporal graph convolution module with a spatial partitioning strategy is designed to capture potential spatial feature of multi-channel sEMG signals. Additionally, we design a cross-attention fusion module to learn and prioritize the importance and correlation of multi-feature domain. Extensive experiment demonstrate that the MFN-SPSCA method outperforms other state-of-the-art methods on self-collected dataset and the Ninapro DB5 dataset. Our work addresses the challenge of recognizing gestures from the multi-modal data collected by armband sensor, emphasizing the importance of integrating time-space-frequency information. Codes are available at https://github.com/ZJUTofBrainIntelligence/MFN-SPSCA.

Multi-branch fusion graph neural network based on multi-head attention for childhood seizure detection

The most common manifestation of neurological disorders in children is the occurrence of epileptic seizures. In this study, we propose a multi-branch graph convolutional network (MGCNA) framework with a multi-head attention mechanism for detecting seizures in children. The MGCNA framework extracts effective and reliable features from high-dimensional data, particularly by exploring the relationships between EEG features and electrodes and considering the spatial and temporal dependencies in epileptic brains. This method incorporates three graph learning approaches to systematically assess the connectivity and synchronization of multi-channel EEG signals. The multi-branch graph convolutional network is employed to dynamically learn temporal correlations and spatial topological structures. Utilizing the multi-head attention mechanism to process multi-branch graph features further enhances the capability to handle local features. Experimental results demonstrate that the MGCNA exhibits superior performance on patient-specific and patient-independent experiments. Our end-to-end model for automatic detection of epileptic seizures could be employed to assist in clinical decision-making.

Time-Frequency-Space EEG Decoding Model Based on Dense Graph Convolutional Network for Stroke

Stroke, a sudden cerebrovascular ailment resulting from brain tissue damage, has prompted the use of motor imagery (MI)-based Brain-Computer Interface (BCI) systems in stroke rehabilitation. However, analyzing EEG signals from stroke patients is challenging because of their low signal-to-noise ratio and high variability. Therefore, we propose a novel approach that combines the modified S-transform (MST) and a dense graph convolutional network (DenseGCN) algorithm to enhance the MI-BCI performance across time, frequency, and space domains. MST is a time-frequency analysis method that efficiently concentrates energy in EEG signals, while DenseGCN is a deep learning model that uses EEG feature maps from each layer as inputs for subsequent layers, facilitating feature reuse and hyper-parameters optimization. Our approach outperforms conventional networks, achieving a peak classification accuracy of 90.22% and an average information transfer rate (ITR) of 68.52 bits per minute. Moreover, we conduct an in-depth analysis of the event-related desynchronization/event-related synchronization (ERD/ERS) phenomenon in the deep-level EEG features of stroke patients. Our experimental results confirm the feasibility and efficacy of the proposed approach for MI-BCI rehabilitation systems.

DSFE: Decoding EEG-Based Finger Motor Imagery Using Feature-Dependent Frequency, Feature Fusion and Ensemble Learning

Accurate decoding finger motor imagery is essential for fine motor control using EEG signals. However, decoding finger motor imagery is particularly challenging compared with ordinary motor imagery. This paper proposed a novel EEG decoding method of feature-dependent frequency band selection, feature fusion, and ensemble learning (DSFE) for finger motor imagery. First, a feature-dependent frequency band selection method based on correlation coefficient (FDCC) was proposed to select feature-specific effective bands. Second, a feature fusion method was proposed to fuse different types of candidate features to produce multiple refined sets of decoding features. Finally, an ensemble model using the weighted voting strategy was proposed to make full use of these diverse sets of final features. The results on a public EEG dataset of five fingers motor imagery showed that the DSFE method is effective and achieves the highest decoding accuracy of 50.64%, which is 7.64% higher than existing studies using exactly the same data. The experiments further revealed that both the effective frequency bands of different subjects and the effective frequency bands of different types of features are different in finger motor imagery. Furthermore, compared with two-hand motor imagery, the effective decoding information of finger motor imagery is transferred to the lower frequency. The idea and findings in this paper provide a valuable perspective for understanding fine motor imagery in-depth.

A novel methodology for emotion recognition through 62-lead EEG signals: multilevel heterogeneous recurrence analysis

Objective

Recognizing emotions from electroencephalography (EEG) signals is a challenging task due to the complex, nonlinear, and nonstationary characteristics of brain activity. Traditional methods often fail to capture these subtle dynamics, while deep learning approaches lack explainability. In this research, we introduce a novel three-phase methodology integrating manifold embedding, multilevel heterogeneous recurrence analysis (MHRA), and ensemble learning to address these limitations in EEG-based emotion recognition.

Approach

The proposed methodology was evaluated using the SJTU-SEED IV database. We first applied uniform manifold approximation and projection (UMAP) for manifold embedding of the 62-lead EEG signals into a lower-dimensional space. We then developed MHRA to characterize the complex recurrence dynamics of brain activity across multiple transition levels. Finally, we employed tree-based ensemble learning methods to classify four emotions (neutral, sad, fear, happy) based on the extracted MHRA features.

Main results

Our approach achieved high performance, with an accuracy of 0.7885 and an AUC of 0.7552, outperforming existing methods on the same dataset. Additionally, our methodology provided the most consistent recognition performance across different emotions. Sensitivity analysis revealed specific MHRA metrics that were strongly associated with each emotion, offering valuable insights into the underlying neural dynamics.

Significance

This study presents a novel framework for EEG-based emotion recognition that effectively captures the complex nonlinear and nonstationary dynamics of brain activity while maintaining explainability. The proposed methodology offers significant potential for advancing our understanding of emotional processing and developing more reliable emotion recognition systems with broad applications in healthcare and beyond.

LGGNet: Learning From Local-Global-Graph Representations for Brain-Computer Interface

Neuropsychological studies suggest that co-operative activities among different brain functional areas drive high-level cognitive processes. To learn the brain activities within and among different functional areas of the brain, we propose local-global-graph network (LGGNet), a novel neurologically inspired graph neural network (GNN), to learn local-global-graph (LGG) representations of electroencephalography (EEG) for brain-computer interface (BCI). The input layer of LGGNet comprises a series of temporal convolutions with multiscale 1-D convolutional kernels and kernel-level attentive fusion. It captures temporal dynamics of EEG which then serves as input to the proposed local- and global-graph-filtering layers. Using a defined neurophysiologically meaningful set of local and global graphs, LGGNet models the complex relations within and among functional areas of the brain. Under the robust nested cross-validation settings, the proposed method is evaluated on three publicly available datasets for four types of cognitive classification tasks, namely the attention, fatigue, emotion, and preference classification tasks. LGGNet is compared with state-of-the-art (SOTA) methods, such as DeepConvNet, EEGNet, R2G-STNN, TSception, regularized graph neural network (RGNN), attention-based multiscale convolutional neural network-dynamical graph convolutional network (AMCNN-DGCN), hierarchical recurrent neural network (HRNN), and GraphNet. The results show that LGGNet outperforms these methods, and the improvements are statistically significant ( ) in most cases. The results show that bringing neuroscience prior knowledge into neural network design yields an improvement of classification performance. The source code can be found at https://github.com/yi-ding-cs/LGG.

A novel deep-learning model based on τ-shaped convolutional network (τNet) with long short-term memory (LSTM) for physiological fatigue detection from EEG and EOG signals

In recent years, fatigue driving has become the main cause of traffic accidents, leading to increased attention towards fatigue detection systems. However, the pooling and strided convolutional operations in fatigue detection algorithm based on traditional deep learning methods may led to the loss of some useful information. This paper proposed a novel τ -shaped convolutional network ( τ Net ) aiming to address this issue. Unlike traditional network structures, τ Net incorporates the operations of upsampling features and concatenating high- and low-level features, enabling full utilization of useful information. Moreover, considering that the fatigue state is a mental state involving temporal evolution, we proposed the novel long short-term memory (LSTM)- τ -shaped convolutional network (LSTM- τ Net ), a parallel structure composed of LSTM and τ Net for fatigue detection, where τ Net extracts time-invariant features with location information, and LSTM extracts long temporal dependencies. We compared LSTM- τ Net with six competing methods based on two datasets. Results showed that the proposed algorithm achieved higher classification accuracy than the other methods, with 94.25% on EEG data (binary classification) and 82.19% on EOG data (triple classification). Additionally, the proposed algorithm exhibits low computational cost, good training stability, and robustness against insufficient training. Therefore, it is promising for further implementation of fatigue online detection systems.

CSF-GTNet: A Novel Multi-Dimensional Feature Fusion Network Based on Convnext-GeLU- BiLSTM for EEG-Signals-Enabled Fatigue Driving Detection

Electroencephalography (EEG) signal has been recognized as an effective fatigue detection method, which can intuitively reflect the drivers' mental state. However, the research on multi-dimensional features in existing work could be much better. The instability and complexity of EEG signals will increase the difficulty of extracting data features. More importantly, most current work only treats deep learning models as classifiers. They ignored the features of different subjects learned by the model. Aiming at the above problems, this paper proposes a novel multi-dimensional feature fusion network, CSF-GTNet, based on time and space-frequency domains for fatigue detection. Specifically, it comprises Gaussian Time Domain Network (GTNet) and Pure Convolutional Spatial Frequency Domain Network (CSFNet). The experimental results show that the proposed method effectively distinguishes between alert and fatigue states. The accuracy rates are 85.16% and 81.48% on the self-made and SEED-VIG datasets, respectively, which are higher than the state-of-the-art methods. Moreover, we analyze the contribution of each brain region for fatigue detection through the brain topology map. In addition, we explore the changing trend of each frequency band and the significance between different subjects in the alert state and fatigue state through the heat map. Our research can provide new ideas in brain fatigue research and play a specific role in promoting the development of this field.

EEG-Based Brain Functional Network Analysis for Differential Identification of Dementia-Related Disorders and Their Onset

Diagnosing and treating dementia, including mild cognitive impairment (MCI), is challenging due to diverse disease types and overlapping symptoms. Early MCI detection is vital as it can precede dementia, yet distinguishing it from later stage dementia is intricate due to subtle symptoms. The primary objective of this study is to adopt a complex network perspective to unravel the underlying pathophysiological mechanisms of dementia-related disorders. Leveraging the extensive availability of electroencephalogram (EEG) data, our study focuses on the meticulous identification and analysis of EEG-based brain functional network (BFNs) associated with dementia-related disorders. To achieve this, we employ the Phase Lag Index (PLI) as a connectivity measure, offering a comprehensive view of neural interactions. To enhance the analytical rigor, we introduce a data-driven threshold selection technique. This innovative approach allows us to compare the topological structures of the formulated BFNs using complex network measures quantitatively and statistically. Furthermore, we harness the power of these BFNs by utilizing them as pre-defined graph inputs for a Graph Convolution Network (GCN-net) based approach. The results demonstrate that graph theory metrics, such as the rich-club coefficient, transitivity, and assortativity coefficients, effectively distinguish between MCI, Alzheimer's disease (AD) and vascular dementia (VD). Furthermore, GCN-net achieves high accuracy (95.07% delta, 80.62% theta) and F1-scores (0.92 delta, 0.67 theta), highlighting the effectiveness of EEG-based BFNs in the analysis of dementia-related disorders.

EEG Information Transfer Changes in Different Daily Fatigue Levels During Drowsy Driving

A significant issue for traffic safety has been drowsy driving for decades. A number of studies have investigated the effects of acute fatigue on spectral power; and recent research has revealed that drowsy driving is associated with a variety of brain connections in a specific cortico-cortical pathway. In spite of this, it is still unclear how different brain regions are connected in drowsy driving at different levels of daily fatigue. This study identified the brain connectivity-behavior relationship among three different daily fatigue levels (low-, median- and high-fatigue) with the EEG data transfer entropy. According to the results, only low- and medium-fatigue groups demonstrated an inverted U-shaped change in connectivity from high performance to poor behavioral performance. In addition, from low- to high-fatigue groups, connectivity magnitude decreased in the frontal region and increased in the occipital region. These study results suggest that brain connectivity and driving behavior would be affected by different levels of daily fatigue.

Resting-state EEG recorded with gel-based vs. consumer dry electrodes: spectral characteristics and across-device correlations

Introduction

Recordings of electroencephalographic (EEG) rhythms and their analyses have been instrumental in basic neuroscience, clinical diagnostics, and the field of brain-computer interfaces (BCIs). While in the past such measurements have been conducted mostly in laboratory settings, recent advancements in dry electrode technology pave way to a broader range of consumer and medical application because of their greater convenience compared to gel-based electrodes.

Methods

Here we conducted resting-state EEG recordings in two groups of healthy participants using three dry-electrode devices, the PSBD Headband, the PSBD Headphones and the Muse Headband, and one standard gel electrode-based system, the NVX. We examined signal quality for various spatial and spectral ranges which are essential for cognitive monitoring and consumer applications.

Results

Distinctive characteristics of signal quality were found, with the PSBD Headband showing sensitivity in low-frequency ranges and replicating the modulations of delta, theta and alpha power corresponding to the eyes-open and eyes-closed conditions, and the NVX system performing well in capturing high-frequency oscillations. The PSBD Headphones were more prone to low-frequency artifacts compared to the PSBD Headband, yet recorded modulations in the alpha power and had a strong alignment with the NVX at the higher EEG frequencies. The Muse Headband had several limitations in signal quality.

Discussion

We suggest that while dry-electrode technology appears to be appropriate for the EEG rhythm-based applications, the potential benefits of these technologies in terms of ease of use and accessibility should be carefully weighed against the capacity of each given system.

Exploring Adaptive Graph Topologies and Temporal Graph Networks for EEG-Based Depression Detection

In recent years, Graph Neural Networks (GNNs) based on deep learning techniques have achieved promising results in EEG-based depression detection tasks but still have some limitations. Firstly, most existing GNN-based methods use pre-computed graph adjacency matrices, which ignore the differences in brain networks between individuals. Additionally, methods based on graph-structured data do not consider the temporal dependency information of brain networks. To address these issues, we propose a deep learning algorithm that explores adaptive graph topologies and temporal graph networks for EEG-based depression detection. Specifically, we designed an Adaptive Graph Topology Generation (AGTG) module that can adaptively model the real-time connectivity of the brain networks, revealing differences between individuals. In addition, we designed a Graph Convolutional Gated Recurrent Unit (GCGRU) module to capture the temporal dynamical changes of brain networks. To further explore the differential features between depressed and healthy individuals, we adopt Graph Topology-based Max-Pooling (GTMP) module to extract graph representation vectors accurately. We conduct a comparative analysis with several advanced algorithms on both public and our own datasets. The results reveal that our final model achieves the highest Area Under the Receiver Operating Characteristic Curve (AUROC) on both datasets, with values of 83% and 99%, respectively. Furthermore, we perform extensive validation experiments demonstrating our proposed method's effectiveness and advantages. Finally, we present a comprehensive discussion on the differences in brain networks between healthy and depressed individuals based on the outputs of our final model's AGTG and GTMP modules.

A regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based cross-subject fatigue detection

Fatigue, one of the most important factors affecting road safety, has attracted many researchers' attention. Most existing fatigue detection methods are based on feature engineering and classification models. The feature engineering is greatly influenced by researchers' domain knowledge, which will lead to a poor performance in fatigue detection, especially in cross-subject experiment design. In addition, fatigue detection is often simplified as a classification problem of several discrete states. Models based on deep learning can realize automatic feature extraction without the limitation of researcher's domain knowledge. Therefore, this paper proposes a regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based (EEG-based) cross-subject fatigue detection. At the same time, a twofold random-offset zero-overlapping sampling method is proposed to train a bigger model and reduce overfitting. Compared with existing results, the proposed method achieves a much better result of 0.94 correlation coefficient (COR) and 0.09 root mean square error (RMSE) in a within-subject experiment design. What is more, there is no misclassification between awake and drowsy states. For cross-subject experiment design, the COR and RMSE are 0.79 and 0.15, respectively, which are close to the existing within-subject results and better than similar cross-subject results. The cross-subject regression model is very important for fatigue detection application since the fatigue indication is more precise than several discrete states and no model calibration is required for a new user. The twofold random-offset zero-overlapping sampling method can also be used as a reference by other EEG-based deep learning research.

Self-Attentive Channel-Connectivity Capsule Network for EEG-Based Driving Fatigue Detection

Deep neural networks have recently been successfully extended to EEG-based driving fatigue detection. Nevertheless, most existing models fail to reveal the intrinsic inter-channel relations that are known to be beneficial for EEG-based classification. Additionally, these models require substantial data for training, which is often impractical due to the high cost of data collection. To simultaneously address these two issues, we propose a Self-Attentive Channel-Connectivity Capsule Network (SACC-CapsNet) for EEG-based driving fatigue detection in this paper. SACC-CapsNet starts with a temporal-channel attention module to investigate the critical temporal information and important channels for driving fatigue detection, refining the input EEG signals. Subsequently, the refined EEG data are transformed into a channel covariance matrix to capture the inter-channel relations, followed by selective kernel attention to extract the highly discriminative channel-connectivity features. Finally, a capsule neural network is employed to effectively learn the relationships between connectivity features, which is more suitable for limited data. To confirm the effectiveness of SACC-CapsNet, we collected 24-channel EEG data from 31 subjects (mean age=23.13±2.68 years, male/female=18/13) in a simulated fatigue driving environment. Extensive experiments were conducted with the acquired data, and the comparison results show that our proposed model outperforms state-of-the-art methods. Additionally, the channel covariance matrix learned from SACC-CapsNet reveals that the frontal pole is most informative for detecting driving fatigue, followed by the parietal and central regions. Intriguingly, the temporal-channel attention module can enhance the significance of these critical regions, and the reconstructed channel covariance matrix generated by the decoder network of SACC-CapsNet can effectively preserve valuable information about them.

A regression method for EEG-based cross-dataset fatigue detection

Introduction: Fatigue is dangerous for certain jobs requiring continuous concentration. When faced with new datasets, the existing fatigue detection model needs a large amount of electroencephalogram (EEG) data for training, which is resource-consuming and impractical. Although the cross-dataset fatigue detection model does not need to be retrained, no one has studied this problem previously. Therefore, this study will focus on the design of the cross-dataset fatigue detection model. Methods: This study proposes a regression method for EEG-based cross-dataset fatigue detection. This method is similar to self-supervised learning and can be divided into two steps: pre-training and the domain-specific adaptive step. To extract specific features for different datasets, a pretext task is proposed to distinguish data on different datasets in the pre-training step. Then, in the domain-specific adaptation stage, these specific features are projected into a shared subspace. Moreover, the maximum mean discrepancy (MMD) is exploited to continuously narrow the differences in the subspace so that an inherent connection can be built between datasets. In addition, the attention mechanism is introduced to extract continuous information on spatial features, and the gated recurrent unit (GRU) is used to capture time series information. Results: The accuracy and root mean square error (RMSE) achieved by the proposed method are 59.10% and 0.27, respectively, which significantly outperforms state-of-the-art domain adaptation methods. Discussion: In addition, this study discusses the effect of labeled samples. When the number of labeled samples is 10% of the total number, the accuracy of the proposed model can reach 66.21%. This study fills a vacancy in the field of fatigue detection. In addition, the EEG-based cross-dataset fatigue detection method can be used for reference by other EEG-based deep learning research practices.

EEG driving fatigue detection based on log-Mel spectrogram and convolutional recurrent neural networks

Driver fatigue detection is one of the essential tools to reduce accidents and improve traffic safety. Its main challenge lies in the problem of how to identify the driver's fatigue state accurately. Existing detection methods include yawning and blinking based on facial expressions and physiological signals. Still, lighting and the environment affect the detection results based on facial expressions. In contrast, the electroencephalographic (EEG) signal is a physiological signal that directly responds to the human mental state, thus reducing the impact on the detection results. This paper proposes a log-Mel spectrogram and Convolution Recurrent Neural Network (CRNN) model based on EEG to implement driver fatigue detection. This structure allows the advantages of the different networks to be exploited to overcome the disadvantages of using them individually. The process is as follows: first, the original EEG signal is subjected to a one-dimensional convolution method to achieve a Short Time Fourier Transform (STFT) and passed through a Mel filter bank to obtain a logarithmic Mel spectrogram, and then the resulting logarithmic Mel spectrogram is fed into a fatigue detection model to complete the fatigue detection task for the EEG signals. The fatigue detection model consists of a 6-layer convolutional neural network (CNN), bi-directional recurrent neural networks (Bi-RNNs), and a classifier. In the modeling phase, spectrogram features are transported to the 6-layer CNN to automatically learn high-level features, thereby extracting temporal features in the bi-directional RNN to obtain spectrogram-temporal information. Finally, the alert or fatigue state is obtained by a classifier consisting of a fully connected layer, a ReLU activation function, and a softmax function. Experiments were conducted on publicly available datasets in this study. The results show that the method can accurately distinguish between alert and fatigue states with high stability. In addition, the performance of four existing methods was compared with the results of the proposed method, all of which showed that the proposed method could achieve the best results so far.

Driving Fatigue Detection with Three Non-Hair-Bearing EEG Channels and Modified Transformer Model

Driving fatigue is the main cause of traffic accidents, which seriously affects people's life and property safety. Many researchers have applied electroencephalogram (EEG) signals for driving fatigue detection to reduce negative effects. The main challenges are the practicality and accuracy of the EEG-based driving fatigue detection method when it is applied on the real road. In our previous study, we attempted to improve the practicality of fatigue detection based on the proposed non-hair-bearing (NHB) montage with fewer EEG channels, but the recognition accuracy was only 76.47% with the random forest (RF) model. In order to improve the accuracy with NHB montage, this study proposed an improved transformer architecture for one-dimensional feature vector classification based on introducing the Gated Linear Unit (GLU) in the Attention sub-block and Feed-Forward Networks (FFN) sub-block of a transformer, called GLU-Oneformer. Moreover, we constructed an NHB-EEG-based feature set, including the same EEG features (power ratio, approximate entropy, and mutual information (MI)) in our previous study, and the lateralization features of the power ratio and approximate entropy based on the strategy of brain lateralization. The results indicated that our GLU-Oneformer method significantly improved the recognition performance and achieved an accuracy of 86.97%. Our framework demonstrated that the combination of the NHB montage and the proposed GLU-Oneformer model could well support driving fatigue detection.

SAST-GCN: Segmentation Adaptive Spatial Temporal-Graph Convolutional Network for P3-Based Video Target Detection

Detecting video-induced P3 is crucial to building the video target detection system based on the brain-computer interface. However, studies have shown that the brain response patterns corresponding to video-induced P3 are dynamic and determined by the interaction of multiple brain regions. This paper proposes a segmentation adaptive spatial-temporal graph convolutional network (SAST-GCN) for P3-based video target detection. To make full use of the dynamic characteristics of the P3 signal data, the data is segmented according to the processing stages of the video-induced P3, and the brain network connections are constructed correspondingly. Then, the spatial-temporal feature of EEG data is extracted by adaptive spatial-temporal graph convolution to discriminate the target and non-target in the video. Especially, a style-based recalibration module is added to select feature maps with higher contributions and increase the feature extraction ability of the network. The experimental results demonstrate the superiority of our proposed model over the baseline methods. Also, the ablation experiments indicate that the segmentation of data to construct the brain connection can effectively improve the recognition performance by reflecting the dynamic connection relationship between EEG channels more accurately.

Wearable Supernumerary Robotic Limb System Using a Hybrid Control Approach Based on Motor Imagery and Object Detection

Motor disorder of upper limbs has seriously affected the daily life of the patients with hemiplegia after stroke. We developed a wearable supernumerary robotic limb (SRL) system using a hybrid control approach based on motor imagery (MI) and object detection for upper-limb motion assistance. SRL system included an SRL hardware subsystem and a hybrid control software subsystem. The system obtained the patient's motion intention through MI electroencephalogram (EEG) recognition method based on graph convolutional network (GCN) and gated recurrent unit network (GRU) to control the left and right movements of SRL, and the object detection technology was used together for a quick grasp of target objects to compensate for the disadvantages when using MI EEG alone like fewer control instructions and lower control efficiency. Offline training experiment was designed to obtain subjects' MI recognition models and evaluate the feasibility of the MI EEG recognition method; online control experiment was designed to verify the effectiveness of our wearable SRL system. The results showed that the proposed MI EEG recognition method (GCN+GRU) could effectively improve the MI classification accuracy (90.04% ± 2.36%) compared with traditional methods; all subjects were able to complete the target object grasping tasks within 23 seconds by controlling the SRL, and the highest average grasping success rate achieved 90.67% in bag grasping task. The SRL system can effectively assist people with upper-limb motor disorder to perform upper-limb tasks in daily life by natural human-robot interaction, and improve their ability of self-help and enhance their confidence of life.

Emotion Recognition from Physiological Channels Using Graph Neural Network

In recent years, a number of new research papers have emerged on the application of neural networks in affective computing. One of the newest trends observed is the utilization of graph neural networks (GNNs) to recognize emotions. The study presented in the paper follows this trend. Within the work, GraphSleepNet (a GNN for classifying the stages of sleep) was adjusted for emotion recognition and validated for this purpose. The key assumption of the validation was to analyze its correctness for the Circumplex model to further analyze the solution for emotion recognition in the Ekman modal. The novelty of this research is not only the utilization of a GNN network with GraphSleepNet architecture for emotion recognition, but also the analysis of the potential of emotion recognition based on differential entropy features in the Ekman model with a neutral state and a special focus on continuous emotion recognition during the performance of an activity The GNN was validated against the AMIGOS dataset. The research shows how the use of various modalities influences the correctness of the recognition of basic emotions and the neutral state. Moreover, the correctness of the recognition of basic emotions is validated for two configurations of the GNN. The results show numerous interesting observations for Ekman's model while the accuracy of the Circumplex model is similar to the baseline methods.

FB-CGANet: filter bank Channel Group Attention network for multi-class motor imagery classification

OBJECTIVE

Motor imagery-based brain computer interface (MI-BCI) is one of the most important BCI paradigms and can identify the target limb of subjects from the feature of MI-based Electroencephalography (EEG) signals. Deep learning methods, especially lightweight neural networks, provide an efficient technique for MI decoding, but the performance of lightweight neural networks is still limited and need further improving. This paper aimed to design a novel lightweight neural network for improving the performance of multi-class MI decoding.

APPROACH

A hybrid filter bank structure that can extract information in both time and frequency domain was proposed and combined with a novel channel attention method Channel Group Attention (CGA) to build a lightweight neural network Filter Bank Channel Group Attention Network (FB-CGANet). Accompanied with FB-CGANet, the Band Exchange data augmentation method was proposed to generate training data for networks with filter bank structure.

MAIN RESULTS

The proposed method can achieve higher 4-class average accuracy (79.4%) than compared methods on the BCI Competition IV IIa dataset in the experiment on the unseen evaluation data. Also, higher average accuracy (93.5%) than compared methods can be obtained in the cross-validation experiment.

SIGNIFICANCE

This work implies the effectiveness of channel attention and filter bank structure in lightweight neural networks and provides a novel option for multi-class motor imagery classification.