Bilinear neural network with 3-D attention for brain decoding of motor imagery movements from the human EEG

MSEI-ENet: A Multi-Scale EEG-Inception Integrated Encoder Network for Motor Imagery EEG Decoding

BACKGROUND

Due to complex signal characteristics and distinct individual differences, the decoding of a motor imagery electroencephalogram (MI-EEG) is limited by the unsatisfactory performance of suboptimal traditional models.

METHODS

A subject-independent model named MSEI-ENet is proposed for multiple-task MI-EEG decoding. It employs a specially designed multi-scale structure EEG-inception module (MSEI) for comprehensive feature learning. The encoder module further helps to detect discriminative information by its multi-head self-attention layer with a larger receptive field, which enhances feature representation and improves recognition efficacy.

RESULTS

The experimental results on Competition IV dataset 2a showed that our proposed model yielded an overall accuracy of 94.30%, MF1 score of 94.31%, and Kappa of 0.92.

CONCLUSIONS

A performance comparison with state-of-the-art methods demonstrated the effectiveness and generalizability of the proposed model on challenging multi-task MI-EEG decoding.

The Effect of Processing Techniques on the Classification Accuracy of Brain-Computer Interface Systems

Background/Objectives: Accurately classifying Electroencephalography (EEG) signals is essential for the effective operation of Brain-Computer Interfaces (BCI), which is needed for reliable neurorehabilitation applications. However, many factors in the processing pipeline can influence classification performance. The objective of this study is to assess the effects of different processing steps on classification accuracy in EEG-based BCI systems. Methods: This study explores the impact of various processing techniques and stages, including the FASTER algorithm for artifact rejection (AR), frequency filtering, transfer learning, and cropped training. The Physionet dataset, consisting of four motor imagery classes, was used as input due to its relatively large number of subjects. The raw EEG was tested with EEGNet and Shallow ConvNet. To examine the impact of adding a spatial dimension to the input data, we also used the Multi-branch Conv3D Net and developed two new models, Conv2D Net and Conv3D Net. Results: Our analysis showed that classification accuracy can be affected by many factors at every stage. Applying the AR method, for instance, can either enhance or degrade classification performance, depending on the subject and the specific network architecture. Transfer learning was effective in improving the performance of all networks for both raw and artifact-rejected data. However, the improvement in classification accuracy for artifact-rejected data was less pronounced compared to unfiltered data, resulting in reduced precision. For instance, the best classifier achieved 46.1% accuracy on unfiltered data, which increased to 63.5% with transfer learning. In the filtered case, accuracy rose from 45.5% to only 55.9% when transfer learning was applied. An unexpected outcome regarding frequency filtering was observed: networks demonstrated better classification performance when focusing on lower-frequency components. Higher frequency ranges were more discriminative for EEGNet and Shallow ConvNet, but only when cropped training was applied. Conclusions: The findings of this study highlight the complex interaction between processing techniques and neural network performance, emphasizing the necessity for customized processing approaches tailored to specific subjects and network architectures.

Large-Scale Cortical Network Analysis and Classification of MI-BCI Tasks Based on Bayesian Nonnegative Matrix Factorization

Motor imagery (MI) is a high-level cognitive process that has been widely applied to clinical rehabilitation and brain-computer interfaces (BCIs). However, the decoding of MI tasks still faces challenges, and the neural mechanisms underlying its application are unclear, which seriously hinders the development of MI-based clinical applications and BCIs. Here, we combined EEG source reconstruction and Bayesian nonnegative matrix factorization (NMF) methods to construct large-scale cortical networks of left-hand and right-hand MI tasks. Compared to right-hand MI, the results showed that the significantly increased functional network connectivities (FNCs) mainly located among the visual network (VN), sensorimotor network (SMN), right temporal network, right central executive network, and right parietal network in the left-hand MI at the β (13-30Hz) and all (8-30Hz) frequency bands. For the network properties analysis, we found that the clustering coefficient, global efficiency, and local efficiency were significantly increased and characteristic path length was significantly decreased in left-hand MI compared to right-hand MI at the β and all frequency bands. These network pattern differences indicated that the left-hand MI may need more modulation of multiple large-scale networks (i.e., VN and SMN) mainly located in the right hemisphere. Finally, based on the spatial pattern network of FNC and network properties, we propose a classification model. The proposed model achieves a top classification accuracy of 78.2% in cross-subject two-class MI-BCI tasks. Overall, our findings provide new insights into the neural mechanisms of MI and a potential network biomarker to identify MI-BCI tasks.

Identifying optimal channels and features for multi-participant motor imagery experiments across a participant's multi-day multi-class EEG data

The concept of the brain-computer interface (BCI) has become one of the popular research topics of recent times because it allows people to express their thoughts and control different applications and devices without actual movement. The communication between the brain and the computer or a machine is generally provided through Electroencephalogram (EEG) signals because they are cost-effective and easy to implement in normal life, not just in healthcare facilities. On the other hand, they are hard to process efficiently due to their nonlinearity and noisy nature. Thus, the field of BCI and EEG needs constant work and improvement. This paper focuses on generalizing the most efficient EEG channels and the most significant features of motor imagery (MI) signals by analyzing the recordings of one participant obtained over 20 different days. Because the classification performance usually decreases with an increasing number of class labels, we have realized the study by analyzing the signals through a new paradigm consisting of multi-class directional labels: right, left, forward, and backward. Afterward, the results are tested on EEG data obtained from 5 participants to see if the results are consistent with each other. The average accuracy of binary and multi-class classification using the Ensemble Subspace Discriminant classifier was found as 87.39 and 61.44%, respectively, with the most efficient 3-channel combination for daily BCI evaluation of one participant. On the other hand, the average accuracy of binary and multi-class classification was found as 71.84 and 50.42%, respectively, for 5 participants, with the most efficient channel combination of 4, where the first three are the same as the daily performance of one participant. During signal processing, the outliers of the signals were discarded by considering the channels separately. An algorithm was developed to dismiss the inconsistent samples within the classes. A novel adaptive filtering approach, correlation-based adaptive variational mode decomposition (CBAVMD), was proposed. The feature selection was realized based on the standard deviation values of the features between the classes. The paradigm based on the direction movements was found to be most effective, especially for binary classification of right and left directions. The generalization of effective channels and features was found to be generally successful.

Understanding the Role of Self-Attention in a Transformer Model for the Discrimination of SCD From MCI Using Resting-State EEG

The identification of EEG biomarkers to discriminate Subjective Cognitive Decline (SCD) from Mild Cognitive Impairment (MCI) conditions is a complex task which requires great clinical effort and expertise. We exploit the self-attention component of the Transformer architecture to obtain physiological explanations of the model's decisions in the discrimination of 56 SCD and 45 MCI patients using resting-state EEG. Specifically, an interpretability workflow leveraging attention scores and time-frequency analysis of EEG epochs through Continuous Wavelet Transform is proposed. In the classification framework, models are trained and validated with 5-fold cross-validation and evaluated on a test set obtained by selecting 20% of the total subjects. Ablation studies and hyperparameter tuning tests are conducted to identify the optimal model configuration. Results show that the best performing model, which achieves acceptable results both on epochs' and patients' classification, is capable of finding specific EEG patterns that highlight changes in the brain activity between the two conditions. We demonstrate the potential of attention weights as tools to guide experts in understanding which disease-relevant EEG features could be discriminative of SCD and MCI.

Revealing brain connectivity: graph embeddings for EEG representation learning and comparative analysis of structural and functional connectivity

This study employs deep learning techniques to present a compelling approach for modeling brain connectivity in EEG motor imagery classification through graph embedding. The compelling aspect of this study lies in its combination of graph embedding, deep learning, and different brain connectivity types, which not only enhances classification accuracy but also enriches the understanding of brain function. The approach yields high accuracy, providing valuable insights into brain connections and has potential applications in understanding neurological conditions. The proposed models consist of two distinct graph-based convolutional neural networks, each leveraging different types of brain connectivities to enhance classification performance and gain a deeper understanding of brain connections. The first model, Adjacency-based Convolutional Neural Network Model (Adj-CNNM), utilizes a graph representation based on structural brain connectivity to embed spatial information, distinguishing it from prior spatial filtering approaches dependent on subjects and tasks. Extensive tests on a benchmark dataset-IV-2a demonstrate that an accuracy of 72.77% is achieved by the Adj-CNNM, surpassing baseline and state-of-the-art methods. The second model, Phase Locking Value Convolutional Neural Network Model (PLV-CNNM), incorporates functional connectivity to overcome structural connectivity limitations and identifies connections between distinct brain regions. The PLV-CNNM achieves an overall accuracy of 75.10% across the 1-51 Hz frequency range. In the preferred 8-30 Hz frequency band, known for motor imagery data classification (including α, μ, and β waves), individual accuracies of 91.9%, 90.2%, and 85.8% are attained for α, μ, and β, respectively. Moreover, the model performs admirably with 84.3% accuracy when considering the entire 8-30 Hz band. Notably, the PLV-CNNM reveals robust connections between different brain regions during motor imagery tasks, including the frontal and central cortex and the central and parietal cortex. These findings provide valuable insights into brain connectivity patterns, enriching the comprehension of brain function. Additionally, the study offers a comprehensive comparative analysis of diverse brain connectivity modeling methods.

Graph Reasoning Module for Alzheimer's Disease Diagnosis: A Plug-and-Play Method

Recent advances in deep learning have led to increased adoption of convolutional neural networks (CNN) for structural magnetic resonance imaging (sMRI)-based Alzheimer's disease (AD) detection. AD results in widespread damage to neurons in different brain regions and destroys their connections. However, current CNN-based methods struggle to relate spatially distant information effectively. To solve this problem, we propose a graph reasoning module (GRM), which can be directly incorporated into CNN-based AD detection models to simulate the underlying relationship between different brain regions and boost AD diagnosis performance. Specifically, in GRM, an adaptive graph Transformer (AGT) block is designed to adaptively construct a graph representation based on the feature map given by CNN, a graph convolutional network (GCN) block is adopted to update the graph representation, and a feature map reconstruction (FMR) block is built to convert the learned graph representation to a feature map. Experimental results demonstrate that the insertion of the GRM in the existing AD classification model can increase its balanced accuracy by more than 4.3%. The GRM-embedded model achieves state-of-the-art performance compared with current deep learning-based AD diagnosis methods, with a balanced accuracy of 86.2%.

A shallow mirror transformer for subject-independent motor imagery BCI

OBJECTIVE

Motor imagery BCI plays an increasingly important role in motor disorders rehabilitation. However, the position and duration of the discriminative segment in an EEG trial vary from subject to subject and even trial to trial, and this leads to poor performance of subject-independent motor imagery classification. Thus, determining how to detect and utilize the discriminative signal segments is crucial for improving the performance of subject-independent motor imagery BCI.

APPROACH

In this paper, a shallow mirror transformer is proposed for subject-independent motor imagery EEG classification. Specifically, a multihead self-attention layer with a global receptive field is employed to detect and utilize the discriminative segment from the entire input EEG trial. Furthermore, the mirror EEG signal and the mirror network structure are constructed to improve the classification precision based on ensemble learning. Finally, the subject-independent setup was used to evaluate the shallow mirror transformer on motor imagery EEG signals from subjects existing in the training set and new subjects.

MAIN RESULTS

The experiments results on BCI Competition IV datasets 2a and 2b and the OpenBMI dataset demonstrated the promising effectiveness of the proposed shallow mirror transformer. The shallow mirror transformer obtained average accuracies of 74.48% and 76.1% for new subjects and existing subjects, respectively, which were highest among the compared state-of-the-art methods. In addition, visualization of the attention score showed the ability of discriminative EEG segment detection. This paper demonstrated that multihead self-attention is effective in capturing global EEG signal information in motor imagery classification.

SIGNIFICANCE

This study provides an effective model based on a multihead self-attention layer for subject-independent motor imagery-based BCIs. To the best of our knowledge, this is the shallowest transformer model available, in which a small number of parameters promotes the performance in motor imagery EEG classification for such a small sample problem.

[Multi-scale feature extraction and classification of motor imagery electroencephalography based on time series data enhancement]

The brain-computer interface (BCI) based on motor imagery electroencephalography (MI-EEG) enables direct information interaction between the human brain and external devices. In this paper, a multi-scale EEG feature extraction convolutional neural network model based on time series data enhancement is proposed for decoding MI-EEG signals. First, an EEG signals augmentation method was proposed that could increase the information content of training samples without changing the length of the time series, while retaining its original features completely. Then, multiple holistic and detailed features of the EEG data were adaptively extracted by multi-scale convolution module, and the features were fused and filtered by parallel residual module and channel attention. Finally, classification results were output by a fully connected network. The application experimental results on the BCI Competition IV 2a and 2b datasets showed that the proposed model achieved an average classification accuracy of 91.87% and 87.85% for the motor imagery task, respectively, which had high accuracy and strong robustness compared with existing baseline models. The proposed model does not require complex signals pre-processing operations and has the advantage of multi-scale feature extraction, which has high practical application value.

Convolutional Neural Network with a Topographic Representation Module for EEG-Based Brain-Computer Interfaces

Convolutional neural networks (CNNs) have shown great potential in the field of brain-computer interfaces (BCIs) due to their ability to directly process raw electroencephalogram (EEG) signals without artificial feature extraction. Some CNNs have achieved better classification accuracy than that of traditional methods. Raw EEG signals are usually represented as a two-dimensional (2-D) matrix composed of channels and time points, ignoring the spatial topological information of electrodes. Our goal is to make a CNN that takes raw EEG signals as inputs have the ability to learn spatial topological features and improve its classification performance while basically maintaining its original structure. We propose an EEG topographic representation module (TRM). This module consists of (1) a mapping block from raw EEG signals to a 3-D topographic map and (2) a convolution block from the topographic map to an output with the same size as the input. According to the size of the convolutional kernel used in the convolution block, we design two types of TRMs, namely TRM-(5,5) and TRM-(3,3). We embed the two TRM types into three widely used CNNs (ShallowConvNet, DeepConvNet and EEGNet) and test them on two publicly available datasets (the Emergency Braking During Simulated Driving Dataset (EBDSDD) and the High Gamma Dataset (HGD)). Results show that the classification accuracies of all three CNNs are improved on both datasets after using the TRMs. With TRM-(5,5), the average classification accuracies of DeepConvNet, EEGNet and ShallowConvNet are improved by 6.54%, 1.72% and 2.07% on the EBDSDD and by 6.05%, 3.02% and 5.14% on the HGD, respectively; with TRM-(3,3), they are improved by 7.76%, 1.71% and 2.17% on the EBDSDD and by 7.61%, 5.06% and 6.28% on the HGD, respectively. We improve the classification performance of three CNNs on both datasets through the use of TRMs, indicating that they have the capability to mine spatial topological EEG information. More importantly, since the output of a TRM has the same size as the input, CNNs with raw EEG signals as inputs can use this module without changing their original structures.

Relation Learning Using Temporal Episodes for Motor Imagery Brain-Computer Interfaces

For practical motor imagery (MI) brain-computer interface (BCI) applications, generating a reliable model for a target subject with few MI trials is important since the data collection process is labour-intensive and expensive. In this paper, we address this issue by proposing a few-shot learning method called temporal episode relation learning (TERL). TERL models MI with only limited trials from the target subject by the ability to compare MI trials through episode-based training. It can be directly applied to a new user without being re-trained, which is vital to improve user experience and realize real-world MIBCI applications. We develop a new and effective approach where, unlike the original episode learning, the temporal pattern between trials in each episode is encoded during the learning to boost the classification performance. We also perform an online evaluation simulation, in addition to the offline analysis that the previous studies only conduct, to better understand the performance of different approaches in real-world scenario. Extensive experiments are completed on four publicly available MIBCI datasets to evaluate the proposed TERL. Results show that TERL outperforms baseline and recent state-of-the-art methods, demonstrating competitive performance for subject-specific MIBCI where few trials are available from a target subject and a considerable number of trials from other source subjects.

Empirical comparison of deep learning methods for EEG decoding

Electroencephalography (EEG) is a technique that can be used in non-invasive brain-machine interface (BMI) systems to register brain electrical activity. The EEG signals are non-linear and non-stationary, making the decoding procedure a complex task. Deep learning techniques have been successfully applied in several research fields, often improving the results compared with traditional approaches. Therefore, it is believed that these techniques can also improve the process of decoding brain signals in BMI systems. In this work, we present the implementation of two deep learning-based decoders and we compared the results with other state of art deep learning methods. The first decoder uses long short-term memory (LSTM) recurrent neural network and the second, entitled EEGNet-LSTM, combines a well-known neural decoder based on convolutional neural networks, called EEGNet, with some LSTM layers. The decoders have been tested using data set 2a from BCI Competition IV, and the results showed that the EEGNet-LSTM decoder has been approximately 23% better than the competition-winning decoder. A Wilcoxon t-test showed a significant difference between the two decoders (Z = 2.524, p = 0.012). The LSTM-based decoder has been approximately 9% higher than the best decoder from the same competition. However, there was no significant difference (Z = 1.540, p = 0.123). In order to verify the replication of the EEGNet-LSTM decoder on another data, we performed a test with PhysioNet's Physiobank EEG Motor Movement/Imagery dataset. The EEGNet-LSTM presented a higher performance (0.85 accuracy) than the EEGNet (0.82 accuracy). The results of this work can be important for the development of new research, as well as EEG-based BMI systems, which can benefit from the high precision of neural decoders.

Subject adaptation convolutional neural network for EEG-based motor imagery classification

Objective.Deep transfer learning has been widely used to address the nonstationarity of electroencephalogram (EEG) data during motor imagery (MI) classification. However, previous deep learning approaches suffer from limited classification accuracy because the temporal and spatial features cannot be effectively extracted.Approach.Here, we propose a novel end-to-end deep subject adaptation convolutional neural network (SACNN) to handle the problem of EEG-based MI classification. Our proposed model jointly optimizes three modules, i.e. a feature extractor, a classifier, and a subject adapter. Specifically, the feature extractor simultaneously extracts the temporal and spatial features from the raw EEG data using a parallel multiscale convolution network. In addition, we design a subject adapter to reduce the feature distribution shift between the source and target subjects by using the maximum mean discrepancy. By minimizing the classification loss and the distribution discrepancy, the model is able to extract the temporal-spatial features to the prediction of a new subject.Main results.Extensive experiments are carried out on three EEG-based MI datasets, i.e. brain-computer interface (BCI) competition IV dataset IIb, BCI competition III dataset IVa, and BCI competition IV dataset I, and the average accuracy reaches to 86.42%, 81.71% and 79.35% on the three datasets respectively. Furthermore, the statistical analysis also indicates the significant performance improvement of SACNN.Significance.This paper reveals the importance of the temporal-spatial features on EEG-based MI classification task. Our proposed SACNN model can make fully use of the temporal-spatial information to achieve the purpose.

Deep learning in neuroimaging data analysis: Applications, challenges, and solutions

Methods for the analysis of neuroimaging data have advanced significantly since the beginning of neuroscience as a scientific discipline. Today, sophisticated statistical procedures allow us to examine complex multivariate patterns, however most of them are still constrained by assuming inherent linearity of neural processes. Here, we discuss a group of machine learning methods, called deep learning, which have drawn much attention in and outside the field of neuroscience in recent years and hold the potential to surpass the mentioned limitations. Firstly, we describe and explain the essential concepts in deep learning: the structure and the computational operations that allow deep models to learn. After that, we move to the most common applications of deep learning in neuroimaging data analysis: prediction of outcome, interpretation of internal representations, generation of synthetic data and segmentation. In the next section we present issues that deep learning poses, which concerns multidimensionality and multimodality of data, overfitting and computational cost, and propose possible solutions. Lastly, we discuss the current reach of DL usage in all the common applications in neuroimaging data analysis, where we consider the promise of multimodality, capability of processing raw data, and advanced visualization strategies. We identify research gaps, such as focusing on a limited number of criterion variables and the lack of a well-defined strategy for choosing architecture and hyperparameters. Furthermore, we talk about the possibility of conducting research with constructs that have been ignored so far or/and moving toward frameworks, such as RDoC, the potential of transfer learning and generation of synthetic data.

A new attention-based 3D densely connected cross-stage-partial network for motor imagery classification in BCI

OBJECTIVE

The challenge for motor imagery (MI) in brain-computer interface (BCI) systems is finding a reliable classification model that has high classification accuracy and excellent robustness. Currently, one of the main problems leading to degraded classification performance is the inaccuracy caused by nonstationarities and low signal-to-noise ratio in electroencephalogram (EEG) signals.

APPROACH

This study proposes a novel attention-based 3D densely connected cross-stage-partial network (DCSPNet) model to achieve efficient EEG-based MI classification. This is an end-to-end classification model framework based on the convolutional neural network (CNN) architecture. In this framework, to fully utilize the complementary features in each dimension, the optimal features are extracted adaptively from the EEG signals through the spatial-spectral-temporal (SST) attention mechanism. The 3D DCSPNet is introduced to reduce the gradient loss by segmenting the extracted feature maps to strengthen the network learning capability. Additionally, the design of the densely connected structure increases the robustness of the network.

MAIN RESULTS

The performance of the proposed method was evaluated using the BCI competition IV 2a and the high gamma dataset, achieving an average accuracy of 84.45% and 97.88%, respectively. Our method outperformed most state-of-the-art classification algorithms, demonstrating its effectiveness and strong generalization ability.

SIGNIFICANCE

The experimental results show that our method is promising for improving the performance of MI-BCI. As a general framework based on time-series classification, it can be applied to BCI-related fields.

EEGSym: Overcoming Inter-Subject Variability in Motor Imagery Based BCIs With Deep Learning

In this study, we present a new Deep Learning (DL) architecture for Motor Imagery (MI) based Brain Computer Interfaces (BCIs) called EEGSym. Our implementation aims to improve previous state-of-the-art performances on MI classification by overcoming inter-subject variability and reducing BCI inefficiency, which has been estimated to affect 10-50% of the population. This convolutional neural network includes the use of inception modules, residual connections and a design that introduces the symmetry of the brain through the mid-sagittal plane into the network architecture. It is complemented with a data augmentation technique that improves the generalization of the model and with the use of transfer learning across different datasets. We compare EEGSym's performance on inter-subject MI classification with ShallowConvNet, DeepConvNet, EEGNet and EEG-Inception. This comparison is performed on 5 publicly available datasets that include left or right hand motor imagery of 280 subjects. This population is the largest that has been evaluated in similar studies to date. EEGSym significantly outperforms the baseline models reaching accuracies of 88.6±9.0 on Physionet, 83.3±9.3 on OpenBMI, 85.1±9.5 on Kaya2018, 87.4±8.0 on Meng2019 and 90.2±6.5 on Stieger2021. At the same time, it allows 95.7% of the tested population (268 out of 280 users) to reach BCI control (≥70% accuracy). Furthermore, these results are achieved using only 16 electrodes of the more than 60 available on some datasets. Our implementation of EEGSym, which includes new advances for EEG processing with DL, outperforms previous state-of-the-art approaches on inter-subject MI classification.

How to successfully classify EEG in motor imagery BCI: a metrological analysis of the state of the art

Objective. Processing strategies are analysed with respect to the classification of electroencephalographic signals related to brain-computer interfaces based on motor imagery. A review of literature is carried out to understand the achievements in motor imagery classification, the most promising trends, and the challenges in replicating these results. Main focus is placed on performance by means of a rigorous metrological analysis carried out in compliance with the international vocabulary of metrology. Hence, classification accuracy and its uncertainty are considered, as well as repeatability and reproducibility.Approach. The paper works included in the review concern the classification of electroencephalographic signals in motor-imagery- based brain-computer interfaces. Article search was carried out in accordance with the PRISMA standard and 89 studies were included.Main results. Statistically-based analyses show that brain-inspired approaches are increasingly proposed, and that these are particularly successful in discriminating against multiple classes. Notably, many proposals involve convolutional neural networks. Instead, classical machine learning approaches are still effective for binary classifications. Many proposals combine common spatial pattern, least absolute shrinkage and selection operator, and support vector machines. Regarding reported classification accuracies, performance above the upper quartile is in the 85 % to 100 % range for the binary case and in the 83 % to 93 % range for multi-class one. Associated uncertainties are up to 6 % while repeatability for a predetermined dataset is up to 8 %. Reproducibility assessment was instead prevented by lack of standardization in experiments.Significance. By relying on the analysed studies, the reader is guided towards the development of a successful processing strategy as a crucial part of a brain-computer interface. Moreover, it is suggested that future studies should extend these approaches on data from more subjects and with custom experiments, even by investigating online operation. This would also enable the quantification of results reproducibility.