1
|
Qin Y, Yang B, Ke S, Liu P, Rong F, Xia X. M-FANet: Multi-Feature Attention Convolutional Neural Network for Motor Imagery Decoding. IEEE Trans Neural Syst Rehabil Eng 2024; 32:401-411. [PMID: 38194394 DOI: 10.1109/tnsre.2024.3351863] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Abstract
Motor imagery (MI) decoding methods are pivotal in advancing rehabilitation and motor control research. Effective extraction of spectral-spatial-temporal features is crucial for MI decoding from limited and low signal-to-noise ratio electroencephalogram (EEG) signal samples based on brain-computer interface (BCI). In this paper, we propose a lightweight Multi-Feature Attention Neural Network (M-FANet) for feature extraction and selection of multi-feature data. M-FANet employs several unique attention modules to eliminate redundant information in the frequency domain, enhance local spatial feature extraction and calibrate feature maps. We introduce a training method called Regularized Dropout (R-Drop) to address training-inference inconsistency caused by dropout and improve the model's generalization capability. We conduct extensive experiments on the BCI Competition IV 2a (BCIC-IV-2a) dataset and the 2019 World robot conference contest-BCI Robot Contest MI (WBCIC-MI) dataset. M-FANet achieves superior performance compared to state-of-the-art MI decoding methods, with 79.28% 4-class classification accuracy (kappa: 0.7259) on the BCIC-IV-2a dataset and 77.86% 3-class classification accuracy (kappa: 0.6650) on the WBCIC-MI dataset. The application of multi-feature attention modules and R-Drop in our lightweight model significantly enhances its performance, validated through comprehensive ablation experiments and visualizations.
Collapse
|
2
|
Sujatha Ravindran A, Contreras-Vidal J. An empirical comparison of deep learning explainability approaches for EEG using simulated ground truth. Sci Rep 2023; 13:17709. [PMID: 37853010 PMCID: PMC10584975 DOI: 10.1038/s41598-023-43871-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2023] [Accepted: 09/29/2023] [Indexed: 10/20/2023] Open
Abstract
Recent advancements in machine learning and deep learning (DL) based neural decoders have significantly improved decoding capabilities using scalp electroencephalography (EEG). However, the interpretability of DL models remains an under-explored area. In this study, we compared multiple model explanation methods to identify the most suitable method for EEG and understand when some of these approaches might fail. A simulation framework was developed to evaluate the robustness and sensitivity of twelve back-propagation-based visualization methods by comparing to ground truth features. Multiple methods tested here showed reliability issues after randomizing either model weights or labels: e.g., the saliency approach, which is the most used visualization technique in EEG, was not class or model-specific. We found that DeepLift was consistently accurate as well as robust to detect the three key attributes tested here (temporal, spatial, and spectral precision). Overall, this study provides a review of model explanation methods for DL-based neural decoders and recommendations to understand when some of these methods fail and what they can capture in EEG.
Collapse
Affiliation(s)
- Akshay Sujatha Ravindran
- Noninvasive Brain-Machine Interface System Laboratory, Department of Electrical and Computer Engineering, University of Houston, Houston, 77204, USA.
- IUCRC BRAIN, University of Houston, Houston, 77204, USA.
- Alto Neuroscience, Los Altos, CA, 94022, USA.
| | - Jose Contreras-Vidal
- Noninvasive Brain-Machine Interface System Laboratory, Department of Electrical and Computer Engineering, University of Houston, Houston, 77204, USA
- IUCRC BRAIN, University of Houston, Houston, 77204, USA
| |
Collapse
|
3
|
Sun C, Mou C. Survey on the research direction of EEG-based signal processing. Front Neurosci 2023; 17:1203059. [PMID: 37521708 PMCID: PMC10372445 DOI: 10.3389/fnins.2023.1203059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Accepted: 06/16/2023] [Indexed: 08/01/2023] Open
Abstract
Electroencephalography (EEG) is increasingly important in Brain-Computer Interface (BCI) systems due to its portability and simplicity. In this paper, we provide a comprehensive review of research on EEG signal processing techniques since 2021, with a focus on preprocessing, feature extraction, and classification methods. We analyzed 61 research articles retrieved from academic search engines, including CNKI, PubMed, Nature, IEEE Xplore, and Science Direct. For preprocessing, we focus on innovatively proposed preprocessing methods, channel selection, and data augmentation. Data augmentation is classified into conventional methods (sliding windows, segmentation and recombination, and noise injection) and deep learning methods [Generative Adversarial Networks (GAN) and Variation AutoEncoder (VAE)]. We also pay attention to the application of deep learning, and multi-method fusion approaches, including both conventional algorithm fusion and fusion between conventional algorithms and deep learning. Our analysis identifies 35 (57.4%), 18 (29.5%), and 37 (60.7%) studies in the directions of preprocessing, feature extraction, and classification, respectively. We find that preprocessing methods have become widely used in EEG classification (96.7% of reviewed papers) and comparative experiments have been conducted in some studies to validate preprocessing. We also discussed the adoption of channel selection and data augmentation and concluded several mentionable matters about data augmentation. Furthermore, deep learning methods have shown great promise in EEG classification, with Convolutional Neural Networks (CNNs) being the main structure of deep neural networks (92.3% of deep learning papers). We summarize and analyze several innovative neural networks, including CNNs and multi-structure fusion. However, we also identified several problems and limitations of current deep learning techniques in EEG classification, including inappropriate input, low cross-subject accuracy, unbalanced between parameters and time costs, and a lack of interpretability. Finally, we highlight the emerging trend of multi-method fusion approaches (49.2% of reviewed papers) and analyze the data and some examples. We also provide insights into some challenges of multi-method fusion. Our review lays a foundation for future studies to improve EEG classification performance.
Collapse
|
4
|
Wang J, Cheng S, Tian J, Gao Y. A 2D CNN-LSTM hybrid algorithm using time series segments of EEG data for motor imagery classification. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104627] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/09/2023]
|
5
|
Chen J, Wang D, Yi W, Xu M, Tan X. Filter bank sinc-convolutional network with channel self-attention for high performance motor imagery decoding. J Neural Eng 2023; 20. [PMID: 36763992 DOI: 10.1088/1741-2552/acbb2c] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Accepted: 02/10/2023] [Indexed: 02/12/2023]
Abstract
Objective.Motor Imagery Brain-Computer Interface (MI-BCI) is an active Brain-Computer Interface (BCI) paradigm focusing on the identification of motor intention, which is one of the most important non-invasive BCI paradigms. In MI-BCI studies, deep learning-based methods (especially lightweight networks) have attracted more attention in recent years, but the decoding performance still needs further improving.Approach.To solve this problem, we designed a filter bank structure with sinc-convolutional layers for spatio-temporal feature extraction of MI-electroencephalography in four motor rhythms. The Channel Self-Attention method was introduced for feature selection based on both global and local information, so as to build a model called Filter Bank Sinc-convolutional Network with Channel Self-Attention for high performance MI-decoding. Also, we proposed a data augmentation method based on multivariate empirical mode decomposition to improve the generalization capability of the model.Main results.We performed an intra-subject evaluation experiment on unseen data of three open MI datasets. The proposed method achieved mean accuracy of 78.20% (4-class scenario) on BCI Competition IV IIa, 87.34% (2-class scenario) on BCI Competition IV IIb, and 72.03% (2-class scenario) on Open Brain Machine Interface (OpenBMI) dataset, which are significantly higher than those of compared deep learning-based methods by at least 3.05% (p= 0.0469), 3.18% (p= 0.0371), and 2.27% (p= 0.0024) respectively.Significance.This work provides a new option for deep learning-based MI decoding, which can be employed for building BCI systems for motor rehabilitation.
Collapse
Affiliation(s)
- Jiaming Chen
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Dan Wang
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Weibo Yi
- Beijing Machine and Equipment Institute, Beijing, People's Republic of China
| | - Meng Xu
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| | - Xiyue Tan
- Faculty of Information Technology, Beijing University of Technology, Beijing, People's Republic of China
| |
Collapse
|
6
|
Analysis of Altered Brain Dynamics During Episodic Recall and Detection of Generalized Anxiety Disorder. Neuroscience 2023:S0306-4522(23)00032-5. [PMID: 36707018 DOI: 10.1016/j.neuroscience.2023.01.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2022] [Revised: 12/12/2022] [Accepted: 01/19/2023] [Indexed: 01/26/2023]
Abstract
Numerous blood oxygenation level-dependent (BOLD) imaging studies have shown that generalized anxiety disorder (GAD) can lead to abnormal activation of specific brain regions in patients. However, these methods lack sufficient temporal resolution to explain the underlying brain dynamics of GAD. The electroencephalogram (EEG) microstate allows us to explore brain dynamics at the subsecond level. We performed microstate analysis and source localization on the EEG data of 15 GADs and 14 healthy controls (HCs). We found two kinds of noncanonical microstate topologies (MS-4 and MS-5) in the episodic recall tasks. Compared with HCs, the duration and coverage of MS-5 were significantly reduced in GADs and positively correlated with the GAD-7 scores. The results of source localization showed obvious activation in the prefrontal lobe, parietal lobe, temporal lobe, and fusiform gyri. Moreover, we propose an improved capsule network to capture EEG spatial features and combine them with temporal parameters of microstates for more reliable GAD detection. The sensor-level EEG data and the source-level EEG data obtained by source reconstruction are used as input to the model. The optimal configuration combined the spatial features of source-level data with microstate features and achieved the highest classification accuracy. Collectively, the statistical results indicated remarkable differences in dynamic brain parameters between the two groups, and patients with GAD may have abnormalities in their higher sensory cortex that affect the processing of anxiety signals. Furthermore, our proposed fusion framework provides a reliable method for GAD automatic detection.
Collapse
|
7
|
Nagarajan A, Robinson N, Guan C. Relevance-based channel selection in motor imagery brain-computer interface. J Neural Eng 2023; 20. [PMID: 36548997 DOI: 10.1088/1741-2552/acae07] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Accepted: 12/22/2022] [Indexed: 12/24/2022]
Abstract
Objective.Channel selection in the electroencephalogram (EEG)-based brain-computer interface (BCI) has been extensively studied for over two decades, with the goal being to select optimal subject-specific channels that can enhance the overall decoding efficacy of the BCI. With the emergence of deep learning (DL)-based BCI models, there arises a need for fresh perspectives and novel techniques to conduct channel selection. In this regard, subject-independent channel selection is relevant, since DL models trained using cross-subject data offer superior performance, and the impact of inherent inter-subject variability of EEG characteristics on subject-independent DL training is not yet fully understood.Approach.Here, we propose a novel methodology for implementing subject-independent channel selection in DL-based motor imagery (MI)-BCI, using layer-wise relevance propagation (LRP) and neural network pruning. Experiments were conducted using Deep ConvNet and 62-channel MI data from the Korea University EEG dataset.Main Results.Using our proposed methodology, we achieved a 61% reduction in the number of channels without any significant drop (p = 0.09) in subject-independent classification accuracy, due to the selection of highly relevant channels by LRP. LRP relevance-based channel selections provide significantly better accuracies compared to conventional weight-based selections while using less than 40% of the total number of channels, with differences in accuracies ranging from 5.96% to 1.72%. The performance of the adapted sparse-LRP model using only 16% of the total number of channels is similar to that of the adapted baseline model (p = 0.13). Furthermore, the accuracy of the adapted sparse-LRP model using only 35% of the total number of channels exceeded that of the adapted baseline model by 0.53% (p = 0.81). Analyses of channels chosen by LRP confirm the neurophysiological plausibility of selection, and emphasize the influence of motor, parietal, and occipital channels in MI-EEG classification.Significance.The proposed method addresses a traditional issue in EEG-BCI decoding, while being relevant and applicable to the latest developments in the field of BCI. We believe that our work brings forth an interesting and important application of model interpretability as a problem-solving technique.
Collapse
Affiliation(s)
- Aarthy Nagarajan
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798, Singapore
| | - Neethu Robinson
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798, Singapore
| | - Cuntai Guan
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798, Singapore
| |
Collapse
|
8
|
Gao C, Liu W, Yang X. Convolutional neural network and riemannian geometry hybrid approach for motor imagery classification. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.08.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
9
|
Ma J, Yang B, Qiu W, Li Y, Gao S, Xia X. A large EEG dataset for studying cross-session variability in motor imagery brain-computer interface. Sci Data 2022; 9:531. [PMID: 36050394 PMCID: PMC9436944 DOI: 10.1038/s41597-022-01647-1] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2022] [Accepted: 08/09/2022] [Indexed: 11/24/2022] Open
Abstract
In building a practical and robust brain-computer interface (BCI), the classification of motor imagery (MI) from electroencephalography (EEG) across multiple days is a long-standing challenge due to the large variability of the EEG signals. We collected a large dataset of MI from 5 different days with 25 subjects, the first open-access dataset to address BCI issues across 5 different days with a large number of subjects. The dataset includes 5 session data from 5 different days (2–3 days apart) for each subject. Each session contains 100 trials of left-hand and right-hand MI. In this report, we provide the benchmarking classification accuracy for three conditions, namely, within-session classification (WS), cross-session classification (CS), and cross-session adaptation (CSA), with subject-specific models. WS achieves an average classification accuracy of up to 68.8%, while CS degrades the accuracy to 53.7% due to the cross-session variability. However, by adaptation, CSA improves the accuracy to 78.9%. We anticipate this new dataset will significantly push further progress in MI BCI research in addressing the cross-session and cross-subject challenge. Measurement(s) | Electroencephalography | Technology Type(s) | motor imagery |
Collapse
Affiliation(s)
- Jun Ma
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Banghua Yang
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China. .,Engineering Research Center of Traditional Chinese Medicine Intelligent Rehabilitation, Ministry of Education, Shanghai, China.
| | - Wenzheng Qiu
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Yunzhe Li
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Shouwei Gao
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Xinxing Xia
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| |
Collapse
|
10
|
Yang B, Ma J, Qiu W, Zhang J, Wang X. The unilateral upper limb classification from fMRI-weighted EEG signals using convolutional neural network. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103855] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
11
|
Tibrewal N, Leeuwis N, Alimardani M. Classification of motor imagery EEG using deep learning increases performance in inefficient BCI users. PLoS One 2022; 17:e0268880. [PMID: 35867703 PMCID: PMC9307149 DOI: 10.1371/journal.pone.0268880] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2021] [Accepted: 05/11/2022] [Indexed: 11/19/2022] Open
Abstract
Motor Imagery Brain-Computer Interfaces (MI-BCIs) are AI-driven systems that capture brain activity patterns associated with mental imagination of movement and convert them into commands for external devices. Traditionally, MI-BCIs operate on Machine Learning (ML) algorithms, which require extensive signal processing and feature engineering to extract changes in sensorimotor rhythms (SMR). In recent years, Deep Learning (DL) models have gained popularity for EEG classification as they provide a solution for automatic extraction of spatio-temporal features in the signals. However, past BCI studies that employed DL models, only attempted them with a small group of participants, without investigating the effectiveness of this approach for different user groups such as inefficient users. BCI inefficiency is a known and unsolved problem within BCI literature, generally defined as the inability of the user to produce the desired SMR patterns for the BCI classifier. In this study, we evaluated the effectiveness of DL models in capturing MI features particularly in the inefficient users. EEG signals from 54 subjects who performed a MI task of left- or right-hand grasp were recorded to compare the performance of two classification approaches; a ML approach vs. a DL approach. In the ML approach, Common Spatial Patterns (CSP) was used for feature extraction and then Linear Discriminant Analysis (LDA) model was employed for binary classification of the MI task. In the DL approach, a Convolutional Neural Network (CNN) model was constructed on the raw EEG signals. Additionally, subjects were divided into high vs. low performers based on their online BCI accuracy and the difference between the two classifiers’ performance was compared between groups. Our results showed that the CNN model improved the classification accuracy for all subjects within the range of 2.37 to 28.28%, but more importantly, this improvement was significantly larger for low performers. Our findings show promise for employment of DL models on raw EEG signals in future MI-BCI systems, particularly for BCI inefficient users who are unable to produce desired sensorimotor patterns for conventional ML approaches.
Collapse
Affiliation(s)
- Navneet Tibrewal
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands
| | - Nikki Leeuwis
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands
- Research Department, Unravel Research, Utrecht, The Netherlands
| | - Maryam Alimardani
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands
- * E-mail:
| |
Collapse
|
12
|
Zhang J, Li K. A Pruned Deep Learning Approach for Classification of Motor Imagery Electroencephalography Signals. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:4072-4075. [PMID: 36085842 DOI: 10.1109/embc48229.2022.9871078] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The Deep Learning (DL) approach has been gaining much popularity in recent years in the development of electroencephalogram (EEG) based Motor Imagery (MI) Brain-Computer Interface (BCI) systems, aiming to improve the performance of existing stroke rehabilitation strategies. A complex deep neural network structure has lots of neurons with thousands of parameters to optimize, and a great deal of data is often required to train the network and the training process can take an extremely long time. High training costs and high model complexity not only have negative impacts on the performance of the BCI system but also on its applicability to meet the real-time requirement to support the rehabilitation exercises of patients. To tackle the challenge, a contribution-based neuron selection method is proposed in this paper. A Convolutional Neural Network (CNN) based motor imagery classification framework is implemented, and a neuron pruning approach is developed and applied. The temporal and spatial features of EEG signals are captured by the CNN layers, and then the fast recursive algorithm (FRA) is applied to prune redundant parameters in the fully connected layers which reduces the computation cost of the CNN model without affecting its performance. The experimental results show that the proposed method can achieve up to 50% model size reduction and 67.09% computation savings.
Collapse
|
13
|
Sellitto M, Terenzi D, Starita F, di Pellegrino G, Battaglia S. The Cost of Imagined Actions in a Reward-Valuation Task. Brain Sci 2022; 12:brainsci12050582. [PMID: 35624971 PMCID: PMC9139426 DOI: 10.3390/brainsci12050582] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2022] [Revised: 04/26/2022] [Accepted: 04/27/2022] [Indexed: 01/26/2023] Open
Abstract
Growing evidence suggests that humans and other animals assign value to a stimulus based not only on its inherent rewarding properties, but also on the costs of the action required to obtain it, such as the cost of time. Here, we examined whether such cost also occurs for mentally simulated actions. Healthy volunteers indicated their subjective value for snack foods while the time to imagine performing the action to obtain the different stimuli was manipulated. In each trial, the picture of one food item and a home position connected through a path were displayed on a computer screen. The path could be either large or thin. Participants first rated the stimulus, and then imagined moving the mouse cursor along the path from the starting position to the food location. They reported the onset and offset of the imagined movements with a button press. Two main results emerged. First, imagery times were significantly longer for the thin than the large path. Second, participants liked significantly less the snack foods associated with the thin path (i.e., with longer imagery time), possibly because the passage of time strictly associated with action imagery discounts the value of the reward. Importantly, such effects were absent in a control group of participants who performed an identical valuation task, except that no action imagery was required. Our findings hint at the idea that imagined actions, like real actions, carry a cost that affects deeply how people assign value to the stimuli in their environment.
Collapse
Affiliation(s)
- Manuela Sellitto
- Centre for Studies and Research in Cognitive Neuroscience, Department of Psychology, University of Bologna, 40126 Bologna, Italy; (M.S.); (F.S.)
- School of Psychology, Bangor University, Bangor LL57 2AS, UK
| | - Damiano Terenzi
- Department of Decision Neuroscience and Nutrition, German Institute of Human Nutrition (DIfE), 14558 Potsdam-Rehbrücke, Germany;
- Charité—Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin Institute of Health, Neuroscience Research Center, 10117 Berlin, Germany
| | - Francesca Starita
- Centre for Studies and Research in Cognitive Neuroscience, Department of Psychology, University of Bologna, 40126 Bologna, Italy; (M.S.); (F.S.)
| | - Giuseppe di Pellegrino
- Centre for Studies and Research in Cognitive Neuroscience, Department of Psychology, University of Bologna, 40126 Bologna, Italy; (M.S.); (F.S.)
- Correspondence: (G.d.P.); (S.B.)
| | - Simone Battaglia
- Centre for Studies and Research in Cognitive Neuroscience, Department of Psychology, University of Bologna, 40126 Bologna, Italy; (M.S.); (F.S.)
- School of Psychology, Bangor University, Bangor LL57 2AS, UK
- Correspondence: (G.d.P.); (S.B.)
| |
Collapse
|
14
|
Autthasan P, Chaisaen R, Sudhawiyangkul T, Rangpong P, Kiatthaveephong S, Dilokthanakul N, Bhakdisongkhram G, Phan H, Guan C, Wilaiprasitporn T. MIN2Net: End-to-End Multi-Task Learning for Subject-Independent Motor Imagery EEG Classification. IEEE Trans Biomed Eng 2021; 69:2105-2118. [PMID: 34932469 DOI: 10.1109/tbme.2021.3137184] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
OBJECTIVE Advances in the motor imagery (MI)-based brain-computer interfaces (BCIs) allow control of several applications by decoding neurophysiological phenomena, which are usually recorded by electroencephalography (EEG) using a non-invasive technique. Despite significant advances in MI-based BCI, EEG rhythms are specific to a subject and various changes over time. These issues point to significant challenges to enhance the classification performance, especially in a subject-independent manner. METHODS To overcome these challenges, we propose MIN2Net, a novel end-to-end multi-task learning to tackle this task. We integrate deep metric learning into a multi-task autoencoder to learn a compact and discriminative latent representation from EEG and perform classification simultaneously. RESULTS This approach reduces the complexity in pre-processing, results in significant performance improvement on EEG classification. Experimental results in a subject-independent manner show that MIN2Net outperforms the state-of-the-art techniques, achieving an F1-score improvement of 6.72 %, and 2.23 % on the SMR-BCI, and OpenBMI datasets, respectively. CONCLUSION We demonstrate that MIN2Net improves discriminative information in the latent representation. SIGNIFICANCE This study indicates the possibility and practicality of using this model to develop MI-based BCI applications for new users without calibration.
Collapse
|
15
|
Qiu W, Yang B, Ma J, Gao S, Zhu Y, Wang W. The Paradigm Design of a Novel 2-class Unilateral Upper Limb Motor Imagery Tasks and its EEG Signal Classification. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:152-155. [PMID: 34891260 DOI: 10.1109/embc46164.2021.9630837] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Multitasking motor imagery (MI) of the unilateral upper limb is potentially more valuable in stroke rehabilitation than the current conventional MI in both hands. In this paper, a novel experimental paradigm was designed to imagine two motions of unilateral upper limb, which is hand gripping and releasing, and elbow reciprocating left and right. During this experiment, the electroencephalogram (EEG) signals were collected from 10 subjects. The time and frequency domains of the EEG signals were analyzed and visualized, indicating the presence of different Event-Related Desynchronization (ERD) or Event-Related Synchronization (ERS) for the two tasks. Then the two tasks were classified through three different EEG decoding methods, in which the optimized convolutional neural network (CNN) based on FBCNet achieved an average accuracy of 67.8%, obtaining a good recognition result. This work not only can advance the studies of MI decoding of unilateral upper limb, but also can provide a basis for better upper limb stroke rehabilitation in MI-BCI.
Collapse
|
16
|
Jeong JH, Choi JH, Kim KT, Lee SJ, Kim DJ, Kim HM. Multi-Domain Convolutional Neural Networks for Lower-Limb Motor Imagery Using Dry vs. Wet Electrodes. SENSORS 2021; 21:s21196672. [PMID: 34640992 PMCID: PMC8513081 DOI: 10.3390/s21196672] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Revised: 10/04/2021] [Accepted: 10/05/2021] [Indexed: 11/29/2022]
Abstract
Motor imagery (MI) brain–computer interfaces (BCIs) have been used for a wide variety of applications due to their intuitive matching between the user’s intentions and the performance of tasks. Applying dry electroencephalography (EEG) electrodes to MI BCI applications can resolve many constraints and achieve practicality. In this study, we propose a multi-domain convolutional neural networks (MD-CNN) model that learns subject-specific and electrode-dependent EEG features using a multi-domain structure to improve the classification accuracy of dry electrode MI BCIs. The proposed MD-CNN model is composed of learning layers for three domain representations (time, spatial, and phase). We first evaluated the proposed MD-CNN model using a public dataset to confirm 78.96% classification accuracy for multi-class classification (chance level accuracy: 30%). After that, 10 healthy subjects participated and performed three classes of MI tasks related to lower-limb movement (gait, sitting down, and resting) over two sessions (dry and wet electrodes). Consequently, the proposed MD-CNN model achieved the highest classification accuracy (dry: 58.44%; wet: 58.66%; chance level accuracy: 43.33%) with a three-class classifier and the lowest difference in accuracy between the two electrode types (0.22%, d = 0.0292) compared with the conventional classifiers (FBCSP, EEGNet, ShallowConvNet, and DeepConvNet) that used only a single domain. We expect that the proposed MD-CNN model could be applied for developing robust MI BCI systems with dry electrodes.
Collapse
Affiliation(s)
- Ji-Hyeok Jeong
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Korea
| | - Jun-Hyuk Choi
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Division of Bio-Medical Science & Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Korea
| | - Keun-Tae Kim
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
| | - Song-Joo Lee
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Division of Bio-Medical Science & Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Korea
| | - Dong-Joo Kim
- Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Korea
- Department of Neurology, Korea University College of Medicine, Seoul 02841, Korea
- Department of Artificial Intelligence, Korea University, Seoul 02841, Korea
- Correspondence: (D.-J.K.); (H.-M.K.)
| | - Hyung-Min Kim
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Division of Bio-Medical Science & Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Korea
- Correspondence: (D.-J.K.); (H.-M.K.)
| |
Collapse
|