A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers

Formulation of Common Spatial Patterns for Multi-Task Hyperscanning BCI

This work proposes a new formulation for common spatial patterns (CSP), often used as a powerful feature extraction technique in brain-computer interfacing (BCI) and other neurological studies. In this approach, applied to multiple subjects' data and named as hyperCSP, the individual covariance and mutual correlation matrices between multiple simultaneously recorded subjects' electroencephalograms are exploited in the CSP formulation. This method aims at effectively isolating the common motor task between multiple heads and alleviate the effects of other spurious or undesired tasks inherently or intentionally performed by the subjects. This technique can provide a satisfactory classification performance while using small data size and low computational complexity. By using the proposed hyperCSP followed by support vector machines classifier, we obtained a classification accuracy of 81.82% over 8 trials in the presence of strong undesired tasks. We hope that this method could reduce the training error in multi-task BCI scenarios. The recorded valuable motor-related hyperscanning dataset is available for public use to promote the research in this area.

Automatic detection of epilepsy from EEGs using a temporal convolutional network with a self-attention layer

BACKGROUND

Over 60% of epilepsy patients globally are children, whose early diagnosis and treatment are critical for their development and can substantially reduce the disease's burden on both families and society. Numerous algorithms for automated epilepsy detection from EEGs have been proposed. Yet, the occurrence of epileptic seizures during an EEG exam cannot always be guaranteed in clinical practice. Models that exclusively use seizure EEGs for detection risk artificially enhanced performance metrics. Therefore, there is a pressing need for a universally applicable model that can perform automatic epilepsy detection in a variety of complex real-world scenarios.

METHOD

To address this problem, we have devised a novel technique employing a temporal convolutional neural network with self-attention (TCN-SA). Our model comprises two primary components: a TCN for extracting time-variant features from EEG signals, followed by a self-attention (SA) layer that assigns importance to these features. By focusing on key features, our model achieves heightened classification accuracy for epilepsy detection.

RESULTS

The efficacy of our model was validated on a pediatric epilepsy dataset we collected and on the Bonn dataset, attaining accuracies of 95.50% on our dataset, and 97.37% (A v. E), and 93.50% (B vs E), respectively. When compared with other deep learning architectures (temporal convolutional neural network, self-attention network, and standardized convolutional neural network) using the same datasets, our TCN-SA model demonstrated superior performance in the automated detection of epilepsy.

CONCLUSION

The proven effectiveness of the TCN-SA approach substantiates its potential as a valuable tool for the automated detection of epilepsy, offering significant benefits in diverse and complex real-world clinical settings.

Self-supervised motor imagery EEG recognition model based on 1-D MTCNN-LSTM network

Objective.Aiming for the research on the brain-computer interface (BCI), it is crucial to design a MI-EEG recognition model, possessing a high classification accuracy and strong generalization ability, and not relying on a large number of labeled training samples.Approach.In this paper, we propose a self-supervised MI-EEG recognition method based on self-supervised learning with one-dimensional multi-task convolutional neural networks and long short-term memory (1-D MTCNN-LSTM). The model is divided into two stages: signal transform identification stage and pattern recognition stage. In the signal transform recognition phase, the signal transform dataset is recognized by the upstream 1-D MTCNN-LSTM network model. Subsequently, the backbone network from the signal transform identification phase is transferred to the pattern recognition phase. Then, it is fine-tuned using a trace amount of labeled data to finally obtain the motion recognition model.Main results.The upstream stage of this study achieves more than 95% recognition accuracy for EEG signal transforms, up to 100%. For MI-EEG pattern recognition, the model obtained recognition accuracies of 82.04% and 87.14% with F1 scores of 0.7856 and 0.839 on the datasets of BCIC-IV-2b and BCIC-IV-2a.Significance.The improved accuracy proves the superiority of the proposed method. It is prospected to be a method for accurate classification of MI-EEG in the BCI system.

Analyzing EEG patterns in young adults exposed to different acrophobia levels: a VR study

Introduction

The primary objective of this research is to examine acrophobia, a widely prevalent and highly severe phobia characterized by an overwhelming dread of heights, which has a substantial impact on a significant proportion of individuals worldwide. The objective of our study was to develop a real-time and precise instrument for evaluating levels of acrophobia by utilizing electroencephalogram (EEG) signals.

Methods

EEG data was gathered from a sample of 18 individuals diagnosed with acrophobia. Subsequently, a range of classifiers, namely Support Vector Classifier (SVC), K-nearest Neighbors (KNN), Random Forest (RF), Decision Tree (DT), Adaboost, Linear Discriminant Analysis (LDA), Convolutional Neural Network (CNN), and Artificial Neural Network (ANN), were employed in the analysis. These methodologies encompass both machine learning (ML) and deep learning (DL) techniques.

Results

The Convolutional Neural Network (CNN) and Artificial Neural Network (ANN) models demonstrated notable efficacy. The Convolutional Neural Network (CNN) model demonstrated a training accuracy of 96% and a testing accuracy of 99%, whereas the Artificial Neural Network (ANN) model attained a training accuracy of 96% and a testing accuracy of 97%. The findings of this study highlight the effectiveness of the proposed methodology in accurately categorizing real-time degrees of acrophobia using EEG data. Further investigation using correlation matrices for each level of acrophobia showed substantial EEG frequency band connections. Beta and Gamma mean values correlated strongly, suggesting cognitive arousal and acrophobic involvement could synchronize activity. Beta and Gamma activity correlated strongly with acrophobia, especially at higher levels.

Discussion

The results underscore the promise of this innovative approach as a dependable and sophisticated method for evaluating acrophobia. This methodology has the potential to make a substantial contribution toward the comprehension and assessment of acrophobia, hence facilitating the development of more individualized and efficacious therapeutic interventions.

Motor Imagery Classification Using Effective Channel Selection of Multichannel EEG

Electroencephalography (EEG) is effectively employed to describe cognitive patterns corresponding to different tasks of motor functions for brain-computer interface (BCI) implementation. Explicit information processing is necessary to reduce the computational complexity of practical BCI systems. This paper presents an entropy-based approach to select effective EEG channels for motor imagery (MI) classification in brain-computer interface (BCI) systems. The method identifies channels with higher entropy scores, which is an indication of greater information content. It discards redundant or noisy channels leading to reduced computational complexity and improved classification accuracy. High entropy means a more disordered pattern, whereas low entropy means a less disordered pattern with less information. The entropy of each channel for individual trials is calculated. The weight of each channel is represented by the mean entropy of the channel over all the trials. A set of channels with higher mean entropy are selected as effective channels for MI classification. A limited number of sub-band signals are created by decomposing the selected channels. To extract the spatial features, the common spatial pattern (CSP) is applied to each sub-band space of EEG signals. The CSP-based features are used to classify the right-hand and right-foot MI tasks using a support vector machine (SVM). The effectiveness of the proposed approach is validated using two publicly available EEG datasets, known as BCI competition III-IV(A) and BCI competition IV-I. The experimental results demonstrate that the proposed approach surpasses cutting-edge techniques.

A causal perspective on brainwave modeling for brain-computer interfaces

Objective. Machine learning (ML) models have opened up enormous opportunities in the field of brain-computer Interfaces (BCIs). Despite their great success, they usually face severe limitations when they are employed in real-life applications outside a controlled laboratory setting.Approach. Mixing causal reasoning, identifying causal relationships between variables of interest, with brainwave modeling can change one's viewpoint on some of these major challenges which can be found in various stages in the ML pipeline, ranging from data collection and data pre-processing to training methods and techniques.Main results. In this work, we employ causal reasoning and present a framework aiming to breakdown and analyze important challenges of brainwave modeling for BCIs.Significance. Furthermore, we present how general ML practices as well as brainwave-specific techniques can be utilized and solve some of these identified challenges. And finally, we discuss appropriate evaluation schemes in order to measure these techniques' performance and efficiently compare them with other methods that will be developed in the future.

Detection of evoked resonant neural activity in Parkinson's disease

Objective. This study investigated a machine-learning approach to detect the presence of evoked resonant neural activity (ERNA) recorded during deep brain stimulation (DBS) of the subthalamic nucleus (STN) in people with Parkinson's disease.Approach. Seven binary classifiers were trained to distinguish ERNA from the background neural activity using eight different time-domain signal features.Main results. Nested cross-validation revealed a strong classification performance of 99.1% accuracy, with 99.6% specificity and 98.7% sensitivity to detect ERNA. Using a semi-simulated ERNA dataset, the results show that a signal-to-noise ratio of 15 dB is required to maintain a 90% classifier sensitivity. ERNA detection is feasible with an appropriate combination of signal processing, feature extraction and classifier. Future work should consider reducing the computational complexity for use in real-time applications.Significance. The presence of ERNA can be used to indicate the location of a DBS electrode array during implantation surgery. The confidence score of the detector could be useful for assisting clinicians to adjust the position of the DBS electrode array inside/outside the STN.

A comprehensive dataset for home appliance control using ERP-based BCIs with the application of inter-subject transfer learning

Brain-computer interfaces (BCIs) have a potential to revolutionize human-computer interaction by enabling direct links between the brain and computer systems. Recent studies are increasingly focusing on practical applications of BCIs-e.g., home appliance control just by thoughts. One of the non-invasive BCIs using electroencephalography (EEG) capitalizes on event-related potentials (ERPs) in response to target stimuli and have shown promise in controlling home appliance. In this paper, we present a comprehensive dataset of online ERP-based BCIs for controlling various home appliances in diverse stimulus presentation environments. We collected online BCI data from a total of 84 subjects among whom 60 subjects controlled three types of appliances (TV: 30, door lock: 15, and electric light: 15) with 4 functions per appliance, 14 subjects controlled a Bluetooth speaker with 6 functions via an LCD monitor, and 10 subjects controlled air conditioner with 4 functions via augmented reality (AR). Using the dataset, we aimed to address the issue of inter-subject variability in ERPs by employing the transfer learning in two different approaches. The first approach, "within-paradigm transfer learning," aimed to generalize the model within the same paradigm of stimulus presentation. The second approach, "cross-paradigm transfer learning," involved extending the model from a 4-class LCD environment to different paradigms. The results demonstrated that transfer learning can effectively enhance the generalizability of BCIs based on ERP across different subjects and environments.

M-FANet: Multi-Feature Attention Convolutional Neural Network for Motor Imagery Decoding

Motor imagery (MI) decoding methods are pivotal in advancing rehabilitation and motor control research. Effective extraction of spectral-spatial-temporal features is crucial for MI decoding from limited and low signal-to-noise ratio electroencephalogram (EEG) signal samples based on brain-computer interface (BCI). In this paper, we propose a lightweight Multi-Feature Attention Neural Network (M-FANet) for feature extraction and selection of multi-feature data. M-FANet employs several unique attention modules to eliminate redundant information in the frequency domain, enhance local spatial feature extraction and calibrate feature maps. We introduce a training method called Regularized Dropout (R-Drop) to address training-inference inconsistency caused by dropout and improve the model's generalization capability. We conduct extensive experiments on the BCI Competition IV 2a (BCIC-IV-2a) dataset and the 2019 World robot conference contest-BCI Robot Contest MI (WBCIC-MI) dataset. M-FANet achieves superior performance compared to state-of-the-art MI decoding methods, with 79.28% 4-class classification accuracy (kappa: 0.7259) on the BCIC-IV-2a dataset and 77.86% 3-class classification accuracy (kappa: 0.6650) on the WBCIC-MI dataset. The application of multi-feature attention modules and R-Drop in our lightweight model significantly enhances its performance, validated through comprehensive ablation experiments and visualizations.

An interpretable model based on graph learning for diagnosis of Parkinson's disease with voice-related EEG

Parkinson's disease (PD) exhibits significant clinical heterogeneity, presenting challenges in the identification of reliable electroencephalogram (EEG) biomarkers. Machine learning techniques have been integrated with resting-state EEG for PD diagnosis, but their practicality is constrained by the interpretable features and the stochastic nature of resting-state EEG. The present study proposes a novel and interpretable deep learning model, graph signal processing-graph convolutional networks (GSP-GCNs), using event-related EEG data obtained from a specific task involving vocal pitch regulation for PD diagnosis. By incorporating both local and global information from single-hop and multi-hop networks, our proposed GSP-GCNs models achieved an averaged classification accuracy of 90.2%, exhibiting a significant improvement of 9.5% over other deep learning models. Moreover, the interpretability analysis revealed discriminative distributions of large-scale EEG networks and topographic map of microstate MS5 learned by our models, primarily located in the left ventral premotor cortex, superior temporal gyrus, and Broca's area that are implicated in PD-related speech disorders, reflecting our GSP-GCN models' ability to provide interpretable insights identifying distinctive EEG biomarkers from large-scale networks. These findings demonstrate the potential of interpretable deep learning models coupled with voice-related EEG signals for distinguishing PD patients from healthy controls with accuracy and elucidating the underlying neurobiological mechanisms.

A New Compound-Limbs Paradigm: Integrating Upper-Limb Swing Improves Lower-Limb Stepping Intention Decoding From EEG

Brain-computer interface (BCI) systems based on spontaneous electroencephalography (EEG) hold the promise to implement human voluntary control of lower-extremity powered exoskeletons. However, current EEG-BCI paradigms do not consider the cooperation of upper and lower limbs during walking, which is inconsistent with natural human stepping patterns. To deal with this problem, this study proposed a stepping-matched human EEG-BCI paradigm that involved actions of both unilateral lower and contralateral upper limbs (also referred to as compound-limbs movement). Experiments were conducted in motor execution (ME) and motor imagery (MI) conditions to validate the feasibility. Common spatial pattern (CSP) proposed subject-specific CSP (SSCSP), and filter-bank CSP (FBCSP) methods were applied for feature extraction, respectively. The best average classification results based on SSCSP indicated that the accuracies of compound-limbs paradigms in ME and MI conditions achieved 89.02% ± 12.84% and 73.70% ± 12.47%, respectively. Although they were 2.03% and 5.68% lower than those of the single-upper-limb mode that does not match human stepping patterns, they were 24.30% and 11.02% higher than those of the single-lower-limb mode. These findings indicated that the proposed compound-limbs EEG-BCI paradigm is feasible for decoding human stepping intention and thus provides a potential way for natural human control of walking assistance devices.

Self-attention-based convolutional neural network and time-frequency common spatial pattern for enhanced motor imagery classification

BACKGROUND

Motor imagery (MI) based brain-computer interfaces (BCIs) have promising potentials in the field of neuro-rehabilitation. However, due to individual variations in active brain regions during MI tasks, the challenge of decoding MI EEG signals necessitates improved classification performance for practical application.

NEW METHOD

This study proposes a self-attention-based Convolutional Neural Network (CNN) in conjunction with a time-frequency common spatial pattern (TFCSP) for enhanced MI classification. Due to the limited availability of training data, a data augmentation strategy is employed to expand the scale of MI EEG datasets. The self-attention-based CNN is trained to automatically extract the temporal and spatial information from EEG signals, allowing the self-attention module to select active channels by calculating EEG channel weights. TFCSP is further implemented to extract multiscale time-frequency-space features from EEG data. Finally, the EEG features derived from TFCSP are concatenated with those from the self-attention-based CNN for MI classification.

RESULTS

The proposed method is evaluated on two publicly accessible datasets, BCI Competition IV IIa and BCI Competition III IIIa, yielding mean accuracies of 79.28 % and 86.39 %, respectively.

CONCLUSIONS

Compared with state-of-the-art methods, our approach achieves superior classification results in accuracy. Self-attention-based CNN combining with TFCSP can make full use of the time-frequency-space information of EEG, and enhance the classification performance.

Image2Brain: a cross-modality model for blind stereoscopic image quality ranking

Objective.Human beings perceive stereoscopic image quality through the cerebral visual cortex, which is a complex brain activity. As a solution, the quality of stereoscopic images can be evaluated more accurately by attempting to replicate the human perception from electroencephalogram (EEG) signals on image quality in a machine, which is different from previous stereoscopic image quality assessment methods focused only on the extraction of image features.Approach.Our proposed method is based on a novel image-to-brain (I2B) cross-modality model including a spatial-temporal EEG encoder (STEE) and an I2B deep convolutional generative adversarial network (I2B-DCGAN). Specifically, the EEG representations are first learned by STEE as real samples of I2B-DCGAN, which is designed to extract both quality and semantic features from the stereoscopic images by a semantic-guided image encoder, and utilize a generator to conditionally create the corresponding EEG features for images. Finally, the generated EEG features are classified to predict the image perceptual quality level.Main results.Extensive experimental results on the collected brain-visual multimodal stereoscopic image quality ranking database, demonstrate that the proposed I2B cross-modality model can better emulate the visual perception mechanism of the human brain and outperform the other methods by achieving an average accuracy of 95.95%.Significance.The proposed method can convert the learned stereoscopic image features into brain representations without EEG signals during testing. Further experiments verify that the proposed method has good generalization ability on new datasets and the potential for practical applications.

Bridging the Gap between Basic Research and Clinical Practice: The Growing Role of Translational Neurorehabilitation

Translational neuroscience is intended as a holistic approach in the field of brain disorders, starting from the basic research of cerebral morphology and with the function of implementing it into clinical practice. This concept can be applied to the rehabilitation field to promote promising results that positively influence the patient's quality of life. The last decades have seen great scientific and technological improvements in the field of neurorehabilitation. In this paper, we discuss the main issues related to translational neurorehabilitation, from basic research to current clinical practice, and we also suggest possible future scenarios.

Deep learning-based framework for real-time upper limb motion intention classification using combined bio-signals

This research study proposes a unique framework that takes input from a surface electromyogram (sEMG) and functional near-infrared spectroscopy (fNIRS) bio-signals. These signals are trained using convolutional neural networks (CNN). The framework entails a real-time neuro-machine interface to decode the human intention of upper limb motions. The bio-signals from the two modalities are recorded for eight movements simultaneously for prosthetic arm functions focusing on trans-humeral amputees. The fNIRS signals are acquired from the human motor cortex, while sEMG is recorded from the human bicep muscles. The selected classification and command generation features are the peak, minimum, and mean ΔHbO and ΔHbR values within a 2-s moving window. In the case of sEMG, wavelength, peak, and mean were extracted with a 150-ms moving window. It was found that this scheme generates eight motions with an enhanced average accuracy of 94.5%. The obtained results validate the adopted research methodology and potential for future real-time neural-machine interfaces to control prosthetic arms.

EEG-based emergency braking intention detection during simulated driving

BACKGROUND

Current research related to electroencephalogram (EEG)-based driver's emergency braking intention detection focuses on recognizing emergency braking from normal driving, with little attention to differentiating emergency braking from normal braking. Moreover, the classification algorithms used are mainly traditional machine learning methods, and the inputs to the algorithms are manually extracted features.

METHODS

To this end, a novel EEG-based driver's emergency braking intention detection strategy is proposed in this paper. The experiment was conducted on a simulated driving platform with three different scenarios: normal driving, normal braking and emergency braking. We compared and analyzed the EEG feature maps of the two braking modes, and explored the use of traditional methods, Riemannian geometry-based methods, and deep learning-based methods to predict the emergency braking intention, all using the raw EEG signals rather than manually extracted features as input.

RESULTS

We recruited 10 subjects for the experiment and used the area under the receiver operating characteristic curve (AUC) and F1 score as evaluation metrics. The results showed that both the Riemannian geometry-based method and the deep learning-based method outperform the traditional method. At 200 ms before the start of real braking, the AUC and F1 score of the deep learning-based EEGNet algorithm were 0.94 and 0.65 for emergency braking vs. normal driving, and 0.91 and 0.85 for emergency braking vs. normal braking, respectively. The EEG feature maps also showed a significant difference between emergency braking and normal braking. Overall, based on EEG signals, it was feasible to detect emergency braking from normal driving and normal braking.

CONCLUSIONS

The study provides a user-centered framework for human-vehicle co-driving. If the driver's intention to brake in an emergency can be accurately identified, the vehicle's automatic braking system can be activated hundreds of milliseconds earlier than the driver's real braking action, potentially avoiding some serious collisions.

Deep-learning-enabled brain hemodynamic mapping using resting-state fMRI

Cerebrovascular disease is a leading cause of death globally. Prevention and early intervention are known to be the most effective forms of its management. Non-invasive imaging methods hold great promises for early stratification, but at present lack the sensitivity for personalized prognosis. Resting-state functional magnetic resonance imaging (rs-fMRI), a powerful tool previously used for mapping neural activity, is available in most hospitals. Here we show that rs-fMRI can be used to map cerebral hemodynamic function and delineate impairment. By exploiting time variations in breathing pattern during rs-fMRI, deep learning enables reproducible mapping of cerebrovascular reactivity (CVR) and bolus arrival time (BAT) of the human brain using resting-state CO2 fluctuations as a natural "contrast media". The deep-learning network is trained with CVR and BAT maps obtained with a reference method of CO2-inhalation MRI, which includes data from young and older healthy subjects and patients with Moyamoya disease and brain tumors. We demonstrate the performance of deep-learning cerebrovascular mapping in the detection of vascular abnormalities, evaluation of revascularization effects, and vascular alterations in normal aging. In addition, cerebrovascular maps obtained with the proposed method exhibit excellent reproducibility in both healthy volunteers and stroke patients. Deep-learning resting-state vascular imaging has the potential to become a useful tool in clinical cerebrovascular imaging.

Machine Learning on Visibility Graph Features Discriminates the Cognitive Event-Related Potentials of Patients with Early Alzheimer's Disease from Healthy Aging

We present a framework for electroencephalography (EEG)-based classification between patients with Alzheimer's Disease (AD) and robust normal elderly (RNE) via a graph theory approach using visibility graphs (VGs). This EEG VG approach is motivated by research that has demonstrated differences between patients with early stage AD and RNE using various features of EEG oscillations or cognitive event-related potentials (ERPs). In the present study, EEG signals recorded during a word repetition experiment were wavelet decomposed into 5 sub-bands (δ,θ,α,β,γ). The raw and band-specific signals were then converted to VGs for analysis. Twelve graph features were tested for differences between the AD and RNE groups, and t-tests employed for feature selection. The selected features were then tested for classification using traditional machine learning and deep learning algorithms, achieving a classification accuracy of 100% with linear and non-linear classifiers. We further demonstrated that the same features can be generalized to the classification of mild cognitive impairment (MCI) converters, i.e., prodromal AD, against RNE with a maximum accuracy of 92.5%. Code is released online to allow others to test and reuse this framework.

A transformer-based deep neural network model for SSVEP classification

Steady-state visual evoked potential (SSVEP) is one of the most commonly used control signals in the brain-computer interface (BCI) systems. However, the conventional spatial filtering methods for SSVEP classification highly depend on the subject-specific calibration data. The need for the methods that can alleviate the demand for the calibration data becomes urgent. In recent years, developing the methods that can work in inter-subject scenario has become a promising new direction. As a popular deep learning model nowadays, Transformer has been used in EEG signal classification tasks owing to its excellent performance. Therefore, in this study, we proposed a deep learning model for SSVEP classification based on Transformer architecture in inter-subject scenario, termed as SSVEPformer, which was the first application of Transformer on the SSVEP classification. Inspired by previous studies, we adopted the complex spectrum features of SSVEP data as the model input, which could enable the model to simultaneously explore the spectral and spatial information for classification. Furthermore, to fully utilize the harmonic information, an extended SSVEPformer based on the filter bank technology (FB-SSVEPformer) was proposed to improve the classification performance. Experiments were conducted using two open datasets (Dataset 1: 10 subjects, 12 targets; Dataset 2: 35 subjects, 40 targets). The experimental results show that the proposed models could achieve better results in terms of classification accuracy and information transfer rate than other baseline methods. The proposed models validate the feasibility of deep learning models based on Transformer architecture for SSVEP data classification, and could serve as potential models to alleviate the calibration procedure in the practical application of SSVEP-based BCI systems.

Electroencephalogram based brain-computer interface: Applications, challenges, and opportunities

Brain-Computer Interfaces (BCI) is an exciting and emerging research area for researchers and scientists. It is a suitable combination of software and hardware to operate any device mentally. This review emphasizes the significant stages in the BCI domain, current problems, and state-of-the-art findings. This article also covers how current results can contribute to new knowledge about BCI, an overview of BCI from its early developments to recent advancements, BCI applications, challenges, and future directions. The authors pointed to unresolved issues and expressed how BCI is valuable for analyzing the human brain. Humans' dependence on machines has led humankind into a new future where BCI can play an essential role in improving this modern world.

EEGformer: A transformer-based brain activity classification method using EEG signal

Background

The effective analysis methods for steady-state visual evoked potential (SSVEP) signals are critical in supporting an early diagnosis of glaucoma. Most efforts focused on adopting existing techniques to the SSVEPs-based brain-computer interface (BCI) task rather than proposing new ones specifically suited to the domain.

Method

Given that electroencephalogram (EEG) signals possess temporal, regional, and synchronous characteristics of brain activity, we proposed a transformer-based EEG analysis model known as EEGformer to capture the EEG characteristics in a unified manner. We adopted a one-dimensional convolution neural network (1DCNN) to automatically extract EEG-channel-wise features. The output was fed into the EEGformer, which is sequentially constructed using three components: regional, synchronous, and temporal transformers. In addition to using a large benchmark database (BETA) toward SSVEP-BCI application to validate model performance, we compared the EEGformer to current state-of-the-art deep learning models using two EEG datasets, which are obtained from our previous study: SJTU emotion EEG dataset (SEED) and a depressive EEG database (DepEEG).

Results

The experimental results show that the EEGformer achieves the best classification performance across the three EEG datasets, indicating that the rationality of our model architecture and learning EEG characteristics in a unified manner can improve model classification performance.

Conclusion

EEGformer generalizes well to different EEG datasets, demonstrating our approach can be potentially suitable for providing accurate brain activity classification and being used in different application scenarios, such as SSVEP-based early glaucoma diagnosis, emotion recognition and depression discrimination.

An Analysis of Deep Learning Models in SSVEP-Based BCI: A Survey

The brain–computer interface (BCI), which provides a new way for humans to directly communicate with robots without the involvement of the peripheral nervous system, has recently attracted much attention. Among all the BCI paradigms, BCIs based on steady-state visual evoked potentials (SSVEPs) have the highest information transfer rate (ITR) and the shortest training time. Meanwhile, deep learning has provided an effective and feasible solution for solving complex classification problems in many fields, and many researchers have started to apply deep learning to classify SSVEP signals. However, the designs of deep learning models vary drastically. There are many hyper-parameters that influence the performance of the model in an unpredictable way. This study surveyed 31 deep learning models (2011–2023) that were used to classify SSVEP signals and analyzed their design aspects including model input, model structure, performance measure, etc. Most of the studies that were surveyed in this paper were published in 2021 and 2022. This survey is an up-to-date design guide for researchers who are interested in using deep learning models to classify SSVEP signals.

A multiwavelet-based sparse time-varying autoregressive modeling for motor imagery EEG classification

Brain-computer Interface (BCI) system based on motor imagery (MI) heavily relies on electroencephalography (EEG) recognition with high accuracy. However, modeling and classification of MI EEG signals remains a challenging task due to the non-linear and non-stationary characteristics of the signals. In this paper, a new time-varying modeling framework combining multiwavelet basis functions and regularized orthogonal forward regression (ROFR) algorithm is proposed for the characterization and classification of MI EEG signals. Firstly, the time-varying coefficients of the time-varying autoregressive (TVAR) model are precisely approximated with the multiwavelet basis functions. Then a powerful ROFR algorithm is employed to dramatically alleviate the redundant model structure and accurately recover the relevant time-varying model parameters to obtain high resolution power spectral density (PSD) features. Finally, the features are sent to different classifiers for the classification task. To effectively improve the accuracy of classification, a principal component analysis (PCA) algorithm is utilized to determine the best feature subset and Bayesian optimization algorithm is performed to obtain the optimal parameters of the classifier. The proposed method achieves satisfactory classification accuracy on the public BCI Competition II Dataset III, which proves that this method potentially improves the recognition accuracy of MI EEG signals, and has great significance for the construction of BCI system based on MI.

Convolutional Neural Network with a Topographic Representation Module for EEG-Based Brain-Computer Interfaces

Convolutional neural networks (CNNs) have shown great potential in the field of brain-computer interfaces (BCIs) due to their ability to directly process raw electroencephalogram (EEG) signals without artificial feature extraction. Some CNNs have achieved better classification accuracy than that of traditional methods. Raw EEG signals are usually represented as a two-dimensional (2-D) matrix composed of channels and time points, ignoring the spatial topological information of electrodes. Our goal is to make a CNN that takes raw EEG signals as inputs have the ability to learn spatial topological features and improve its classification performance while basically maintaining its original structure. We propose an EEG topographic representation module (TRM). This module consists of (1) a mapping block from raw EEG signals to a 3-D topographic map and (2) a convolution block from the topographic map to an output with the same size as the input. According to the size of the convolutional kernel used in the convolution block, we design two types of TRMs, namely TRM-(5,5) and TRM-(3,3). We embed the two TRM types into three widely used CNNs (ShallowConvNet, DeepConvNet and EEGNet) and test them on two publicly available datasets (the Emergency Braking During Simulated Driving Dataset (EBDSDD) and the High Gamma Dataset (HGD)). Results show that the classification accuracies of all three CNNs are improved on both datasets after using the TRMs. With TRM-(5,5), the average classification accuracies of DeepConvNet, EEGNet and ShallowConvNet are improved by 6.54%, 1.72% and 2.07% on the EBDSDD and by 6.05%, 3.02% and 5.14% on the HGD, respectively; with TRM-(3,3), they are improved by 7.76%, 1.71% and 2.17% on the EBDSDD and by 7.61%, 5.06% and 6.28% on the HGD, respectively. We improve the classification performance of three CNNs on both datasets through the use of TRMs, indicating that they have the capability to mine spatial topological EEG information. More importantly, since the output of a TRM has the same size as the input, CNNs with raw EEG signals as inputs can use this module without changing their original structures.

Early-stage fusion of EEG and fNIRS improves classification of motor imagery

Introduction

Many research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear.

Methods

In this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bimodal information in different stages.

Results

The results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N = 57, P < 0.05). With the proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-out cross-validation, using bi-modal data as network inputs respectively, which is in the same level as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data.

Siam Deep Feature KCF Method and Experimental Study for Pedestrian Tracking

The tracking of a particular pedestrian is an important issue in computer vision to guarantee societal safety. Due to the limited computing performances of unmanned aerial vehicle (UAV) systems, the Correlation Filter (CF) algorithm has been widely used to perform the task of tracking. However, it has a fixed template size and cannot effectively solve the occlusion problem. Thus, a tracking-by-detection framework was designed in the current research. A lightweight YOLOv3-based (You Only Look Once version 3) mode which had Efficient Channel Attention (ECA) was integrated into the CF algorithm to provide deep features. In addition, a lightweight Siamese CNN with Cross Stage Partial (CSP) provided the representations of features learned from massive face images, allowing the target similarity in data association to be guaranteed. As a result, a Deep Feature Kernelized Correlation Filters method coupled with Siamese-CSP(Siam-DFKCF) was established to increase the tracking robustness. From the experimental results, it can be concluded that the anti-occlusion and re-tracking performance of the proposed method was increased. The tracking accuracy Distance Precision (DP) and Overlap Precision (OP) had been increased to 0.934 and 0.909 respectively in our test data.

A Systematic Review of Using Deep Learning Technology in the Steady-State Visually Evoked Potential-Based Brain-Computer Interface Applications: Current Trends and Future Trust Methodology

The significance of deep learning techniques in relation to steady-state visually evoked potential- (SSVEP-) based brain-computer interface (BCI) applications is assessed through a systematic review. Three reliable databases, PubMed, ScienceDirect, and IEEE, were considered to gather relevant scientific and theoretical articles. Initially, 125 papers were found between 2010 and 2021 related to this integrated research field. After the filtering process, only 30 articles were identified and classified into five categories based on their type of deep learning methods. The first category, convolutional neural network (CNN), accounts for 70% (n = 21/30). The second category, recurrent neural network (RNN), accounts for 10% (n = 3/30). The third and fourth categories, deep neural network (DNN) and long short-term memory (LSTM), account for 6% (n = 30). The fifth category, restricted Boltzmann machine (RBM), accounts for 3% (n = 1/30). The literature's findings in terms of the main aspects identified in existing applications of deep learning pattern recognition techniques in SSVEP-based BCI, such as feature extraction, classification, activation functions, validation methods, and achieved classification accuracies, are examined. A comprehensive mapping analysis was also conducted, which identified six categories. Current challenges of ensuring trustworthy deep learning in SSVEP-based BCI applications were discussed, and recommendations were provided to researchers and developers. The study critically reviews the current unsolved issues of SSVEP-based BCI applications in terms of development challenges based on deep learning techniques and selection challenges based on multicriteria decision-making (MCDM). A trust proposal solution is presented with three methodology phases for evaluating and benchmarking SSVEP-based BCI applications using fuzzy decision-making techniques. Valuable insights and recommendations for researchers and developers in the SSVEP-based BCI and deep learning are provided.

Improving pre-movement pattern detection with filter bank selection

Objective. Pre-movement decoding plays an important role in detecting the onsets of actions using low-frequency electroencephalography (EEG) signals before the movement of an upper limb. In this work, a binary classification method is proposed between two different states.Approach. The proposed method, referred to as filter bank standard task-related component analysis (FBTRCA), is to incorporate filter bank selection into the standard task-related component analysis (STRCA) method. In FBTRCA, the EEG signals are first divided into multiple sub-bands which start at specific fixed frequencies and end frequencies that follow in an arithmetic sequence. The STRCA method is then applied to the EEG signals in these bands to extract CCPs. The minimum redundancy maximum relevance feature selection method is used to select essential features from these correlation patterns in all sub-bands. Finally, the selected features are classified using the binary support vector machine classifier. A convolutional neural network (CNN) is an alternative approach to select canonical correlation patterns.Main Results. Three methods were evaluated using EEG signals in the time window from 2 s before the movement onset to 1 s after the movement onset. In the binary classification between a movement state and the resting state, the FBTRCA achieved an average accuracy of 0.8968 ± 0.0847 while the accuracies of STRCA and CNN were 0.8228 ± 0.1149 and 0.8828 ± 0.0917, respectively. In the binary classification between two actions, the accuracies of STRCA, CNN, and FBTRCA were 0.6611 ± 0.1432, 0.6993 ± 0.1271, 0.7178 ± 0.1274, respectively. Feature selection using filter banks, as in FBTRCA, produces comparable results to STRCA.Significance. The proposed method provides a way to select filter banks in pre-movement decoding, and thus it improves the classification performance. The improved pre-movement decoding of single upper limb movements is expected to provide people with severe motor disabilities with a more natural, non-invasive control of their external devices.

Good scientific practice in EEG and MEG research: Progress and perspectives

Good scientific practice (GSP) refers to both explicit and implicit rules, recommendations, and guidelines that help scientists to produce work that is of the highest quality at any given time, and to efficiently share that work with the community for further scrutiny or utilization. For experimental research using magneto- and electroencephalography (MEEG), GSP includes specific standards and guidelines for technical competence, which are periodically updated and adapted to new findings. However, GSP also needs to be regularly revisited in a broader light. At the LiveMEEG 2020 conference, a reflection on GSP was fostered that included explicitly documented guidelines and technical advances, but also emphasized intangible GSP: a general awareness of personal, organizational, and societal realities and how they can influence MEEG research. This article provides an extensive report on most of the LiveMEEG contributions and new literature, with the additional aim to synthesize ongoing cultural changes in GSP. It first covers GSP with respect to cognitive biases and logical fallacies, pre-registration as a tool to avoid those and other early pitfalls, and a number of resources to enable collaborative and reproducible research as a general approach to minimize misconceptions. Second, it covers GSP with respect to data acquisition, analysis, reporting, and sharing, including new tools and frameworks to support collaborative work. Finally, GSP is considered in light of ethical implications of MEEG research and the resulting responsibility that scientists have to engage with societal challenges. Considering among other things the benefits of peer review and open access at all stages, the need to coordinate larger international projects, the complexity of MEEG subject matter, and today's prioritization of fairness, privacy, and the environment, we find that current GSP tends to favor collective and cooperative work, for both scientific and for societal reasons.

Explanations of Machine Learning Models in Repeated Nested Cross-Validation: An Application in Age Prediction Using Brain Complexity Features

SHAP (Shapley additive explanations) is a framework for explainable AI that makes explanations locally and globally. In this work, we propose a general method to obtain representative SHAP values within a repeated nested cross-validation procedure and separately for the training and test sets of the different cross-validation rounds to assess the real generalization abilities of the explanations. We applied this method to predict individual age using brain complexity features extracted from MRI scans of 159 healthy subjects. In particular, we used four implementations of the fractal dimension (FD) of the cerebral cortex—a measurement of brain complexity. Representative SHAP values highlighted that the most recent implementation of the FD had the highest impact over the others and was among the top-ranking features for predicting age. SHAP rankings were not the same in the training and test sets, but the top-ranking features were consistent. In conclusion, we propose a method—and share all the source code—that allows a rigorous assessment of the SHAP explanations of a trained model in a repeated nested cross-validation setting.

A Bayesian-optimized design for an interpretable convolutional neural network to decode and analyze the P300 response in autism

OBJECTIVE

P300 can be analyzed in autism spectrum disorder (ASD) to derive biomarkers and can be decoded in BCIs to reinforce ASD impaired skills. Convolutional neural networks (CNNs) have been proposed for P300 decoding, outperforming traditional algorithms but they i) do not investigate optimal designs in different training conditions; ii) lack in interpretability. To overcome these limitations, an interpretable CNN (ICNN), that we recently proposed for motor decoding, has been modified and adopted here, with its optimal design searched via Bayesian optimization.

APPROACH

The ICNN provides a straightforward interpretation of spectral and spatial features learned to decode P300. The Bayesian-optimized (BO) ICNN design was investigated separately for different training strategies (within-subject, within-session, and cross-subject) and BO models were used for the subsequent analyses. Specifically, transfer learning (TL) potentialities were investigated by assessing how pretrained cross-subject BO models performed on a new subject vs. random-initialized models. Furthermore, within-subject BO-derived models were combined with an Explanation Technique (ICNN+ET) to analyze P300 spectral and spatial features.

MAIN RESULTS

The ICNN resulted comparable or even outperformed existing CNNs, at the same time being lighter. Bayesian-optimized ICNN designs differed depending on the training strategy, needing more capacity as the training set variability increased. Furthermore, TL provided higher performance than networks trained from scratch. The ICNN+ET analysis suggested the frequency range [2, 5.8] Hz as the most relevant, and spatial features showed a right-hemispheric parietal asymmetry. The ICNN+ET-derived features, but not ERP-derived features, resulted significantly and highly correlated to ADOS clinical scores.

SIGNIFICANCE

This study substantiates the idea that a CNN can be designed both accurate and interpretable for P300 decoding, with an optimized design depending on the training condition. The novel ICNN-based analysis tool was able to better capture ASD neural signatures than traditional ERP analysis, possibly paving the way for identifying novel biomarkers.

IENet: a robust convolutional neural network for EEG based brain-computer interfaces

OBJECTIVE

Brain-computer interfaces (BCIs) based on electroencephalogram (EEG) develop into novel application areas with more complex scenarios, which put forward higher requirements for the robustness of EEG signal processing algorithms. Deep learning can automatically extract discriminative features and potential dependencies via deep structures, demonstrating strong analytical capabilities in numerous domains such as computer vision (CV) and natural language processing (NLP). Making full use of deep learning technology to design a robust algorithm that is capable of analyzing EEG across BCI paradigms is our main work in this paper.

APPROACH

Inspired by InceptionV4 and InceptionTime architecture, we introduce a neural network ensemble named InceptionEEG-Net (IENet), where multi-scale convolutional layer and convolution of length 1 enable model to extract rich high-dimensional features with limited parameters. In addition, we propose the average receptive field gain for convolutional neural networks (CNNs), which optimizes IENet to detect long patterns at a smaller cost. We compare with the current state-of-the-art method across five EEG-BCI paradigms: steady-state visual evoked potentials, epilepsy EEG, overt attention P300 visual-evoked potentials, covert attention P300 visual-evoked potentials and movement-related cortical potentials.

MAIN RESULTS

The classification results show that the generalizability of IENet is on par with the state-of-the-art paradigm-agnostic models on test datasets. Furthermore, the feature explainability analysis of IENet illustrates its capability to extract neurophysiologically interpretable features for different BCI paradigms, ensuring the reliability of algorithm. Significance. It can be seen from our results that IENet can generalize to different BCI paradigms. And it is essential for deep CNNs to increase the receptive field size using average receptive field gain.

A CNN-Based Deep Learning Approach for SSVEP Detection Targeting Binaural Ear-EEG

This paper discusses a machine learning approach for detecting SSVEP at both ears with minimal channels. SSVEP is a robust EEG signal suitable for many BCI applications. It is strong at the visual cortex around the occipital area, but the SNR gets worse when detected from other areas of the head. To make use of SSVEP measured around the ears following the ear-EEG concept, especially for practical binaural implementation, we propose a CNN structure coupled with regressed softmax outputs to improve accuracy. Evaluating on a public dataset, we studied classification performance for both subject-dependent and subject-independent trainings. It was found that with the proposed structure using a group training approach, a 69.21% accuracy was achievable. An ITR of 6.42 bit/min given 63.49 % accuracy was recorded while only monitoring data from T7 and T8. This represents a 12.47% improvement from a single ear implementation and illustrates potential of the approach to enhance performance for practical implementation of wearable EEG.

Past, Present, and Future of EEG-Based BCI Applications

An electroencephalography (EEG)-based brain-computer interface (BCI) is a system that provides a pathway between the brain and external devices by interpreting EEG. EEG-based BCI applications have initially been developed for medical purposes, with the aim of facilitating the return of patients to normal life. In addition to the initial aim, EEG-based BCI applications have also gained increasing significance in the non-medical domain, improving the life of healthy people, for instance, by making it more efficient, collaborative and helping develop themselves. The objective of this review is to give a systematic overview of the literature on EEG-based BCI applications from the period of 2009 until 2019. The systematic literature review has been prepared based on three databases PubMed, Web of Science and Scopus. This review was conducted following the PRISMA model. In this review, 202 publications were selected based on specific eligibility criteria. The distribution of the research between the medical and non-medical domain has been analyzed and further categorized into fields of research within the reviewed domains. In this review, the equipment used for gathering EEG data and signal processing methods have also been reviewed. Additionally, current challenges in the field and possibilities for the future have been analyzed.

Multi-Hierarchical Fusion to Capture the Latent Invariance for Calibration-Free Brain-Computer Interfaces

Brain-computer interfaces (BCI) based motor imagery (MI) has become a research hotspot for establishing a flexible communication channel for patients with apoplexy or degenerative pathologies. Accurate decoding of motor imagery electroencephalography (MI-EEG) signals, while essential for effective BCI systems, is still challenging due to the significant noise inherent in the EEG signals and the lack of informative correlation between the signals and brain activities. The application of deep learning for EEG feature representation has been rarely investigated, nevertheless bringing improvements to the performance of motor imagery classification. This paper proposes a deep learning decoding method based on multi-hierarchical representation fusion (MHRF) on MI-EEG. It consists of a concurrent framework constructed of bidirectional LSTM (Bi-LSTM) and convolutional neural network (CNN) to fully capture the contextual correlations of MI-EEG and the spectral feature. Also, the stacked sparse autoencoder (SSAE) is employed to concentrate these two domain features into a high-level representation for cross-session and subject training guidance. The experimental analysis demonstrated the efficacy and practicality of the proposed approach using a public dataset from BCI competition IV and a private one collected by our MI task. The proposed approach can serve as a robust and competitive method to improve inter-session and inter-subject transferability, adding anticipation and prospective thoughts to the practical implementation of a calibration-free BCI system.

AJILE12: Long-term naturalistic human intracranial neural recordings and pose

Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. AJILE12 includes synchronized intracranial neural recordings and upper body pose trajectories across 55 semi-continuous days of naturalistic movements, along with relevant metadata, including thousands of wrist movement events and annotated behavioral states. Neural recordings are available at 500 Hz from at least 64 electrodes per participant, for a total of 1280 hours. Pose trajectories at 9 upper-body keypoints were estimated from 118 million video frames. To facilitate data exploration and reuse, we have shared AJILE12 on The DANDI Archive in the Neurodata Without Borders (NWB) data standard and developed a browser-based dashboard.

Automatic Muscle Artifacts Identification and Removal from Single-Channel EEG Using Wavelet Transform with Meta-Heuristically Optimized Non-Local Means Filter

Electroencephalogram (EEG) signals may get easily contaminated by muscle artifacts, which may lead to wrong interpretation in the brain–computer interface (BCI) system as well as in various medical diagnoses. The main objective of this paper is to remove muscle artifacts without distorting the information contained in the EEG. A novel multi-stage EEG denoising method is proposed for the first time in which wavelet packet decomposition (WPD) is combined with a modified non-local means (NLM) algorithm. At first, the artifact EEG signal is identified through a pre-trained classifier. Next, the identified EEG signal is decomposed into wavelet coefficients and corrected through a modified NLM filter. Finally, the artifact-free EEG is reconstructed from corrected wavelet coefficients through inverse WPD. To optimize the filter parameters, two meta-heuristic algorithms are used in this paper for the first time. The proposed system is first validated on simulated EEG data and then tested on real EEG data. The proposed approach achieved average mutual information (MI) as 2.9684 ± 0.7045 on real EEG data. The result reveals that the proposed system outperforms recently developed denoising techniques with higher average MI, which indicates that the proposed approach is better in terms of quality of reconstruction and is fully automatic.

Decoding Bilateral Hindlimb Kinematics From Cat Spinal Signals Using Three-Dimensional Convolutional Neural Network

To date, decoding limb kinematic information mostly relies on neural signals recorded from the peripheral nerve, dorsal root ganglia (DRG), ventral roots, spinal cord gray matter, and the sensorimotor cortex. In the current study, we demonstrated that the neural signals recorded from the lateral and dorsal columns within the spinal cord have the potential to decode hindlimb kinematics during locomotion. Experiments were conducted using intact cats. The cats were trained to walk on a moving belt in a hindlimb-only condition, while their forelimbs were kept on the front body of the treadmill. The bilateral hindlimb joint angles were decoded using local field potential signals recorded using a microelectrode array implanted in the dorsal and lateral columns of both the left and right sides of the cat spinal cord. The results show that contralateral hindlimb kinematics can be decoded as accurately as ipsilateral kinematics. Interestingly, hindlimb kinematics of both legs can be accurately decoded from the lateral columns within one side of the spinal cord during hindlimb-only locomotion. The results indicated that there was no significant difference between the decoding performances obtained using neural signals recorded from the dorsal and lateral columns. The results of the time-frequency analysis show that event-related synchronization (ERS) and event-related desynchronization (ERD) patterns in all frequency bands could reveal the dynamics of the neural signals during movement. The onset and offset of the movement can be clearly identified by the ERD/ERS patterns. The results of the mutual information (MI) analysis showed that the theta frequency band contained significantly more limb kinematics information than the other frequency bands. Moreover, the theta power increased with a higher locomotion speed.

Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain-computer interface

The brain-computer interface (BCI) provides an alternate means of communication between the brain and external devices by recognizing the brain activities and translating them into external commands. The functional Near-Infrared Spectroscopy (fNIRS) is becoming popular as a non-invasive modality for brain activity detection. The recent trends show that deep learning has significantly enhanced the performance of the BCI systems. But the inherent bottleneck for deep learning (in the domain of BCI) is the requirement of the vast amount of training data, lengthy recalibrating time, and expensive computational resources for training deep networks. Building a high-quality, large-scale annotated dataset for deep learning-based BCI systems is exceptionally tedious, complex, and expensive. This study investigates the novel application of transfer learning for fNIRS-based BCI to solve three objective functions (concerns), i.e., the problem of insufficient training data, reduced training time, and increased accuracy. We applied symmetric homogeneous feature-based transfer learning on convolutional neural network (CNN) designed explicitly for fNIRS data collected from twenty-six (26) participants performing the n-back task. The results suggested that the proposed method achieves the maximum saturated accuracy sooner and outperformed the traditional CNN model on averaged accuracy by 25.58% in the exact duration of training time, reducing the training time, recalibrating time, and computational resources.

A Long Short-Term Memory Biomarker-Based Prediction Framework for Alzheimer's Disease

The early prediction of Alzheimer's disease (AD) can be vital for the endurance of patients and establishes as an accommodating and facilitative factor for specialists. The proposed work presents a robotized predictive structure, dependent on machine learning (ML) methods for the forecast of AD. Neuropsychological measures (NM) and magnetic resonance imaging (MRI) biomarkers are deduced and passed on to a recurrent neural network (RNN). In the RNN, we have used long short-term memory (LSTM), and the proposed model will predict the biomarkers (feature vectors) of patients after 6, 12, 21 18, 24, and 36 months. These predicted biomarkers will go through fully connected neural network layers. The NN layers will then predict whether these RNN-predicted biomarkers belong to an AD patient or a patient with a mild cognitive impairment (MCI). The developed methodology has been tried on an openly available informational dataset (ADNI) and accomplished an accuracy of 88.24%, which is superior to the next-best available algorithms.

Image-Based Learning Using Gradient Class Activation Maps for Enhanced Physiological Interpretability of Motor Imagery Skills

Brain activity stimulated by the motor imagery paradigm (MI) is measured by Electroencephalography (EEG), which has several advantages to be implemented with the widely used Brain–Computer Interfaces (BCIs) technology. However, the substantial inter/intra variability of recorded data significantly influences individual skills on the achieved performance. This study explores the ability to distinguish between MI tasks and the interpretability of the brain’s ability to produce elicited mental responses with improved accuracy. We develop a Deep and Wide Convolutional Neuronal Network fed by a set of topoplots extracted from the multichannel EEG data. Further, we perform a visualization technique based on gradient-based class activation maps (namely, GradCam++) at different intervals along the MI paradigm timeline to account for intra-subject variability in neural responses over time. We also cluster the dynamic spatial representation of the extracted maps across the subject set to come to a deeper understanding of MI-BCI coordination skills. According to the results obtained from the evaluated GigaScience Database of motor-evoked potentials, the developed approach enhances the physiological explanation of motor imagery in aspects such as neural synchronization between rhythms, brain lateralization, and the ability to predict the MI onset responses and their evolution during training sessions.

Deep learning for biosignal control: insights from basic to real-time methods with recommendations

Biosignal control is an interaction modality that allows users to interact with electronic devices by decoding the biological signals emanating from the movements or thoughts of the user. This manner of interaction with devices can enhance the sense of agency for users and enable persons suffering from a paralyzing condition to interact with everyday devices that would otherwise be challenging for them to use. It can also improve control of prosthetic devices and exoskeletons by making the interaction feel more natural and intuitive. However, with the current state of the art, several issues still need to be addressed to reliably decode user intent from biosignals and provide an improved user experience over other interaction modalities. One solution is to leverage advances in Deep Learning (DL) methods to provide more reliable decoding at the expense of added computational complexity. This scoping review introduces the basic concepts of DL and assists readers in deploying DL methods to a real-time control system that should operate under real-world conditions. The scope of this review covers any electronic device, but with an emphasis on robotic devices, as this is the most active area of research in biosignal control. We review the literature pertaining to the implementation and evaluation of control systems that incorporate DL to identify the main gaps and issues in the field, and formulate suggestions on how to mitigate them. Additionally, we formulate guidelines on the best approach to designing, implementing and evaluating research prototypes that use DL in their biosignal control systems.

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

With the advent of brain imaging techniques and machine learning tools, much effort has been devoted to building computational models to capture the encoding of visual information in the human brain. One of the most challenging brain decoding tasks is the accurate reconstruction of the perceived natural images from brain activities measured by functional magnetic resonance imaging (fMRI). In this work, we survey the most recent deep learning methods for natural image reconstruction from fMRI. We examine these methods in terms of architectural design, benchmark datasets, and evaluation metrics and present a fair performance evaluation across standardized evaluation metrics. Finally, we discuss the strengths and limitations of existing studies and present potential future directions.

Abnormal Activity Recognition from Surveillance Videos Using Convolutional Neural Network

Background and motivation: Every year, millions of Muslims worldwide come to Mecca to perform the Hajj. In order to maintain the security of the pilgrims, the Saudi government has installed about 5000 closed circuit television (CCTV) cameras to monitor crowd activity efficiently.

PROBLEM

As a result, these cameras generate an enormous amount of visual data through manual or offline monitoring, requiring numerous human resources for efficient tracking. Therefore, there is an urgent need to develop an intelligent and automatic system in order to efficiently monitor crowds and identify abnormal activity.

METHOD

The existing method is incapable of extracting discriminative features from surveillance videos as pre-trained weights of different architectures were used. This paper develops a lightweight approach for accurately identifying violent activity in surveillance environments. As the first step of the proposed framework, a lightweight CNN model is trained on our own pilgrim's dataset to detect pilgrims from the surveillance cameras. These preprocessed salient frames are passed to a lightweight CNN model for spatial features extraction in the second step. In the third step, a Long Short Term Memory network (LSTM) is developed to extract temporal features. Finally, in the last step, in the case of violent activity or accidents, the proposed system will generate an alarm in real time to inform law enforcement agencies to take appropriate action, thus helping to avoid accidents and stampedes.

RESULTS

We have conducted multiple experiments on two publicly available violent activity datasets, such as Surveillance Fight and Hockey Fight datasets; our proposed model achieved accuracies of 81.05 and 98.00, respectively.

Identification of Food/Nonfood Visual Stimuli from Event-Related Brain Potentials

Although food consumption is one of the most basic human behaviors, the factors underlying nutritional preferences are not yet clear. The use of classification algorithms can clarify the understanding of these factors. This study was aimed at measuring electrophysiological responses to food/nonfood stimuli and applying classification techniques to discriminate the responses using a single-sweep dataset. Twenty-one right-handed male athletes with body mass index (BMI) levels between 18.5% and 25% (mean age: 21.05 ± 2.5) participated in this study voluntarily. The participants were asked to focus on the food and nonfood images that were randomly presented on the monitor without performing any motor task, and EEG data have been collected using a 16-channel amplifier with a sampling rate of 1024 Hz. The SensoMotoric Instruments (SMI) iView XTM RED eye tracking technology was used simultaneously with the EEG to measure the participants' attention to the presented stimuli. Three datasets were generated using the amplitude, time-frequency decomposition, and time-frequency connectivity metrics of P300 and LPP components to separate food and nonfood stimuli. We have implemented k-nearest neighbor (kNN), support vector machine (SVM), Linear Discriminant Analysis (LDA), Logistic Regression (LR), Bayesian classifier, decision tree (DT), and Multilayer Perceptron (MLP) classifiers on these datasets. Finally, the response to food-related stimuli in the hunger state is discriminated from nonfood with an accuracy value close to 78% for each dataset. The results obtained in this study motivate us to employ classifier algorithms using the features obtained from single-trial measurements in amplitude and time-frequency space instead of applying more complex ones like connectivity metrics.

Design a novel BCI for neurorehabilitation using concurrent LFP and EEG features: a case study

Brain-computer interfaces (BCI) that enables people with severe motor disabilities to use their brain signals for direct control of objects have attracted increased interest in rehabilitation. To date, no study has investigated feasibility of the BCI framework incorporating both intracortical and scalp signals. Methods: Concurrent local field potential (LFP) from the hand-knob area and scalp EEG were recorded in a paraplegic patient undergoing a spike-based close-loop neurorehabilitation training. Based upon multimodal spatio-spectral feature extraction and Naive Bayes classification, we developed, for the first time, a novel LFP-EEG-BCI for motor intention decoding. A transfer learning (TL) approach was employed to further improve the feasibility. The performance of the proposed LFP-EEG-BCI for four-class upper-limb motor intention decoding was assessed. Results: Using a decision fusion strategy, we showed that the LFP-EEG-BCI significantly (p <0.05) outperformed single modal BCI (LFP-BCI and EEG-BCI) in terms of decoding accuracy with the best performance achieved using regularized common spatial pattern features. Interrogation of feature characteristics revealed discriminative spatial and spectral patterns, which may lead to new insights for better understanding of brain dynamics during different motor imagery tasks and promote development of efficient decoding algorithms. Moreover, we showed that similar classification performance could be obtained with few training trials, therefore highlighting the efficacy of TL. Conclusion: The present findings demonstrated the superiority of the novel LFP-EEG-BCI in motor intention decoding. Significance: This work introduced a novel LFP-EEG-BCI that may lead to new directions for developing practical neurorehabilitation systems with high detection accuracy and multi-paradigm feasibility in clinical applications.