Reduce brain computer interface inefficiency by combining sensory motor rhythm and movement-related cortical potential features

Brain and muscle derived features to discriminate simple hand motor tasks for a rehabilitative BCI: comparative study on healthy and post-stroke individuals

Objective.Brain-Computer Interfaces targeting post-stroke recovery of the upper limb employ mainly electroencephalography to decode movement-related brain activation. Recently hybrid systems including muscular activity were introduced. We compared the motor task discrimination abilities of three different features, namely event-related desynchronization/synchronization (ERD/ERS) and movement-related cortical potential (MRCP) as brain-derived features and cortico-muscular coherence (CMC) as a hybrid brain-muscle derived feature, elicited in 13 healthy subjects and 13 stroke patients during the execution/attempt of two simple hand motor tasks (finger extension and grasping) commonly employed in upper limb rehabilitation protocols.Approach. We employed a three-way statistical design to investigate whether their ability to discriminate the two movements follows a specific temporal evolution along the movement execution and is eventually different among the three features and between the two groups. We also investigated the differences in performance at the single-subject level.Main results. The ERD/ERS and the CMC-based classification showed similar temporal evolutions of the performance with a significant increase in accuracy during the execution phase while MRCP-based accuracy peaked at movement onset. Such temporal dynamics were similar but slower in stroke patients when the movements were attempted with the affected hand (AH). Moreover, CMC outperformed the two brain features in healthy subjects and stroke patients when performing the task with their unaffected hand, whereas a higher variability across subjects was observed in patients performing the tasks with their AH. Interestingly, brain features performed better in this latter condition with respect to healthy subjects.Significance.Our results provide hints to improve the design of Brain-Computer Interfaces for post-stroke rehabilitation, emphasizing the need for personalized approaches tailored to patients' characteristics and to the intended rehabilitative target.

ADFCNN: Attention-Based Dual-Scale Fusion Convolutional Neural Network for Motor Imagery Brain-Computer Interface

Convolutional neural networks (CNNs) have been successfully applied to motor imagery (MI)-based brain-computer interface (BCI). Nevertheless, single-scale CNN fail to extract abundant information over a wide spectrum from EEG signals, while typical multi-scale CNNs cannot effectively fuse information from different scales with concatenation-based methods. To overcome these challenges, we propose a new scheme equipped with attention-based dual-scale fusion convolutional neural network (ADFCNN), which jointly extracts and fuses EEG spectral and spatial information at different scales. This scheme also provides novel insight through self-attention for effective information fusion from different scales. Specifically, temporal convolutions with two different kernel sizes identify EEG μ and β rhythms, while spatial convolutions at two different scales generate global and detailed spatial information, respectively, and the self-attention mechanism performs feature fusion based on the internal similarity of the concatenated features extracted by the dual-scale CNN. The proposed scheme achieves the superior performance compared with state-of-the-art methods in subject-specific motor imagery recognition on BCI Competition IV dataset 2a, 2b and OpenBMI dataset, with the cross-session average classification accuracies of 79.39% and significant improvements of 9.14% on BCI-IV2a, 87.81% and 7.66% on BCI-IV2b, 65.26% and 7.2% on OpenBMI dataset, and the within-session average classification accuracies of 86.87% and significant improvements of 10.89% on BCI-IV2a, 87.26% and 8.07% on BCI-IV2b, 84.29% and 5.17% on OpenBMI dataset, respectively. What is more, ablation experiments are conducted to investigate the mechanism and demonstrate the effectiveness of the dual-scale joint temporal-spatial CNN and self-attention modules. Visualization is also used to reveal the learning process and feature distribution of the model.

Decoding movement frequencies and limbs based on steady-state movement-related rhythms from noninvasive EEG

Objective.Decoding different types of movements noninvasively from electroencephalography (EEG) is an essential topic in neural engineering, especially in brain-computer interface. Although the widely used sensorimotor rhythm (SMR) is efficient in limb decoding, it lacks efficacy in decoding movement frequencies. Accumulating evidence supports the notion that the movement frequency is encoded in the steady-state movement-related rhythm (SSMRR). Our study has two primary objectives: firstly, to investigate the spatial-spectral representation of SSMRR in EEG during voluntary movements; secondly, to assess whether movement frequencies and limbs can be effectively decoded based on SSMRR.Approach.To comprehensively examine the representation of SSMRR, we investigated the frequency characteristics and spatial patterns associated with various rhythmic finger movements. Coherence analysis was performed between the sensor or source domain EEG and finger movements recorded by data gloves. A fusion model based on spectral SNR features and filter-bank common spatial pattern features was utilized to decode movement frequencies and limbs.Main results.At the group-level, sensor domain, and source domain coherence maps demonstrated that the accurate movement frequency (f0) and its first harmonic (f1) were encoded in the contralateral motor cortex. For the four-class classification, including two movement frequencies for both hands, the decoding accuracies for externally paced and internally paced movements were 73.14 ± 15.86% and 66.30 ± 17.26% (averaged across ten subjects, chance levels at 31.05% and 30.96%). Notably, the average results of five subjects with the highest decoding accuracies reached 87.21 ± 7.44% and 80.44 ± 7.99%.Significance.Our results verified the EEG representation of SSMRR and proved that the movement frequency and limb could be effectively decoded based on spatial-spectral features extracted from SSMRR. We suggest that SSMRR can serve as a complement to SMR to expand the range of decodable movement types and the approaches of limb decoding.

Spatio-temporal evolution of human neural activity during visually cued hand movements

Making hand movements in response to visual cues is common in daily life. It has been well known that this process activates multiple areas in the brain, but how these neural activations progress across space and time remains largely unknown. Taking advantage of intracranial electroencephalographic (iEEG) recordings using depth and subdural electrodes from 36 human subjects using the same task, we applied single-trial and cross-trial analyses to high-frequency iEEG activity. The results show that the neural activation was widely distributed across the human brain both within and on the surface of the brain, and focused specifically on certain areas in the parietal, frontal, and occipital lobes, where parietal lobes present significant left lateralization on the activation. We also demonstrate temporal differences across these brain regions. Finally, we evaluated the degree to which the timing of activity within these regions was related to sensory or motor function. The findings of this study promote the understanding of task-related neural processing of the human brain, and may provide important insights for translational applications.

Representation and decoding of bilateral arm motor imagery using unilateral cerebral LFP signals

Introduction

In the field of upper limb brain computer interfaces (BCIs), the research focusing on bilateral decoding mostly based on the neural signals from two cerebral hemispheres. In addition, most studies used spikes for decoding. Here we examined the representation and decoding of different laterality and regions arm motor imagery in unilateral motor cortex based on local field potentials (LFPs).

Methods

The LFP signals were recorded from a 96-channel Utah microelectrode array implanted in the left primary motor cortex of a paralyzed participant. There were 7 kinds of tasks: rest, left, right and bilateral elbow and wrist flexion. We performed time-frequency analysis on the LFP signals and analyzed the representation and decoding of different tasks using the power and energy of different frequency bands.

Results

The frequency range of <8 Hz and >38 Hz showed power enhancement, whereas 8-38 Hz showed power suppression in spectrograms while performing motor imagery. There were significant differences in average energy between tasks. What's more, the movement region and laterality were represented in two dimensions by demixed principal component analysis. The 135-300 Hz band signal had the highest decoding accuracy among all frequency bands and the contralateral and bilateral signals had more similar single-channel power activation patterns and larger signal correlation than contralateral and ipsilateral signals, bilateral and ipsilateral signals.

Discussion

The results showed that unilateral LFP signals had different representations for bilateral motor imagery on the average energy of the full array and single-channel power levels, and different tasks could be decoded. These proved the feasibility of multilateral BCI based on the unilateral LFP signal to broaden the application of BCI technology.

Clinical trial registration

https://www.chictr.org.cn/showproj.aspx?proj=130829, identifier ChiCTR2100050705.

Discrepancy between inter- and intra-subject variability in EEG-based motor imagery brain-computer interface: Evidence from multiple perspectives

Introduction

Inter- and intra-subject variability are caused by the variability of the psychological and neurophysiological factors over time and across subjects. In the application of in Brain-Computer Interfaces (BCI), the existence of inter- and intra-subject variability reduced the generalization ability of machine learning models seriously, which further limited the use of BCI in real life. Although many transfer learning methods can compensate for the inter- and intra-subject variability to some extent, there is still a lack of clear understanding about the change of feature distribution between the cross-subject and cross-session electroencephalography (EEG) signal.

Methods

To investigate this issue, an online platform for motor-imagery BCI decoding has been built in this work. The EEG signal from both the multi-subject (Exp1) and multi-session (Exp2) experiments has been analyzed from multiple perspectives.

Results

Firstly we found that with the similar variability of classification results, the time-frequency response of the EEG signal within-subject in Exp2 is more consistent than cross-subject results in Exp1. Secondly, the standard deviation of the common spatial pattern (CSP) feature has a significant difference between Exp1 and Exp2. Thirdly, for model training, different strategies for the training sample selection should be applied for the cross-subject and cross-session tasks.

Discussion

All these findings have deepened the understanding of inter- and intra-subject variability. They can also guide practice for the new transfer learning methods development in EEG-based BCI. In addition, these results also proved that BCI inefficiency was not caused by the subject's unable to generate the event-related desynchronization/synchronization (ERD/ERS) signal during the motor imagery.

Decoding EEG rhythms offline and online during motor imagery for standing and sitting based on a brain-computer interface

Motor imagery (MI)-based brain-computer interface (BCI) systems have shown promising advances for lower limb motor rehabilitation. The purpose of this study was to develop an MI-based BCI for the actions of standing and sitting. Thirty-two healthy subjects participated in the study using 17 active EEG electrodes. We used a combination of the filter bank common spatial pattern (FBCSP) method and the regularized linear discriminant analysis (RLDA) technique for decoding EEG rhythms offline and online during motor imagery for standing and sitting. The offline analysis indicated the classification of motor imagery and idle state provided a mean accuracy of 88.51 ± 1.43% and 85.29 ± 1.83% for the sit-to-stand and stand-to-sit transitions, respectively. The mean accuracies of the sit-to-stand and stand-to-sit online experiments were 94.69 ± 1.29% and 96.56 ± 0.83%, respectively. From these results, we believe that the MI-based BCI may be useful to future brain-controlled standing systems.

A flexible and accurate method for electroencephalography rhythms extraction based on circulant singular spectrum analysis

Rhythms extraction from electroencephalography (EEG) signals can be used to monitor the physiological and pathological states of the brain and has attracted much attention in recent studies. A flexible and accurate method for EEG rhythms extraction was proposed by incorporating a novel circulant singular spectrum analysis (CiSSA). The EEG signals are decomposed into the sum of a set of orthogonal reconstructed components (RCs) at known frequencies. The frequency bandwidth of each RC is limited to a particular brain rhythm band, with no frequency mixing between different RCs. The RCs are then grouped flexibly to extract the desired EEG rhythms based on the known frequencies. The extracted brain rhythms are accurate and no mixed components of other rhythms or artifacts are included. Simulated EEG data based on the Markov Process Amplitude EEG model and experimental EEG data in the eyes-open and eyes-closed states were used to verify the CiSSA-based method. The results showed that the CiSSA-based method is flexible in alpha rhythms extraction and has a higher accuracy in distinguishing between the eyes-open and eyes-closed states, compared with the basic SSA method, the wavelet decomposition method, and the finite impulse response filtering method.

Motor Imagery Classification Using Inter-Task Transfer Learning via A Channel-Wise Variational Autoencoder-based Convolutional Neural Network

Highly sophisticated control based on a brain-computer interface (BCI) requires decoding kinematic information from brain signals. The forearm is a region of the upper limb that is often used in everyday life, but intuitive movements within the same limb have rarely been investigated in previous BCI studies. In this study, we focused on various forearm movement decoding from electroencephalography (EEG) signals using a small number of samples. Ten healthy participants took part in an experiment and performed motor execution (ME) and motor imagery (MI) of the intuitive movement tasks (Dataset I). We propose a convolutional neural network using a channel-wise variational autoencoder (CVNet) based on inter-task transfer learning. We approached that training the reconstructed ME-EEG signals together will also achieve more sufficient classification performance with only a small amount of MI-EEG signals. The proposed CVNet was validated on our own Dataset I and a public dataset, BNCI Horizon 2020 (Dataset II). The classification accuracies of various movements are confirmed to be 0.83 (±0.04) and 0.69 (±0.04) for Dataset I and II, respectively. The results show that the proposed method exhibits performance increases of approximately 0.09~0.27 and 0.08~0.24 compared with the conventional models for Dataset I and II, respectively. The outcomes suggest that the training model for decoding imagined movements can be performed using data from ME and a small number of data samples from MI. Hence, it is presented the feasibility of BCI learning strategies that can sufficiently learn deep learning with a few amount of calibration dataset and time only, with stable performance.

Robust anticipation of continuous steering actions from electroencephalographic data during simulated driving

Driving a car requires high cognitive demands, from sustained attention to perception and action planning. Recent research investigated the neural processes reflecting the planning of driving actions, aiming to better understand the factors leading to driving errors and to devise methodologies to anticipate and prevent such errors by monitoring the driver’s cognitive state and intention. While such anticipation was shown for discrete driving actions, such as emergency braking, there is no evidence for robust neural signatures of continuous action planning. This study aims to fill this gap by investigating continuous steering actions during a driving task in a car simulator with multimodal recordings of behavioural and electroencephalography (EEG) signals. System identification is used to assess whether robust neurophysiological signatures emerge before steering actions. Linear decoding models are then used to determine whether such cortical signals can predict continuous steering actions with progressively longer anticipation. Results point to significant EEG signatures of continuous action planning. Such neural signals show consistent dynamics across participants for anticipations up to 1 s, while individual-subject neural activity could reliably decode steering actions and predict future actions for anticipations up to 1.8 s. Finally, we use canonical correlation analysis to attempt disentangling brain and non-brain contributors to the EEG-based decoding. Our results suggest that low-frequency cortical dynamics are involved in the planning of steering actions and that EEG is sensitive to that neural activity. As a result, we propose a framework to investigate anticipatory neural activity in realistic continuous motor tasks.

Sparse representation-based classification with two-dimensional dictionary optimization for motor imagery EEG pattern recognition

BACKGROUND

Sparse representation-based classification (SRC) has more advantages in motor imagery EEG pattern recognition, and the quality of dictionary construction directly determines the performance of SRC. In this paper, we proposed a two-dimensional dictionary optimization (TDDO) method to directly improve the performance of SRC.

NEW METHOD

Firstly, an initial dictionary was constructed with multi-band features extracted by filter band common spatial pattern (FBCSP). Then Lasso regression is used to select significant features in each atom synchronously in the horizontal direction, and the KNN-based method is used to clean up noise atoms in the vertical direction. Finally, an SRC method by training samples linearly representing test samples was implemented in classification.

RESULTS

The results show the necessity and rationality of TDDO-SRC method. The highest average classification accuracy of 86.5% and 92.4% is obtained on two public datasets.

COMPARISON WITH EXISTING METHOD(S)

The proposed method has more superior classification accuracy compared to traditional methods and existing winners' methods.

CONCLUSIONS

The quality of dictionary construction has a great impact on the robustness of SRC. And compared with the original SRC, the classification accuracy of the optimized TDDO-SRC is greatly improved.

Transcranial Focused Ultrasound Neuromodulation of Voluntary Movement-Related Cortical Activity in Humans

OBJECTIVE

Transcranial focused ultrasound (tFUS) is an emerging non-invasive brain stimulation tool for safely and reversibly modulating brain circuits. The effectiveness of tFUS on human brain has been demonstrated, but how tFUS influences the human voluntary motor processing in the brain remains unclear.

METHODS

We apply low-intensity tFUS to modulate the movement-related cortical potential (MRCP) originating from human subjects practicing a voluntary foot tapping task. 64-channel electroencephalograph (EEG) is recorded concurrently and further used to reconstruct the brain source activity specifically at the primary leg motor cortical area using the electrophysiological source imaging (ESI).

RESULTS

The ESI illustrates the ultrasound modulated MRCP source dynamics with high spatiotemporal resolutions. The MRCP source is imaged and its source profile is further evaluated for assessing the tFUS neuromodulatory effects on the voluntary MRCP. Moreover, the effect of ultrasound pulse repetition frequency (UPRF) is further assessed in modulating the MRCP. The ESI results show that tFUS significantly increases the MRCP source profile amplitude (MSPA) comparing to a sham ultrasound condition, and further, a high UPRF enhances the MSPA more than a low UPRF does.

CONCLUSION

The present results demonstrate the neuromodulatory effects of the low-intensity tFUS on enhancing the human voluntary movement-related cortical activities evidenced through the ESI.

SIGNIFICANCE

This work provides the first evidence of tFUS enhancing the human endogenous motor cortical activities through excitatory modulation.

Learning Common Time-Frequency-Spatial Patterns for Motor Imagery Classification

The common spatial patterns (CSP) algorithm is the most popular spatial filtering method applied to extract electroencephalogram (EEG) features for motor imagery (MI) based brain-computer interface (BCI) systems. The effectiveness of the CSP algorithm depends on optimal selection of the frequency band and time window from the EEG. Many algorithms have been designed to optimize frequency band selection for CSP, while few algorithms seek to optimize the time window. This study proposes a novel framework, termed common time-frequency-spatial patterns (CTFSP), to extract sparse CSP features from multi-band filtered EEG data in multiple time windows. Specifically, the whole MI period is first segmented into multiple subseries using a sliding time window approach. Then, sparse CSP features are extracted from multiple frequency bands in each time window. Finally, multiple support vector machine (SVM) classifiers with the Radial Basis Function (RBF) kernel are trained to identify the MI tasks and the voting result of these classifiers determines the final output of the BCI. This study applies the proposed CTFSP algorithm to three public EEG datasets (BCI competition III dataset IVa, BCI competition III dataset IIIa, and BCI competition IV dataset 1) to validate its effectiveness, compared against several other state-of-the-art methods. The experimental results demonstrate that the proposed algorithm is a promising candidate for improving the performance of MI-BCI systems.

Regression Networks for Neurophysiological Indicator Evaluation in Practicing Motor Imagery Tasks

Motor Imagery (MI) promotes motor learning in activities, like developing professional motor skills, sports gestures, and patient rehabilitation. However, up to 30% of users may not develop enough coordination skills after training sessions because of inter and intra-subject variability. Here, we develop a data-driven estimator, termed Deep Regression Network (DRN), which jointly extracts and performs the regression analysis in order to assess the efficiency of the individual brain networks in practicing MI tasks. The proposed double-stage estimator initially learns a pool of deep patterns, extracted from the input data, in order to feed a neural regression model, allowing for infering the distinctiveness between subject assemblies having similar variability. The results, which were obtained on real-world MI data, prove that the DRN estimator fosters pre-training neural desynchronization and initial training synchronization to predict the bi-class accuracy response, thus providing a better understanding of the Brain-Computer Interface inefficiency of subjects.