From classic motor imagery to complex movement intention decoding: The noninvasive Graz-BCI approach

A Few-Shot Transfer Learning Approach for Motion Intention Decoding from Electroencephalographic Signals

In this study, a few-shot transfer learning approach was introduced to decode movement intention from electroencephalographic (EEG) signals, allowing to recognize new tasks with minimal adaptation. To this end, a dataset of EEG signals recorded during the preparation of complex sub-movements was created from a publicly available data collection. The dataset was divided into two parts: the source domain dataset (including 5 classes) and the support (target domain) dataset, (including 2 classes) with no overlap between the two datasets in terms of classes. The proposed methodology consists in projecting EEG signals into the space-frequency-time domain, in processing such projections (rearranged in channels × frequency frames) by means of a custom EEG-based deep neural network (denoted as EEGframeNET5), and then adapting the system to recognize new tasks through a few-shot transfer learning approach. The proposed method achieved an average accuracy of 72.45 ± 4.19% in the 5-way classification of samples from the source domain dataset, outperforming comparable studies in the literature. In the second phase of the study, a few-shot transfer learning approach was proposed to adapt the neural system and make it able to recognize new tasks in the support dataset. The results demonstrated the system's ability to adapt and recognize new tasks with an average accuracy of 80 ± 0.12% in discriminating hand opening/closing preparation and outperforming reported results in the literature. This study suggests the effectiveness of EEG in capturing information related to the motor preparation of complex movements, potentially paving the way for BCI systems based on motion planning decoding. The proposed methodology could be straightforwardly extended to advanced EEG signal processing in other scenarios, such as motor imagery or neural disorder classification.

How Integration of a Brain-Machine Interface and Obstacle Detection System Can Improve Wheelchair Control via Movement Imagery

This study presents a human-computer interaction combined with a brain-machine interface (BMI) and obstacle detection system for remote control of a wheeled robot through movement imagery, providing a potential solution for individuals facing challenges with conventional vehicle operation. The primary focus of this work is the classification of surface EEG signals related to mental activity when envisioning movement and deep relaxation states. Additionally, this work presents a system for obstacle detection based on image processing. The implemented system constitutes a complementary part of the interface. The main contributions of this work include the proposal of a modified 10-20-electrode setup suitable for motor imagery classification, the design of two convolutional neural network (CNNs) models employed to classify signals acquired from sixteen EEG channels, and the implementation of an obstacle detection system based on computer vision integrated with a brain-machine interface. The models developed in this study achieved an accuracy of 83% in classifying EEG signals. The resulting classification outcomes were subsequently utilized to control the movement of a mobile robot. Experimental trials conducted on a designated test track demonstrated real-time control of the robot. The findings indicate the feasibility of integration of the obstacle detection system for collision avoidance with the classification of motor imagery for the purpose of brain-machine interface control of vehicles. The elaborated solution could help paralyzed patients to safely control a wheelchair through EEG and effectively prevent unintended vehicle movements.

Advanced Electrode Technologies for Noninvasive Brain-Computer Interfaces

Brain-computer interfaces (BCIs) have garnered significant attention in recent years due to their potential applications in medical, assistive, and communication technologies. Building on this, noninvasive BCIs stand out as they provide a safe and user-friendly method for interacting with the human brain. In this work, we provide a comprehensive overview of the latest developments and advancements in material, design, and application of noninvasive BCIs electrode technology. We also explore the challenges and limitations currently faced by noninvasive BCI electrode technology and sketch out the technological roadmap from three dimensions: Materials and Design; Performances; Mode and Function. We aim to unite research efforts within the field of noninvasive BCI electrode technology, focusing on the consolidation of shared goals and fostering integrated development strategies among a diverse array of multidisciplinary researchers.

Augmented action observation: Theory and practical applications in sensorimotor rehabilitation

Sensory feedback is a fundamental aspect of effective motor learning in sport and clinical contexts. One way to provide this is through sensory augmentation, where extrinsic sensory information are associated with, and modulated by, movement. Traditionally, sensory augmentation has been used as an online strategy, where feedback is provided during physical execution of an action. In this article, we argue that action observation can be an additional effective channel to provide augmented feedback, which would be complementary to other, more traditional, motor learning and sensory augmentation strategies. Given these similarities between observing and executing an action, action observation could be used when physical training is difficult or not feasible, for example during immobilization or during the initial stages of a rehabilitation protocol when peripheral fatigue is a common issue. We review the benefits of observational learning and preliminary evidence for the effectiveness of using augmented action observation to improve learning. We also highlight current knowledge gaps which make the transition from laboratory to practical contexts difficult. Finally, we highlight the key areas of focus for future research.

Decoding movement frequencies and limbs based on steady-state movement-related rhythms from noninvasive EEG

Objective.Decoding different types of movements noninvasively from electroencephalography (EEG) is an essential topic in neural engineering, especially in brain-computer interface. Although the widely used sensorimotor rhythm (SMR) is efficient in limb decoding, it lacks efficacy in decoding movement frequencies. Accumulating evidence supports the notion that the movement frequency is encoded in the steady-state movement-related rhythm (SSMRR). Our study has two primary objectives: firstly, to investigate the spatial-spectral representation of SSMRR in EEG during voluntary movements; secondly, to assess whether movement frequencies and limbs can be effectively decoded based on SSMRR.Approach.To comprehensively examine the representation of SSMRR, we investigated the frequency characteristics and spatial patterns associated with various rhythmic finger movements. Coherence analysis was performed between the sensor or source domain EEG and finger movements recorded by data gloves. A fusion model based on spectral SNR features and filter-bank common spatial pattern features was utilized to decode movement frequencies and limbs.Main results.At the group-level, sensor domain, and source domain coherence maps demonstrated that the accurate movement frequency (f0) and its first harmonic (f1) were encoded in the contralateral motor cortex. For the four-class classification, including two movement frequencies for both hands, the decoding accuracies for externally paced and internally paced movements were 73.14 ± 15.86% and 66.30 ± 17.26% (averaged across ten subjects, chance levels at 31.05% and 30.96%). Notably, the average results of five subjects with the highest decoding accuracies reached 87.21 ± 7.44% and 80.44 ± 7.99%.Significance.Our results verified the EEG representation of SSMRR and proved that the movement frequency and limb could be effectively decoded based on spatial-spectral features extracted from SSMRR. We suggest that SSMRR can serve as a complement to SMR to expand the range of decodable movement types and the approaches of limb decoding.

A Combined Virtual Electrode-Based ESA and CNN Method for MI-EEG Signal Feature Extraction and Classification

A Brain-Computer Interface (BCI) is a medium for communication between the human brain and computers, which does not rely on other human neural tissues, but only decodes Electroencephalography (EEG) signals and converts them into commands to control external devices. Motor Imagery (MI) is an important BCI paradigm that generates a spontaneous EEG signal without external stimulation by imagining limb movements to strengthen the brain's compensatory function, and it has a promising future in the field of computer-aided diagnosis and rehabilitation technology for brain diseases. However, there are a series of technical difficulties in the research of motor imagery-based brain-computer interface (MI-BCI) systems, such as: large individual differences in subjects and poor performance of the cross-subject classification model; a low signal-to-noise ratio of EEG signals and poor classification accuracy; and the poor online performance of the MI-BCI system. To address the above problems, this paper proposed a combined virtual electrode-based EEG Source Analysis (ESA) and Convolutional Neural Network (CNN) method for MI-EEG signal feature extraction and classification. The outcomes reveal that the online MI-BCI system developed based on this method can improve the decoding ability of multi-task MI-EEG after training, it can learn generalized features from multiple subjects in cross-subject experiments and has some adaptability to the individual differences of new subjects, and it can decode the EEG intent online and realize the brain control function of the intelligent cart, which provides a new idea for the research of an online MI-BCI system.

Getting off track: Cortical feedback processing network modulated by continuous error signal during target-feedback mismatch

Performance monitoring and feedback processing - especially in the wake of erroneous outcomes - represent a crucial aspect of everyday life, allowing us to deal with imminent threats in the short term but also promoting necessary behavioral adjustments in the long term to avoid future conflicts. Over the last thirty years, research extensively analyzed the neural correlates of processing discrete error stimuli, unveiling the error-related negativity (ERN) and error positivity (Pe) as two main components of the cognitive response. However, the connection between the ERN/Pe and distinct stages of error processing, ranging from action monitoring to subsequent corrective behavior, remains ambiguous. Furthermore, mundane actions such as steering a vehicle already transgress the scope of discrete erroneous events and demand fine-tuned feedback control, and thus, the processing of continuous error signals - a topic scarcely researched at present. We analyzed two electroencephalography datasets to investigate the processing of continuous erroneous signals during a target tracking task, employing feedback in various levels and modalities. We observed significant differences between correct (slightly delayed) and erroneous feedback conditions in the larger one of the two datasets that we analyzed, both in sensor and source space. Furthermore, we found strong error-induced modulations that appeared consistent across datasets and error conditions, indicating a clear order of engagement of specific brain regions that correspond to individual components of error processing.

Transfer Learning in Trajectory Decoding: Sensor or Source Space?

In this study, across-participant and across-session transfer learning was investigated to minimize the calibration time of the brain-computer interface (BCI) system in the context of continuous hand trajectory decoding. We reanalyzed data from a study with 10 able-bodied participants across three sessions. A leave-one-participant-out (LOPO) model was utilized as a starting model. Recursive exponentially weighted partial least squares regression (REW-PLS) was employed to overcome the memory limitation due to the large pool of training data. We considered four scenarios: generalized with no update (Gen), generalized with cumulative update (GenC), and individual models with cumulative (IndC) and non-cumulative (Ind) updates, with each one trained with sensor-space features or source-space features. The decoding performance in generalized models (Gen and GenC) was lower than the chance level. In individual models, the cumulative update (IndC) showed no significant improvement over the non-cumulative model (Ind). The performance showed the decoder's incapability to generalize across participants and sessions in this task. The results suggested that the best correlation could be achieved with the sensor-space individual model, despite additional anatomical information in the source-space features. The decoding pattern showed a more localized pattern around the precuneus over three sessions in Ind models.

Analysis of the Relationship Between Motor Imagery and Age-Related Fatigue for CNN Classification of the EEG Data

Background

The aging of the world population poses a major health challenge, and brain–computer interface (BCI) technology has the potential to provide assistance and rehabilitation for the elderly.

Objectives

This study aimed to investigate the electroencephalogram (EEG) characteristics during motor imagery by comparing young and elderly, and study Convolutional Neural Networks (CNNs) classification for the elderly population in terms of fatigue analysis in both frontal and parietal regions.

Methods

A total of 20 healthy individuals participated in the study, including 10 young and 10 older adults. All participants completed the left- and right-hand motor imagery experiment. The energy changes in the motor imagery process were analyzed using time–frequency graphs and quantified event-related desynchronization (ERD) values. The fatigue level of the motor imagery was assessed by two indicators: (θ + α)/β and θ/β, and fatigue-sensitive channels were distinguished from the parietal region of the brain. Then, rhythm entropy was introduced to analyze the complexity of the cognitive activity. The phase-lock values related to the parietal and frontal lobes were calculated, and their temporal synchronization was discussed. Finally, the motor imagery EEG data was classified by CNNs, and the accuracy was discussed based on the analysis results.

Result

For the young and elderly, ERD was observed in C3 and C4 channels, and their fatigue-sensitive channels in the parietal region were slightly different. During the experiment, the rhythm entropy of the frontal lobe showed a decreasing trend with time for most of the young subjects, while there was an increasing trend for most of the older ones. Using the CNN classification method, the elderly achieved around 70% of the average classification accuracy, which is almost the same for the young adults.

Conclusion

Compared with the young adults, the elderly are less affected by the level of cognitive fatigue during motor imagery, but the classification accuracy of motor imagery data in the elderly may be slightly lower than that in young persons. At the same time, the deep learning method also provides a potentially feasible option for the application of motor-imagery BCI (MI-BCI) in the elderly by considering the ERD and fatigue phenomenon together.

A meta-learning BCI for estimating decision confidence

OBJECTIVE

We investigated whether a recently introduced transfer-learning technique based on meta-learning could improve the performance of Brain-Computer Interfaces (BCIs) for decision-confidence prediction with respect to more traditional machine learning methods.

APPROACH

We adapted the meta-learning by biased regularisation algorithm to the problem of predicting decision confidence from EEG and EOG data on a decision-by-decision basis in a difficult target discrimination task based on video feeds. The method exploits previous participants' data to produce a prediction algorithm that is then quickly tuned to new participants. We compared it with with the traditional single-subject training almost universally adopted in BCIs, a state-of-the-art transfer learning technique called Domain Adversarial Neural Networks (DANN), a transfer-learning adaptation of a zero-training method we used recently for a similar task, and with a simple baseline algorithm.

MAIN RESULTS

The meta-learning approach was significantly better than other approaches in most conditions, and much better in situations where limited data from a new participant are available for training/tuning. Meta-learning by biased regularisation allowed our BCI to seamlessly integrate information from past participants with data from a specific user to produce high-performance predictors. Its robustness in the presence of small training sets is a real-plus in BCI applications, as new users need to train the BCI for a much shorter period.

SIGNIFICANCE

Due to the variability and noise of EEG/EOG data, BCIs need to be normally trained with data from a specific participant. This work shows that even better performance can be obtained using our version of meta-learning by biased regularisation.

Continuous 2D trajectory decoding from attempted movement: across-session performance in able-bodied and feasibility in a spinal cord injured participant

OBJECTIVE

In people with a cervical spinal cord injury (SCI) or degenerative diseases leading to limited motor function, restoration of upper limb movement has been a goal of the brain-computer interface (BCI) field for decades. Recently, research from our group investigated non-invasive and real-time decoding of continuous movement in able-bodied participants from low-frequency brain signals during a target-tracking task. To advance our setup towards motor-impaired end users, we consequently chose a new paradigm based on attempted movement.

APPROACH

Here, we present the results of two studies. During the first study, data of ten able-bodied participants completing a target-tracking/shape-tracing task on-screen were investigated in terms of improvements in decoding performance due to user training. In a second study, a spinal cord injured participant underwent the same tasks. To investigate the merit of employing attempted movement in end users with SCI, data of the spinal cord injured participant were recorded twice; once within an observation only condition, and once while simultaneously attempting movement.

MAIN RESULTS

We observed mean correlation well above chance level for continuous motor decoding based on attempted movement in able-bodied participants. No global improvement over three sessions, both in sensor and source space, could be observed across all participants and movement parameters. In the participant with SCI, decoding performance well above chance was found.

SIGNIFICANCE

No presence of a learning effect in continuous attempted movement decoding in able-bodied participants could be observed. In contrast, non-significantly varying decoding patterns may promote the use of source space decoding in terms of generalized decoders utilizing transfer learning. Furthermore, above-chance correlations for attempted movement decoding ranging between those of observation only and executed movement were seen in one spinal cord injured participant, suggesting attempted movement decoding as a possible link between feasibility studies in able-bodied and actual applications in motor impaired end users.

Neurofeedback Training of Alpha Relative Power Improves the Performance of Motor Imagery Brain-Computer Interface

Significant variation in performance in motor imagery (MI) tasks impedes their wide adoption for brain-computer interface (BCI) applications. Previous researchers have found that resting-state alpha-band power is positively correlated with MI-BCI performance. In this study, we designed a neurofeedback training (NFT) protocol based on the up-regulation of the alpha band relative power (RP) to investigate its effect on MI-BCI performance. The principal finding of this study is that alpha NFT could successfully help subjects increase alpha-rhythm power and improve their MI-BCI performance. An individual difference was also found in this study in that subjects who increased alpha power more had a better performance improvement. Additionally, the functional connectivity (FC) of the frontal-parietal (FP) network was found to be enhanced after alpha NFT. However, the enhancement failed to reach a significant level after multiple comparisons correction. These findings contribute to a better understanding of the neurophysiological mechanism of cognitive control through alpha regulation.

Applying Dimensionality Reduction Techniques in Source-Space Electroencephalography via Template and Magnetic Resonance Imaging-Derived Head Models to Continuously Decode Hand Trajectories

Several studies showed evidence supporting the possibility of hand trajectory decoding from low-frequency electroencephalography (EEG). However, the decoding in the source space via source localization is scarcely investigated. In this study, we tried to tackle the problem of collinearity due to the higher number of signals in the source space by two folds: first, we selected signals in predefined regions of interest (ROIs); second, we applied dimensionality reduction techniques to each ROI. The dimensionality reduction techniques were computing the mean (Mean), principal component analysis (PCA), and locality preserving projections (LPP). We also investigated the effect of decoding between utilizing a template head model and a subject-specific head model during the source localization. The results indicated that applying source-space decoding with PCA yielded slightly higher correlations and signal-to-noise (SNR) ratios than the sensor-space approach. We also observed slightly higher correlations and SNRs when applying the subject-specific head model than the template head model. However, the statistical tests revealed no significant differences between the source-space and sensor-space approaches and no significant differences between subject-specific and template head models. The decoder with Mean and PCA utilizes information mainly from precuneus and cuneus to decode the velocity kinematics similarly in the subject-specific and template head models.

Realizing the Application of EEG Modeling in BCI Classification: Based on a Conditional GAN Converter

Electroencephalogram (EEG) modeling in brain-computer interface (BCI) provides a theoretical foundation for its development. However, limited by the lack of guidelines in model parameter selection and the inability to obtain personal tissue information in practice, EEG modeling in BCI is mainly focused on the theoretical qualitative level which shows a gap between the theory and its application. Based on such problems, this work combined the surface EEG simulation with a converter based on the generative adversarial network (GAN), to establish the connection from simulated EEG to its application in BCI classification. For the scalp EEGs modeling, a mathematical model was built according to the physics of surface EEG, which consisted of the parallel 3-population neural mass model, the equivalent dipole, and the forward computation. For application, a converter based on the conditional GAN was designed, to transfer the simulated theoretical-only EEG to its practical version, in the lack of individual bio-information. To verify the feasibility, based on the latest microexpression-assisted BCI paradigm proposed by our group, the converted simulated EEGs were used in the training of BCI classifiers. The results indicated that, compared with training with insufficient real data, by adding the simulated EEGs, the overall performance showed a significant improvement (P = 0.04 < 0.05), and the test performance can be improved by 2.17% ± 4.23, in which the largest increase was up to 12.60% ± 1.81. Through this work, the link from theoretical EEG simulation to BCI classification has been initially established, providing an enhanced novel solution for the application of EEG modeling in BCI.

Single-Trial MEG Data Can Be Denoised Through Cross-Subject Predictive Modeling

A pervasive challenge in brain imaging is the presence of noise that hinders investigation of underlying neural processes, with Magnetoencephalography (MEG) in particular having very low Signal-to-Noise Ratio (SNR). The established strategy to increase MEG's SNR involves averaging multiple repetitions of data corresponding to the same stimulus. However, repetition of stimulus can be undesirable, because underlying neural activity has been shown to change across trials, and repeating stimuli limits the breadth of the stimulus space experienced by subjects. In particular, the rising popularity of naturalistic studies with a single viewing of a movie or story necessitates the discovery of new approaches to increase SNR. We introduce a simple framework to reduce noise in single-trial MEG data by leveraging correlations in neural responses across subjects as they experience the same stimulus. We demonstrate its use in a naturalistic reading comprehension task with 8 subjects, with MEG data collected while they read the same story a single time. We find that our procedure results in data with reduced noise and allows for better discovery of neural phenomena. As proof-of-concept, we show that the N400m's correlation with word surprisal, an established finding in literature, is far more clearly observed in the denoised data than the original data. The denoised data also shows higher decoding and encoding accuracy than the original data, indicating that the neural signals associated with reading are either preserved or enhanced after the denoising procedure.

Does feedback based on FES-evoked nociceptive withdrawal reflex condition event-related desynchronization? An exploratory study with brain-computer interfaces

Introduction.Event-related desynchronization (ERD) is used in brain-computer interfaces (BCI) to detect the user's motor intention (MI) and convert it into a command for an actuator to provide sensory feedback or mobility, for example by means of functional electrical stimulation (FES). Recent studies have proposed to evoke the nociceptive withdrawal reflex (NWR) using FES, in order to evoke synergistic movements of the lower limb and to facilitate the gait rehabilitation of stroke patients. The use of NWR to provide sensorimotor feedback in ERD-based BCI is novel; thererfore, the conditioning effect that nociceptive stimuli might have on MI is still unknown.Objetive.To assess the ERD produced during the MI after FES-evoked NWR, in order to evaluate if nociceptive stimuli condition subsequent ERDs.Methods. Data from 528 electroencephalography trials of 8 healthy volunteers were recorded and analyzed. Volunteers used an ERD-based BCI, which provided two types of feedback: intrisic by the FES-evoked NWR and extrinsic by virtual reality. The electromyogram of the tibialis anterior muscle was also recorded. The main outcome variables were the normalized root mean square of the evoked electromyogram (RMS_norm), the average electroencephalogram amplitude at the ERD frequency during MI (A¯MI) and the percentage decrease ofA¯MIrelative to rest (ERD%) at the first MI subsequent to the activation of the BCI.Results.No evidence of changes of theRMS_normon both theA¯MI(p = 0.663) and theERD%(p = 0.252) of the subsequent MI was detected. A main effect of the type of feedback was found in the subsequentA¯MI(p < 0.001), with intrinsic feedback resulting in a largerA¯MI.Conclusions.No evidence of ERD conditioning was observed using BCI feedback based on FES-evoked NWR .Significance.FES-evoked NWR could constitute a potential feedback modality in an ERD-based BCI to facilitate motor recovery of stroke people.

A Hybrid-Domain Deep Learning-Based BCI For Discriminating Hand Motion Planning From EEG Sources

In this paper, a hybrid-domain deep learning (DL)-based neural system is proposed to decode hand movement preparation phases from electroencephalographic (EEG) recordings. The system exploits information extracted from the temporal-domain and time-frequency-domain, as part of a hybrid strategy, to discriminate the temporal windows (i.e. EEG epochs) preceding hand sub-movements (open/close) and the resting state. To this end, for each EEG epoch, the associated cortical source signals in the motor cortex and the corresponding time-frequency (TF) maps are estimated via beamforming and Continuous Wavelet Transform (CWT), respectively. Two Convolutional Neural Networks (CNNs) are designed: specifically, the first CNN is trained over a dataset of temporal (T) data (i.e. EEG sources), and is referred to as T-CNN; the second CNN is trained over a dataset of TF data (i.e. TF-maps of EEG sources), and is referred to as TF-CNN. Two sets of features denoted as T-features and TF-features, extracted from T-CNN and TF-CNN, respectively, are concatenated in a single features vector (denoted as TTF-features vector) which is used as input to a standard multi-layer perceptron for classification purposes. Experimental results show a significant performance improvement of our proposed hybrid-domain DL approach as compared to temporal-only and time-frequency-only-based benchmark approaches, achieving an average accuracy of [Formula: see text]%.

Decoding Covert Speech From EEG-A Comprehensive Review

Over the past decade, many researchers have come up with different implementations of systems for decoding covert or imagined speech from EEG (electroencephalogram). They differ from each other in several aspects, from data acquisition to machine learning algorithms, due to which, a comparison between different implementations is often difficult. This review article puts together all the relevant works published in the last decade on decoding imagined speech from EEG into a single framework. Every important aspect of designing such a system, such as selection of words to be imagined, number of electrodes to be recorded, temporal and spatial filtering, feature extraction and classifier are reviewed. This helps a researcher to compare the relative merits and demerits of the different approaches and choose the one that is most optimal. Speech being the most natural form of communication which human beings acquire even without formal education, imagined speech is an ideal choice of prompt for evoking brain activity patterns for a BCI (brain-computer interface) system, although the research on developing real-time (online) speech imagery based BCI systems is still in its infancy. Covert speech based BCI can help people with disabilities to improve their quality of life. It can also be used for covert communication in environments that do not support vocal communication. This paper also discusses some future directions, which will aid the deployment of speech imagery based BCI for practical applications, rather than only for laboratory experiments.

Motor Imagery Hand Movement Direction Decoding Using Brain Computer Interface to Aid Stroke Recovery and Rehabilitation

Motor Imagery (MI)-based Brain Computer Interface (BCI) system is a potential technology for active neurorehabilitation of stroke patients by complementing the conventional passive rehabilitation methods. Research to date mainly focused on classifying left vs. right hand/foot MI of stroke patients. Though a very few studies have reported decoding imagined hand movement directions using electroencephalogram (EEG)-based BCI, the experiments were conducted on healthy subjects. Our work analyzes MI-based brain cortical activity from EEG signals and decodes the imagined hand movement directions in stroke patients. The decoded direction (left vs. right) of hand movement imagination is used to provide control commands to a motorized arm support on which patient's affected (paralyzed) arm is placed. This enables the patient to move his/her stroke-affected hand towards the intended (imagined) direction that aids neuroplasticity in the brain. The synchronization measure called Phase Locking Value (PLV), extracted from EEG, is the neuronal signature used to decode the directional movement of the MI task. Event-related desynchronization/synchronization (ERD/ERS) analysis on Mu and Beta frequency bands of EEG is done to select the time bin corresponding to the MI task. The dissimilarities between the two directions of MI tasks are identified by selecting the most significant channel pairs that provided maximum difference in PLV features. The training protocol has an initial calibration session followed by a feedback session with 50 trials of MI task in each session. The feedback session extracts PLV features corresponding to most significant channel pairs which are identified in the calibration session and is used to predict the direction of MI task in left/right direction. An average MI direction classification accuracy of 74.44% is obtained in performing the training protocol and 68.63% from the prediction protocol during feedback session on 16 stroke patients.

Multiclass Classification Based on Combined Motor Imageries

Motor imagery (MI) allows the design of self-paced brain–computer interfaces (BCIs), which can potentially afford an intuitive and continuous interaction. However, the implementation of non-invasive MI-based BCIs with more than three commands is still a difficult task. First, the number of MIs for decoding different actions is limited by the constraint of maintaining an adequate spacing among the corresponding sources, since the electroencephalography (EEG) activity from near regions may add up. Second, EEG generates a rather noisy image of brain activity, which results in a poor classification performance. Here, we propose a solution to address the limitation of identifiable motor activities by using combined MIs (i.e., MIs involving 2 or more body parts at the same time). And we propose two new multilabel uses of the Common Spatial Pattern (CSP) algorithm to optimize the signal-to-noise ratio, namely MC2CMI and MC2SMI approaches. We recorded EEG signals from seven healthy subjects during an 8-class EEG experiment including the rest condition and all possible combinations using the left hand, right hand, and feet. The proposed multilabel approaches convert the original 8-class problem into a set of three binary problems to facilitate the use of the CSP algorithm. In the case of the MC2CMI method, each binary problem groups together in one class all the MIs engaging one of the three selected body parts, while the rest of MIs that do not engage the same body part are grouped together in the second class. In this way, for each binary problem, the CSP algorithm produces features to determine if the specific body part is engaged in the task or not. Finally, three sets of features are merged together to predict the user intention by applying an 8-class linear discriminant analysis. The MC2SMI method is quite similar, the only difference is that any of the combined MIs is considered during the training phase, which drastically accelerates the calibration time. For all subjects, both the MC2CMI and the MC2SMI approaches reached a higher accuracy than the classic pair-wise (PW) and one-vs.-all (OVA) methods. Our results show that, when brain activity is properly modulated, multilabel approaches represent a very interesting solution to increase the number of commands, and thus to provide a better interaction.

Neural decoding of continuous upper limb movements: a meta-analysis

OBJECTIVE

EEG-based motion trajectory decoding makes a promising approach for neurotechnology which can be used for neural control of motion reconstruction and neurorehabilitation tools. However, the feasibility and validity of continuous motion decoding by non-invasive brain activity are not clear. The main aim of this study was to perform a meta-analysis across studies that examined the ability of EEG-based continuous motion decoding of upper limb movements.

APPROACH

Pearson's correlation coefficient (CC) was used to evaluate the model performance of the studies and considered as an effect size. To estimate the overall effect size of neural decoding of motion trajectory across studies, characteristics of included studies were addressed and the random effect model was applied to the heterogeneous studies which estimated overall effect size distribution. Furthermore, the significant difference between the two subgroups of imagined and executed movements was analysed.

MAIN RESULTS

The mean of the overall effect size was computed 0.46 across the nonhomogeneous studies. The results showed no significant difference between imagined and executed movements (Chi²=0.28, df = 1, p = 0.60).

SIGNIFICANCE

Meta-analysis results confirm that imagination like execution movements can be used for neural decoding of motion trajectory in neural motor control systems. Also, nonlinear compare with linear model statistically confirmed to be more beneficial for complex movements. Furthermore, a new approach of synergy-based motion decoding can be significantly effective to increase model performance and more research needs to evaluate this method for different levels of complexity of movements.IMPLICATIONS FOR REHABILITATIONNeural decoding methods base on EEG as a non-invasive brain activity, are more user friendly for neurorehabilitation than invasive methods that developing of it makes it more applicable for reconstructing activities of daily living.Neurotechnology for neural control of motion reconstruction, makes the rehabilitation tools to be more synchrony with human intentional movement that can be used to improve the brain neuroplastisity in stroke or other paralysed people.The feasibility and validity of imagined movements equal with executed movements show that amputee people also can benefit EEG-based motion decoding for controling rehabilitation tools just by imagination of their intentional movements.For neurorehabilitation tools, comparing the study outcomes illucidate that the approach of synergy-based motor control in brain activities concluded significantly high performance that highlighted the need it to more investigated in future research.

Analyzing and Decoding Natural Reach-and-Grasp Actions Using Gel, Water and Dry EEG Systems

Reaching and grasping is an essential part of everybody's life, it allows meaningful interaction with the environment and is key to independent lifestyle. Recent electroencephalogram (EEG)-based studies have already shown that neural correlates of natural reach-and-grasp actions can be identified in the EEG. However, it is still in question whether these results obtained in a laboratory environment can make the transition to mobile applicable EEG systems for home use. In the current study, we investigated whether EEG-based correlates of natural reach-and-grasp actions can be successfully identified and decoded using mobile EEG systems, namely the water-based EEG-Versatile ^TM system and the dry-electrodes EEG-Hero ^TM headset. In addition, we also analyzed gel-based recordings obtained in a laboratory environment (g.USBamp/g.Ladybird, gold standard), which followed the same experimental parameters. For each recording system, 15 study participants performed 80 self-initiated reach-and-grasp actions toward a glass (palmar grasp) and a spoon (lateral grasp). Our results confirmed that EEG-based correlates of reach-and-grasp actions can be successfully identified using these mobile systems. In a single-trial multiclass-based decoding approach, which incorporated both movement conditions and rest, we could show that the low frequency time domain (LFTD) correlates were also decodable. Grand average peak accuracy calculated on unseen test data yielded for the water-based electrode system 62.3% (9.2% STD), whereas for the dry-electrodes headset reached 56.4% (8% STD). For the gel-based electrode system 61.3% (8.6% STD) could be achieved. To foster and promote further investigations in the field of EEG-based movement decoding, as well as to allow the interested community to make their own conclusions, we provide all datasets publicly available in the BNCI Horizon 2020 database (http://bnci-horizon-2020.eu/database/data-sets).

Neural coordination of bilateral power and precision finger movements

The dexterity of hands and fingers is related to the strength of control by cortico-motoneuronal connections which exclusively exist in primates. The cortical command is associated with a task-specific, rapid proprioceptive adaptation of forces applied by hands and fingers to an object. This neural control differs between "power grip" movements (e.g., reach and grasp of a cup) where hand and fingers act as a unity and "precision grip" movements (e.g., picking up a raspberry) where fingers move independently from the hand. In motor tasks requiring hands and fingers of both sides a "neural coupling" (reflected in bilateral reflex responses to unilateral stimulations) coordinates power grip movements (e.g., opening a bottle). In contrast, during bilateral precision movements, such as playing piano, the fingers of both hands move independently, due to a direct cortico-motoneuronal control, while the hands are coupled (e.g., to maintain the rhythm between the two sides). While most studies on prehension concern unilateral hand movements, many activities of daily life are tackled by bilateral power grips where a neural coupling serves for an automatic movement performance. In primates this mode of motor control is supplemented by a system that enables the uni- or bilateral performance of skilled individual finger movements.

Non-linear online low-frequency EEG decoding of arm movements during a pursuit tracking task

Decoding upper-limb movements in invasive recordings has become a reality, but neural tuning in non-invasive low-frequency recordings is still under discussion. Recent studies managed to decode movement positions and velocities using linear decoders, even developing an online system. The decoded signals, however, exhibited smaller amplitudes than actual movements, affecting feedback and user experience. Recently, we showed that a non-linear offline decoder can combine directional (e.g., velocity) and non-directional (e.g., speed) information. In this study, it is assessed if the non-linear decoder can be used online to provide real-time feedback. Five healthy subjects were asked to track a moving target by controlling a robotic arm. Initially, the robot was controlled by their right hand; then, the control was gradually switched until it was entirely controlled by the electroencephalogram (EEG). Correlations between actual and decoded movements were generally above chance level. Results suggest that information about speed was also encoded in the EEG, demonstrating that the proposed non-linear decoder is suitable for decoding real-time arm movements.

Applying intuitive EEG-controlled grasp neuroprostheses in individuals with spinal cord injury: Preliminary results from the MoreGrasp clinical feasibility study

The aim of the MoreGrasp project is to develop a non-invasive, multimodal user interface including a brain-computer interface (BCI) for control of a grasp neuroprostheses in individuals with high spinal cord injury (SCI). The first results of the ongoing MoreGrasp clinical feasibility study involving end users with SCI are presented. This includes BCI screening sessions, in which we investigate the electroencephalography (EEG) patterns associated with single, natural movements of the upper limb. These patterns will later be used to control the neuroprosthesis. Additionally, the MoreGrasp grasp neuroprosthesis consisting of electrode arrays embedded in an individualized textile forearm sleeve is presented. The general feasibility of this electrode array in terms of corrections of misalignments during donning is shown together with the functional results in end users of the electrode forearm sleeve.

Electroencephalography

The electroencephalogram (EEG) was invented almost 100 years ago and is still a method of choice for many research questions, even applications-from functional brain imaging in neuroscientific investigations during movement to real-time applications like brain-computer interfacing. This chapter gives some background information on the establishment and properties of the EEG. This chapter starts with a closer look at the sources of EEG at a micro or neuronal level, followed by recording techniques, types of electrodes, and common EEG artifacts. Then an overview on EEG phenomena, namely, spontaneous EEG and event-related potentials build the middle part of this chapter. The last part discusses brain signals, which are used in current BCI research, including short descriptions and examples of applications.

Unimanual and Bimanual Reach-and-Grasp Actions Can Be Decoded From Human EEG

While most tasks of daily life can be handled through a small number of different grasps, many tasks require the action of both hands. In these bimanual tasks, the second hand has either a supporting role (e.g. for fixating a jar) or a more active role (e.g. grasping a pot on both handles). In this study we attempt to discriminate the neural correlates of unimanual (performed with left and right hand) from bimanual reach-and-grasp actions using the low-frequency time-domain electroencephalogram (EEG). In a self-initiated movement task, 15 healthy participants were asked to perform unimanual (palmar and lateral grasps with left and right hand) and bimanual (double lateral, mixed palmar/lateral) reach-and-grasps on objects of daily life. Using EEG time-domain features in the frequency range of 0.3-3 Hz, we achieved multiclass-classification accuracies of 38.6 ± 6.6% (7 classes, 17.1% chance level) for a combination of 6 movements and 1 rest condition. The grand average confusion matrix shows highest true positive rates (TPR) for the rest (63%) condition while TPR for the movement classes varied between 33 to 41%. The underlying movement-related cortical potentials (MRCPs) show significant differences between unimanual (e.g left hand vs. right hand grasps) as well unimanual vs. bimanual conditions which both can be attributed to lateralization effects. We believe that these findings can be exploited and further used for attempts in providing persons with spinal cord injury a form of natural control for bimanual neuroprostheses.

Direct comparison of supervised and semi-supervised retraining approaches for co-adaptive BCIs

For Brain-Computer interfaces (BCIs), system calibration is a lengthy but necessary process for successful operation. Co-adaptive BCIs aim to shorten training and imply positive motivation to users by presenting feedback already at early stages: After just 5 min of gathering calibration data, the systems are able to provide feedback and engage users in a mutual learning process. In this work, we investigate whether the retraining stage of co-adaptive BCIs can be adapted to a semi-supervised concept, where only a small amount of labeled data is available and all additional data needs to be labeled by the BCI itself. The aim of the current work was to evaluate whether a semi-supervised co-adaptive BCI could successfully compete with a supervised co-adaptive BCI model. In a supporting two-class (190 trials per condition) BCI study based on motor imagery tasks, we evaluated both approaches in two separate groups of 10 participants online, while we simulated the other approach in each group offline. Our results indicate that despite the lack of true labeled data, the semi-supervised driven BCI did not perform significantly worse (p > 0.05) than the supervised counterpart. We believe that these findings contribute to developing BCIs for long-term use, where continuous adaptation becomes imperative for maintaining meaningful BCI performance.

Graphical abstract

Combining frequency and time-domain EEG features for classification of self-paced reach-and-grasp actions

Brain-computer interfaces (BCIs) might provide an intuitive way for severely motor impaired persons to operate assistive devices to perform daily life activities. Recent studies have shown that complex hand movements, such as reach-and-grasp tasks, can be decoded from the low frequency of the electroencephalogram (EEG). In this work we investigated whether additional features extracted from the frequency-domain of alpha and beta bands could improve classification performance of rest vs. palmar vs. lateral grasp. We analysed two multi-class classification approaches, the first using features from the low frequency time-domain, and the second in which we combined the time-domain with frequency-domain features from alpha and beta bands. We measured EEG of ten participants without motor disability which performed self-paced reach-and-grasp actions on objects of daily life. For the time-domain classification approach, participants reached an average peak accuracy of 65%. For the combined approach, an average peak accuracy of 75% was reached. In both approaches and for all subjects, performance was significantly higher than chance level (38.1%, 3-class scenario). By computing the confusion matrices as well as feature rankings through the Fisher score, we show that movement vs. rest classification performance increased considerably in the combined approach and was the main responsible for the multi-class higher performance. These findings could help the development of BCIs in real-life scenarios, where decreasing false movement detections could drastically increase the end-user acceptance and usability of BCIs.

Decoding Voluntary Movement of Single Hand Based on Analysis of Brain Connectivity by Using EEG Signals

Research about decoding neurophysiological signals mainly aims to elucidate the details of human motion control from the perspective of neural activity. We performed brain connectivity analysis with EEG to propose a brain functional network (BFN) and used a feature extraction algorithm for decoding the voluntary hand movement of a subject. By analyzing the characteristic parameters obtained from the BFN, we extracted the most important electrode nodes and frequencies for identifying the direction of movement of a hand. The results demonstrated that the most sensitive EEG components were for frequencies delta, theta, and gamma1 from electrodes F4, F8, C3, Cz, C4, CP4, T3, and T4. Finally, we proposed a model for decoding voluntary movement of the right hand by using a hierarchical linear model (HLM). Through a voluntary hand movement experiment in a spiral trajectory, the Poisson coefficient between the measurement trajectory and the decoding trajectory was used as a test standard to compare the HLM with the traditional multiple linear regression model. It was found that the decoding model based on the HLM obtained superior results. This paper contributes a feature extraction method based on brain connectivity analysis that can mine more comprehensive feature information related to a specific mental state of a subject. The decoding model based on the HLM possesses a strong structure for data manipulation that facilitates precise decoding.

EEG patterns of self-paced movement imaginations towards externally-cued and internally-selected targets

In this study, we investigate the neurophysiological signature of the interacting processes which lead to a single reach-and-grasp movement imagination (MI). While performing this task, the human healthy participants could either define their movement targets according to an external cue, or through an internal selection process. After defining their target, they could start the MI whenever they wanted. We recorded high density electroencephalographic (EEG) activity and investigated two neural correlates: the event-related potentials (ERPs) associated with the target selection, which reflect the perceptual and cognitive processes prior to the MI, and the movement-related cortical potentials (MRCPs), associated with the planning of the self-paced MI. We found differences in frontal and parietal areas between the late ERP components related to the internally-driven selection and the externally-cued process. Furthermore, we could reliably estimate the MI onset of the self-paced task. Next, we extracted MRCP features around the MI onset to train classifiers of movement vs. rest directly on self-paced MI data. We attained performance significantly higher than chance level for both time-locked and asynchronous classification. These findings contribute to the development of more intuitive brain-computer interfaces in which movement targets are defined internally and the movements are self-paced.

Utilizing sensory prediction errors for movement intention decoding: A new methodology

We propose a new methodology for decoding movement intentions of humans. This methodology is motivated by the well-documented ability of the brain to predict sensory outcomes of self-generated and imagined actions using so-called forward models. We propose to subliminally stimulate the sensory modality corresponding to a user's intended movement, and decode a user's movement intention from his electroencephalography (EEG), by decoding for prediction errors-whether the sensory prediction corresponding to a user's intended movement matches the subliminal sensory stimulation we induce. We tested our proposal in a binary wheelchair turning task in which users thought of turning their wheelchair either left or right. We stimulated their vestibular system subliminally, toward either the left or the right direction, using a galvanic vestibular stimulator and show that the decoding for prediction errors from the EEG can radically improve movement intention decoding performance. We observed an 87.2% median single-trial decoding accuracy across tested participants, with zero user training, within 96 ms of the stimulation, and with no additional cognitive load on the users because the stimulation was subliminal.

Decoding Imagined 3D Hand Movement Trajectories From EEG: Evidence to Support the Use of Mu, Beta, and Low Gamma Oscillations

Objective: To date, motion trajectory prediction (MTP) of a limb from non-invasive electroencephalography (EEG) has relied, primarily, on band-pass filtered samples of EEG potentials i.e., the potential time-series model. Most MTP studies involve decoding 2D and 3D arm movements i.e., executed arm movements. Decoding of observed or imagined 3D movements has been demonstrated with limited success and only reported in a few studies. MTP studies normally use EEG potentials filtered in the low delta (~1 Hz) band for reconstructing the trajectory of an executed or an imagined/observed movement. In contrast to MTP, multiclass classification based sensorimotor rhythm brain-computer interfaces aim to classify movements using the power spectral density of mu (8–12 Hz) and beta (12–28 Hz) bands.

Approach: We investigated if replacing the standard potentials time-series input with a power spectral density based bandpower time-series improves trajectory decoding accuracy of kinesthetically imagined 3D hand movement tasks (i.e., imagined 3D trajectory of the hand joint) and whether imagined 3D hand movements kinematics are encoded also in mu and beta bands. Twelve naïve subjects were asked to generate or imagine generating pointing movements with their right dominant arm to four targets distributed in 3D space in synchrony with an auditory cue (beep).

Main results: Using the bandpower time-series based model, the highest decoding accuracy for motor execution was observed in mu and beta bands whilst for imagined movements the low gamma (28–40 Hz) band was also observed to improve decoding accuracy for some subjects. Moreover, for both (executed and imagined) movements, the bandpower time-series model with mu, beta, and low gamma bands produced significantly higher reconstruction accuracy than the commonly used potential time-series model and delta oscillations.

Significance: Contrary to many studies that investigated only executed hand movements and recommend using delta oscillations for decoding directional information of a single limb joint, our findings suggest that motor kinematics for imagined movements are reflected mostly in power spectral density of mu, beta and low gamma bands, and that these bands may be most informative for decoding 3D trajectories of imagined limb movements.

Hierarchical decoding of grasping commands from EEG

Brain-Computer Interfaces may present an intuitive way for motor impaired end users to operate assistive devices of daily life. Recent studies showed that complex kinematics like grasping can be successfully decoded from low frequency electroencephalogram. In this work we present a hierarchical method to asynchronously discriminate two different grasps often used in daily life actions (palmar, pincer) from a combined set of motor execution and motor intention. We compared sensorimotor rhythms based features and time features from the low frequency spectrum for best discrimination results. Our results show not only the principle feasibility of the proposed method with detection of asynchronous motor intention at rates of 80% accuracy and subsequent grasping discrimination over 60%, but also that low frequency time domain features provide a more consistent detection pattern. Although the basis of this results is still an off-line analysis we are confident that these results can be transferred to on-line use.

EEG neural correlates of goal-directed movement intention

Using low-frequency time-domain electroencephalographic (EEG) signals we show, for the same type of upper limb movement, that goal-directed movements have different neural correlates than movements without a particular goal. In a reach-and-touch task, we explored the differences in the movement-related cortical potentials (MRCPs) between goal-directed and non-goal-directed movements. We evaluated if the detection of movement intention was influenced by the goal-directedness of the movement. In a single-trial classification procedure we found that classification accuracies are enhanced if there is a goal-directed movement in mind. Furthermore, by using the classifier patterns and estimating the corresponding brain sources, we show the importance of motor areas and the additional involvement of the posterior parietal lobule in the discrimination between goal-directed movements and non-goal-directed movements. We discuss next the potential contribution of our results on goal-directed movements to a more reliable brain-computer interface (BCI) control that facilitates recovery in spinal-cord injured or stroke end-users.