Decoding natural grasp types from human ECoG

Decoding individual finger movements from one hand using human EEG signals

Brain computer interface (BCI) is an assistive technology, which decodes neurophysiological signals generated by the human brain and translates them into control signals to control external devices, e.g., wheelchairs. One problem challenging noninvasive BCI technologies is the limited control dimensions from decoding movements of, mainly, large body parts, e.g., upper and lower limbs. It has been reported that complicated dexterous functions, i.e., finger movements, can be decoded in electrocorticography (ECoG) signals, while it remains unclear whether noninvasive electroencephalography (EEG) signals also have sufficient information to decode the same type of movements. Phenomena of broadband power increase and low-frequency-band power decrease were observed in EEG in the present study, when EEG power spectra were decomposed by a principal component analysis (PCA). These movement-related spectral structures and their changes caused by finger movements in EEG are consistent with observations in previous ECoG study, as well as the results from ECoG data in the present study. The average decoding accuracy of 77.11% over all subjects was obtained in classifying each pair of fingers from one hand using movement-related spectral changes as features to be decoded using a support vector machine (SVM) classifier. The average decoding accuracy in three epilepsy patients using ECoG data was 91.28% with the similarly obtained features and same classifier. Both decoding accuracies of EEG and ECoG are significantly higher than the empirical guessing level (51.26%) in all subjects (p<0.05). The present study suggests the similar movement-related spectral changes in EEG as in ECoG, and demonstrates the feasibility of discriminating finger movements from one hand using EEG. These findings are promising to facilitate the development of BCIs with rich control signals using noninvasive technologies.

Prediction of hand trajectory from electrocorticography signals in primary motor cortex

Due to their potential as a control modality in brain-machine interfaces, electrocorticography (ECoG) has received much focus in recent years. Studies using ECoG have come out with success in such endeavors as classification of arm movements and natural grasp types, regression of arm trajectories in two and three dimensions, estimation of muscle activity time series and so on. However, there still remains considerable work to be done before a high performance ECoG-based neural prosthetic can be realized. In this study, we proposed an algorithm to decode hand trajectory from 15 and 32 channel ECoG signals recorded from primary motor cortex (M1) in two primates. To determine the most effective areas for prediction, we applied two electrode selection methods, one based on position relative to the central sulcus (CS) and another based on the electrodes' individual prediction performance. The best coefficients of determination for decoding hand trajectory in the two monkeys were 0.4815 ± 0.0167 and 0.7780 ± 0.0164. Performance results from individual ECoG electrodes showed that those with higher performance were concentrated at the lateral areas and areas close to the CS. The results of prediction according with different numbers of electrodes based on proposed methods were also shown and discussed. These results also suggest that superior decoding performance can be achieved from a group of effective ECoG signals rather than an entire ECoG array.

Gamma band activity associated with BCI performance: simultaneous MEG/EEG study

While brain computer interface (BCI) can be employed with patients and healthy subjects, there are problems that must be resolved before BCI can be useful to the public. In the most popular motor imagery (MI) BCI system, a significant number of target users (called “BCI-Illiterates”) cannot modulate their neuronal signals sufficiently to use the BCI system. This causes performance variability among subjects and even among sessions within a subject. The mechanism of such BCI-Illiteracy and possible solutions still remain to be determined. Gamma oscillation is known to be involved in various fundamental brain functions, and may play a role in MI. In this study, we investigated the association of gamma activity with MI performance among subjects. Ten simultaneous MEG/EEG experiments were conducted; MI performance for each was estimated by EEG data, and the gamma activity associated with BCI performance was investigated with MEG data. Our results showed that gamma activity had a high positive correlation with MI performance in the prefrontal area. This trend was also found across sessions within one subject. In conclusion, gamma rhythms generated in the prefrontal area appear to play a critical role in BCI performance.

The role of ECoG magnitude and phase in decoding position, velocity, and acceleration during continuous motor behavior

In neuronal population signals, including the electroencephalogram (EEG) and electrocorticogram (ECoG), the low-frequency component (LFC) is particularly informative about motor behavior and can be used for decoding movement parameters for brain-machine interface (BMI) applications. An idea previously expressed, but as of yet not quantitatively tested, is that it is the LFC phase that is the main source of decodable information. To test this issue, we analyzed human ECoG recorded during a game-like, one-dimensional, continuous motor task with a novel decoding method suitable for unfolding magnitude and phase explicitly into a complex-valued, time-frequency signal representation, enabling quantification of the decodable information within the temporal, spatial and frequency domains and allowing disambiguation of the phase contribution from that of the spectral magnitude. The decoding accuracy based only on phase information was substantially (at least 2 fold) and significantly higher than that based only on magnitudes for position, velocity and acceleration. The frequency profile of movement-related information in the ECoG data matched well with the frequency profile expected when assuming a close time-domain correlate of movement velocity in the ECoG, e.g., a (noisy) "copy" of hand velocity. No such match was observed with the frequency profiles expected when assuming a copy of either hand position or acceleration. There was also no indication of additional magnitude-based mechanisms encoding movement information in the LFC range. Thus, our study contributes to elucidating the nature of the informative LFC of motor cortical population activity and may hence contribute to improve decoding strategies and BMI performance.

Simultaneous neural control of simple reaching and grasping with the modular prosthetic limb using intracranial EEG

Intracranial electroencephalographic (iEEG) signals from two human subjects were used to achieve simultaneous neural control of reaching and grasping movements with the Johns Hopkins University Applied Physics Lab (JHU/APL) Modular Prosthetic Limb (MPL), a dexterous robotic prosthetic arm. We performed functional mapping of high gamma activity while the subject made reaching and grasping movements to identify task-selective electrodes. Independent, online control of reaching and grasping was then achieved using high gamma activity from a small subset of electrodes with a model trained on short blocks of reaching and grasping with no further adaptation. Classification accuracy did not decline (p < 0.05, one-way ANOVA) over three blocks of testing in either subject. Mean classification accuracy during independently executed overt reach and grasp movements for (Subject 1, Subject 2) were (0.85, 0.81) and (0.80, 0.96), respectively, and during simultaneous execution they were (0.83, 0.88) and (0.58, 0.88), respectively. Our models leveraged knowledge of the subject's individual functional neuroanatomy for reaching and grasping movements, allowing rapid acquisition of control in a time-sensitive clinical setting. We demonstrate the potential feasibility of verifying functionally meaningful iEEG-based control of the MPL prior to chronic implantation, during which additional capabilities of the MPL might be exploited with further training.

Current challenges to the clinical translation of brain machine interface technology

Development of neural prostheses over the past few decades has produced a number of clinically relevant brain-machine interfaces (BMIs), such as the cochlear prostheses and deep brain stimulators. Current research pursues the restoration of communication or motor function to individuals with neurological disorders. Efforts in the field, such as the BrainGate trials, have already demonstrated that such interfaces can enable humans to effectively control external devices with neural signals. However, a number of significant issues regarding BMI performance, device capabilities, and surgery must be resolved before clinical use of BMI technology can become widespread. This chapter reviews challenges to clinical translation and discusses potential solutions that have been reported in recent literature, with focuses on hardware reliability, state-of-the-art decoding algorithms, and surgical considerations during implantation.

Prediction of three-dimensional arm trajectories based on ECoG signals recorded from human sensorimotor cortex

Brain-machine interface techniques have been applied in a number of studies to control neuromotor prostheses and for neurorehabilitation in the hopes of providing a means to restore lost motor function. Electrocorticography (ECoG) has seen recent use in this regard because it offers a higher spatiotemporal resolution than non-invasive EEG and is less invasive than intracortical microelectrodes. Although several studies have already succeeded in the inference of computer cursor trajectories and finger flexions using human ECoG signals, precise three-dimensional (3D) trajectory reconstruction for a human limb from ECoG has not yet been achieved. In this study, we predicted 3D arm trajectories in time series from ECoG signals in humans using a novel preprocessing method and a sparse linear regression. Average Pearson’s correlation coefficients and normalized root-mean-square errors between predicted and actual trajectories were 0.44∼0.73 and 0.18∼0.42, respectively, confirming the feasibility of predicting 3D arm trajectories from ECoG. We foresee this method contributing to future advancements in neuroprosthesis and neurorehabilitation technology.

Neuroprosthetic technology for individuals with spinal cord injury

CONTEXT

Spinal cord injury (SCI) results in a loss of function and sensation below the level of the lesion. Neuroprosthetic technology has been developed to help restore motor and autonomic functions as well as to provide sensory feedback.

FINDINGS

This paper provides an overview of neuroprosthetic technology that aims to address the priorities for functional restoration as defined by individuals with SCI. We describe neuroprostheses that are in various stages of preclinical development, clinical testing, and commercialization including functional electrical stimulators, epidural and intraspinal microstimulation, bladder neuroprosthesis, and cortical stimulation for restoring sensation. We also discuss neural recording technologies that may provide command or feedback signals for neuroprosthetic devices.

CONCLUSION/CLINICAL RELEVANCE

Neuroprostheses have begun to address the priorities of individuals with SCI, although there remains room for improvement. In addition to continued technological improvements, closing the loop between the technology and the user may help provide intuitive device control with high levels of performance.

Stereoelectroencephalography for continuous two-dimensional cursor control in a brain-machine interface

Stereoelectroencephalography (SEEG) is becoming more prevalent as a planning tool for surgical treatment of intractable epilepsy. Stereoelectroencephalography uses long, thin, cylindrical "depth" electrodes containing multiple recording contacts along each electrode's length. Each lead is inserted into the brain percutaneously. The advantage of SEEG is that the electrodes can easily target deeper brain structures that are inaccessible with subdural grid electrodes, and SEEG does not require a craniotomy. Brain-machine interface (BMI) research is also becoming more common in the Epilepsy Monitoring Unit. A brain-machine interface decodes a person's desired movement or action from the recorded brain activity and then uses the decoded brain activity to control an assistive device in real time. Although BMIs are primarily being developed for use by severely paralyzed individuals, epilepsy patients undergoing invasive brain monitoring provide an opportunity to test the effectiveness of different invasive recording electrodes for use in BMI systems. This study investigated the ability to use SEEG electrodes for control of 2D cursor velocity in a BMI. Two patients who were undergoing SEEG for intractable epilepsy participated in this study. Participants were instructed to wiggle or rest the hand contralateral to their SEEG electrodes to control the horizontal velocity of a cursor on a screen. Simultaneously they were instructed to wiggle or rest their feet to control the vertical component of cursor velocity. The BMI system was designed to detect power spectral changes associated with hand and foot activity and translate those spectral changes into horizontal and vertical cursor movements in real time. During testing, participants used their decoded SEEG signals to move the brain-controlled cursor to radial targets that appeared on the screen. Although power spectral information from 28 to 32 electrode contacts were used for cursor control during the experiment, post hoc analysis indicated that better control may have been possible using only a single SEEG depth electrode containing multiple recording contacts in both hand and foot cortical areas. These results suggest that the advantages of using SEEG for epilepsy monitoring may also apply to using SEEG electrodes in BMI systems. Specifically, SEEG electrodes can target deeper brain structures, such as foot motor cortex, and both hand and foot areas can be targeted with a single SEEG electrode implanted percutaneously. Therefore, SEEG electrodes may be an attractive option for simple BMI systems that use power spectral modulation in hand and foot cortex for independent control of 2 degrees of freedom.

Modulations of EEG beta power during planning and execution of grasping movements

Although beta oscillations (≈ 13–35 Hz) are often considered as a sensorimotor rhythm, their functional role remains debated. In particular, the modulations of beta power during preparation and execution of complex movements in different contexts were barely investigated. Here, we analysed the beta oscillations recorded with electroencephalography (EEG) in a precued grasping task in which we manipulated two critical parameters: the grip type (precision vs. side grip) and the force (high vs. low force) required to pull an object along a horizontal axis. A cue was presented 3 s before a GO signal and provided full, partial or no information about the two movement parameters. We measured beta power over the centro-parietal areas during movement preparation and execution as well as during object hold. We explored the modulations of power in relation to the amount and type of prior information provided by the cue. We also investigated how beta power was affected by the grip and force parameters.

We observed an increase in beta power around the cue onset followed by a decrease during movement preparation and execution. These modulations were followed by a transient power increase during object hold. This pattern of modulations did not differ between the 4 movement types (2 grips ×2 forces). However, the amount and type of prior information provided by the cue had a significant effect on the beta power during the preparatory delay. We discuss how these results fit with current hypotheses on the functional role of beta oscillations.

Detection of error related neuronal responses recorded by electrocorticography in humans during continuous movements

Background

Brain-machine interfaces (BMIs) can translate the neuronal activity underlying a user’s movement intention into movements of an artificial effector. In spite of continuous improvements, errors in movement decoding are still a major problem of current BMI systems. If the difference between the decoded and intended movements becomes noticeable, it may lead to an execution error. Outcome errors, where subjects fail to reach a certain movement goal, are also present during online BMI operation. Detecting such errors can be beneficial for BMI operation: (i) errors can be corrected online after being detected and (ii) adaptive BMI decoding algorithm can be updated to make fewer errors in the future.

Methodology/Principal Findings

Here, we show that error events can be detected from human electrocorticography (ECoG) during a continuous task with high precision, given a temporal tolerance of 300–400 milliseconds. We quantified the error detection accuracy and showed that, using only a small subset of 2×2 ECoG electrodes, 82% of detection information for outcome error and 74% of detection information for execution error available from all ECoG electrodes could be retained.

Conclusions/Significance

The error detection method presented here could be used to correct errors made during BMI operation or to adapt a BMI algorithm to make fewer errors in the future. Furthermore, our results indicate that smaller ECoG implant could be used for error detection. Reducing the size of an ECoG electrode implant used for BMI decoding and error detection could significantly reduce the medical risk of implantation.

Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas

OBJECTIVE

Brain-machine interface systems translate recorded neural signals into command signals for assistive technology. In individuals with upper limb amputation or cervical spinal cord injury, the restoration of a useful hand grasp could significantly improve daily function. We sought to determine if electrocorticographic (ECoG) signals contain sufficient information to select among multiple hand postures for a prosthetic hand, orthotic, or functional electrical stimulation system.

APPROACH

We recorded ECoG signals from subdural macro- and microelectrodes implanted in motor areas of three participants who were undergoing inpatient monitoring for diagnosis and treatment of intractable epilepsy. Participants performed five distinct isometric hand postures, as well as four distinct finger movements. Several control experiments were attempted in order to remove sensory information from the classification results. Online experiments were performed with two participants.

MAIN RESULTS

Classification rates were 68%, 84% and 81% for correct identification of 5 isometric hand postures offline. Using 3 potential controls for removing sensory signals, error rates were approximately doubled on average (2.1×). A similar increase in errors (2.6×) was noted when the participant was asked to make simultaneous wrist movements along with the hand postures. In online experiments, fist versus rest was successfully classified on 97% of trials; the classification output drove a prosthetic hand. Online classification performance for a larger number of hand postures remained above chance, but substantially below offline performance. In addition, the long integration windows used would preclude the use of decoded signals for control of a BCI system.

SIGNIFICANCE

These results suggest that ECoG is a plausible source of command signals for prosthetic grasp selection. Overall, avenues remain for improvement through better electrode designs and placement, better participant training, and characterization of non-stationarities such that ECoG could be a viable signal source for grasp control for amputees or individuals with paralysis.

Grasp detection from human ECoG during natural reach-to-grasp movements

Various movement parameters of grasping movements, like velocity or type of the grasp, have been successfully decoded from neural activity. However, the question of movement event detection from brain activity, that is, decoding the time at which an event occurred (e.g. movement onset), has been addressed less often. Yet, this may be a topic of key importance, as a brain-machine interface (BMI) that controls a grasping prosthesis could be realized by detecting the time of grasp, together with an optional decoding of which type of grasp to apply. We, therefore, studied the detection of time of grasps from human ECoG recordings during a sequence of natural and continuous reach-to-grasp movements. Using signals recorded from the motor cortex, a detector based on regularized linear discriminant analysis was able to retrieve the time-point of grasp with high reliability and only few false detections. Best performance was achieved using a combination of signal components from time and frequency domains. Sensitivity, measured by the amount of correct detections, and specificity, represented by the amount of false detections, depended strongly on the imposed restrictions on temporal precision of detection and on the delay between event detection and the time the event occurred. Including neural data from after the event into the decoding analysis, slightly increased accuracy, however, reasonable performance could also be obtained when grasping events were detected 125 ms in advance. In summary, our results provide a good basis for using detection of grasping movements from ECoG to control a grasping prosthesis.

Continuous decoding of grasping tasks for a prospective implantable cortical neuroprosthesis

Background

In the recent past several invasive cortical neuroprostheses have been developed. Signals recorded from the motor cortex (area MI) have been decoded and used to control computer cursors and robotic devices. Nevertheless, few attempts have been carried out to predict different grips.

A Support Vector Machines (SVMs) classifier has been trained for a continuous decoding of four/six grip types using signals recorded in two monkeys from motor neurons of the ventral premotor cortex (area F5) during a reach-to-grasp task.

Findings

The results showed that four/six grip types could be extracted with classification accuracy higher than 96% using window width of 75–150 ms.

Conclusions

These results open new and promising possibilities for the development of invasive cortical neural prostheses for the control of reaching and grasping.

fNIRS exhibits weak tuning to hand movement direction

Functional near-infrared spectroscopy (fNIRS) has become an established tool to investigate brain function and is, due to its portability and resistance to electromagnetic noise, an interesting modality for brain-machine interfaces (BMIs). BMIs have been successfully realized using the decoding of movement kinematics from intra-cortical recordings in monkey and human. Recently, it has been shown that hemodynamic brain responses as measured by fMRI are modulated by the direction of hand movements. However, quantitative data on the decoding of movement direction from hemodynamic responses is still lacking and it remains unclear whether this can be achieved with fNIRS, which records signals at a lower spatial resolution but with the advantage of being portable. Here, we recorded brain activity with fNIRS above different cortical areas while subjects performed hand movements in two different directions. We found that hemodynamic signals in contralateral sensorimotor areas vary with the direction of movements, though only weakly. Using these signals, movement direction could be inferred on a single-trial basis with an accuracy of ∼65% on average across subjects. The temporal evolution of decoding accuracy resembled that of typical hemodynamic responses observed in motor experiments. Simultaneous recordings with a head tracking system showed that head movements, at least up to some extent, do not influence the decoding of fNIRS signals. Due to the low accuracy, fNIRS is not a viable alternative for BMIs utilizing decoding of movement direction. However, due to its relative resistance to head movements, it is promising for studies investigating brain activity during motor experiments.

Prediction of muscle activities from electrocorticograms in primary motor cortex of primates

Electrocorticography (ECoG) has drawn attention as an effective recording approach for brain-machine interfaces (BMI). Previous studies have succeeded in classifying movement intention and predicting hand trajectories from ECoG. Despite such successes, however, there still remains considerable work for the realization of ECoG-based BMIs as neuroprosthetics. We developed a method to predict multiple muscle activities from ECoG measurements. We also verified that ECoG signals are effective for predicting muscle activities in time varying series when performing sequential movements. ECoG signals were band-pass filtered into separate sensorimotor rhythm bands, z-score normalized, and smoothed with a Gaussian filter. We used sparse linear regression to find the best fit between frequency bands of ECoG and electromyographic activity. The best average correlation coefficient and the normalized root-mean-square error were 0.92±0.06 and 0.06±0.10, respectively, in the flexor digitorum profundus finger muscle. The δ (1.5∼4Hz) and γ2 (50∼90Hz) bands contributed significantly more strongly than other frequency bands (P<0.001). These results demonstrate the feasibility of predicting muscle activity from ECoG signals in an online fashion.

The ups and downs of β oscillations in sensorimotor cortex

Since the first descriptions of sensorimotor rhythms by Berger (1929) and by Jasper and Penfield (1949), the potential role of beta oscillations (~13-30 Hz) in the brain has been intensely investigated. We start this review by showing that experimental studies in humans and monkeys have reached a consensus on the facts that sensorimotor beta power is low during movement, transiently increases after movement end (the "beta rebound") and tonically increases during object grasping. Recently, a new surge of studies exploiting more complex sensorimotor tasks including multiple events, such as instructed delay tasks, reveal novel characteristics of beta oscillatory activity. We therefore proceed by critically reviewing also this literature to understand whether modulations of beta oscillations in task epochs other than those during and after movement are consistent across studies, and whether they can be reconciled with a role for beta oscillations in sensorimotor transmission. We indeed find that there are additional processes that also strongly affect sensorimotor beta oscillations, such as visual cue anticipation and processing, fitting with the view that beta oscillations reflect heightened sensorimotor transmission beyond somatosensation. However, there are differences among studies, which may be interpreted more readily if we assume multiple processes, whose effects on the overall measured beta power overlap in time. We conclude that beta oscillations observed in sensorimotor cortex may serve large-scale communication between sensorimotor and other areas and the periphery.

An online brain-machine interface using decoding of movement direction from the human electrocorticogram

A brain-machine interface (BMI) can be used to control movements of an artificial effector, e.g. movements of an arm prosthesis, by motor cortical signals that control the equivalent movements of the corresponding body part, e.g. arm movements. This approach has been successfully applied in monkeys and humans by accurately extracting parameters of movements from the spiking activity of multiple single neurons. We show that the same approach can be realized using brain activity measured directly from the surface of the human cortex using electrocorticography (ECoG). Five subjects, implanted with ECoG implants for the purpose of epilepsy assessment, took part in our study. Subjects used directionally dependent ECoG signals, recorded during active movements of a single arm, to control a computer cursor in one out of two directions. Significant BMI control was achieved in four out of five subjects with correct directional decoding in 69%-86% of the trials (75% on average). Our results demonstrate the feasibility of an online BMI using decoding of movement direction from human ECoG signals. Thus, to achieve such BMIs, ECoG signals might be used in conjunction with or as an alternative to intracortical neural signals.

Reaching movement onset- and end-related characteristics of EEG spectral power modulations

The spectral power of intracranial field potentials shows movement-related modulations during reaching movements to different target positions that in frequencies up to the high-γ range (approximately 50 to above 200 Hz) can be reliably used for single-trial inference of movement parameters. However, identifying spectral power modulations suitable for single-trial analysis for non-invasive approaches remains a challenge. We recorded non-invasive electroencephalography (EEG) during a self-paced center-out and center-in arm movement task, resulting in eight reaching movement classes (four center-out, four center-in). We found distinct slow (≤5 Hz), μ (7.5–10 Hz), β (12.5–25 Hz), low-γ (approximately 27.5–50 Hz), and high-γ (above 50 Hz) movement onset- and end-related responses. Movement class-specific spectral power modulations were restricted to the β band at approximately 1 s after movement end and could be explained by the sensitivity of this response to different static, post-movement electromyography (EMG) levels. Based on the β band, significant single-trial inference of reaching movement endpoints was possible. The findings of the present study support the idea that single-trial decoding of different reaching movements from non-invasive EEG spectral power modulations is possible, but also suggest that the informative time window is after movement end and that the informative frequency range is restricted to the β band.