Give me a sign: decoding four complex hand gestures based on high-density ECoG

Decoding sign language finger flexions from high-density electrocorticography using graph-optimized block term tensor regression

Objective.A novel method is introduced to regress over the sign language finger movements from human electrocorticography (ECoG) recordings.Approach.The proposed graph-optimized block-term tensor regression (Go-BTTR) method consists of two components: a deflation-based regression model that sequentially Tucker-decomposes multiway ECoG data into a series of blocks, and a causal graph process (CGP) that accounts for the complex relationship between finger movements when expressing sign language gestures. Prior to each regression block, CGP is applied to decide which fingers should be kept separate or grouped and should therefore be referred to BTTR or its extended version eBTTR, respectively.Main results.Two ECoG datasets were used, one recorded in five patients expressing four hand gestures of the American sign language alphabet, and another in two patients expressing all gestures of the Flemish sign language alphabet. As Go-BTTR combines fingers in a flexible way, it can better account for the nonlinear relationship ECoG exhibits when expressing hand gestures, including unintentional finger co-activations. This is reflected by the superior joint finger trajectory predictions compared to eBTTR, and predictions that are on par with BTTR in single-finger scenarios. For the American sign language alphabet (Utrecht dataset), the average correlation across all fingers for all subjects was 0.73 for Go-BTTR, 0.719 for eBTTR and 0.70 for BTTR. For the Flemish sign language alphabet (Leuven dataset), the average correlation across all fingers for all subjects was 0.37 for Go-BTTR, 0.34 for eBTTR and 0.33 for BTTR.Significance.Our findings show that Go-BTTR is capable of decoding complex hand gestures taken from the sign language alphabet. Go-BTTR also demonstrates computational efficiency, providing a notable benefit when intracranial electrodes are inserted during a patient's pre-surgical evaluation. This efficiency helps reduce the time required for developing and testing a brain-computer interface solution.

Decoding Single and Paired Phonemes Using 7T Functional MRI

Several studies have shown that mouth movements related to the pronunciation of individual phonemes are represented in the sensorimotor cortex. This would theoretically allow for brain computer interfaces that are capable of decoding continuous speech by training classifiers based on the activity in the sensorimotor cortex related to the production of individual phonemes. To address this, we investigated the decodability of trials with individual and paired phonemes (pronounced consecutively with one second interval) using activity in the sensorimotor cortex. Fifteen participants pronounced 3 different phonemes and 3 combinations of two of the same phonemes in a 7T functional MRI experiment. We confirmed that support vector machine (SVM) classification of single and paired phonemes was possible. Importantly, by combining classifiers trained on single phonemes, we were able to classify paired phonemes with an accuracy of 53% (33% chance level), demonstrating that activity of isolated phonemes is present and distinguishable in combined phonemes. A SVM searchlight analysis showed that the phoneme representations are widely distributed in the ventral sensorimotor cortex. These findings provide insights about the neural representations of single and paired phonemes. Furthermore, it supports the notion that speech BCI may be feasible based on machine learning algorithms trained on individual phonemes using intracranial electrode grids.

A Review of Motor Brain-Computer Interfaces Using Intracranial Electroencephalography Based on Surface Electrodes and Depth Electrodes

Brain-computer interfaces (BCIs) provide a communication interface between the brain and external devices and have the potential to restore communication and control in patients with neurological injury or disease. For the invasive BCIs, most studies recruited participants from hospitals requiring invasive device implantation. Three widely used clinical invasive devices that have the potential for BCIs applications include surface electrodes used in electrocorticography (ECoG) and depth electrodes used in Stereo-electroencephalography (SEEG) and deep brain stimulation (DBS). This review focused on BCIs research using surface (ECoG) and depth electrodes (including SEEG, and DBS electrodes) for movement decoding on human subjects. Unlike previous reviews, the findings presented here are from the perspective of the decoding target or task. In detail, five tasks will be considered, consisting of the kinematic decoding, kinetic decoding,identification of body parts, dexterous hand decoding, and motion intention decoding. The typical studies are surveyed and analyzed. The reviewed literature demonstrated a distributed motor-related network that spanned multiple brain regions. Comparison between surface and depth studies demonstrated that richer information can be obtained using surface electrodes. With regard to the decoding algorithms, deep learning exhibited superior performance using raw signals than traditional machine learning algorithms. Despite the promising achievement made by the open-loop BCIs, closed-loop BCIs with sensory feedback are still in their early stage, and the chronic implantation of both ECoG surface and depth electrodes has not been thoroughly evaluated.

Boosting brain-computer interfaces with functional electrical stimulation: potential applications in people with locked-in syndrome

Individuals with a locked-in state live with severe whole-body paralysis that limits their ability to communicate with family and loved ones. Recent advances in brain-computer interface (BCI) technology have presented a potential alternative for these people to communicate by detecting neural activity associated with attempted hand or speech movements and translating the decoded intended movements to a control signal for a computer. A technique that could potentially enrich the communication capacity of BCIs is functional electrical stimulation (FES) of paralyzed limbs and face to restore body and facial movements of paralyzed individuals, allowing to add body language and facial expression to communication BCI utterances. Here, we review the current state of the art of existing BCI and FES work in people with paralysis of body and face and propose that a combined BCI-FES approach, which has already proved successful in several applications in stroke and spinal cord injury, can provide a novel promising mode of communication for locked-in individuals.

Platelets and hemostatic proteins are co-localized with chronic neuroinflammation surrounding implanted intracortical microelectrodes

Intracortical microelectrodes induce vascular injury upon insertion into the cortex. As blood vessels rupture, blood proteins and blood-derived cells (including platelets) are introduced into the 'immune privileged' brain tissues at higher-than-normal levels, passing through the damaged blood-brain barrier. Blood proteins adhere to implant surfaces, increasing the likelihood of cellular recognition leading to activation of immune and inflammatory cells. Persistent neuroinflammation is a major contributing factor to declining microelectrode recording performance. We investigated the spatial and temporal relationship of blood proteins fibrinogen and von Willebrand Factor (vWF), platelets, and type IV collagen, in relation to glial scarring markers for microglia and astrocytes following implantation of non-functional multi-shank silicon microelectrode probes into rats. Together with type IV collagen, fibrinogen and vWF augment platelet recruitment, activation, and aggregation. Our main results indicate blood proteins participating in hemostasis (fibrinogen and vWF) persisted at the microelectrode interface for up to 8-weeks after implantation. Further, type IV collagen and platelets surrounded the probe interface with similar spatial and temporal trends as vWF and fibrinogen. In addition to prolonged blood-brain barrier instability, specific blood and extracellular matrix proteins may play a role in promoting the inflammatory activation of platelets and recruitment to the microelectrode interface. STATEMENT OF SIGNIFICANCE: Implanted microelectrodes have substantial potential for restoring function to people with paralysis and amputation by providing signals that feed into natural control algorithms that drive prosthetic devices. Unfortunately, these microelectrodes do not display robust performance over time. Persistent neuroinflammation is widely thought to be a primary contributor to the devices' progressive decline in performance. Our manuscript reports on the highly local and persistent accumulation of platelets and hemostatic blood proteins around the microelectrode interface of brain implants. To our knowledge neuroinflammation driven by cellular and non-cellular responses associated with hemostasis and coagulation has not been rigorously quantified elsewhere. Our findings identify potential targets for therapeutic intervention and a better understanding of the driving mechanisms to neuroinflammation in the brain.

Towards clinical application of implantable brain-computer interfaces for people with late-stage ALS: medical and ethical considerations

Individuals with amyotrophic lateral sclerosis (ALS) frequently develop speech and communication problems in the course of their disease. Currently available augmentative and alternative communication technologies do not present a solution for many people with advanced ALS, because these devices depend on residual and reliable motor activity. Brain-computer interfaces (BCIs) use neural signals for computer control and may allow people with late-stage ALS to communicate even when conventional technology falls short. Recent years have witnessed fast progression in the development and validation of implanted BCIs, which place neural signal recording electrodes in or on the cortex. Eventual widespread clinical application of implanted BCIs as an assistive communication technology for people with ALS will have significant consequences for their daily life, as well as for the clinical management of the disease, among others because of the potential interaction between the BCI and other procedures people with ALS undergo, such as tracheostomy. This article aims to facilitate responsible real-world implementation of implanted BCIs. We review the state of the art of research on implanted BCIs for communication, as well as the medical and ethical implications of the clinical application of this technology. We conclude that the contribution of all BCI stakeholders, including clinicians of the various ALS-related disciplines, will be needed to develop procedures for, and shape the process of, the responsible clinical application of implanted BCIs.

Phase-amplitude coupling-based adaptive filters for neural signal decoding

Bandpass filters play a core role in ECoG signal processing. Commonly used frequency bands such as alpha, beta, and gamma bands can reflect the normal rhythm of the brain. However, the universally predefined bands might not be optimal for a specific task. Especially the gamma band usually covers a wide frequency span (i.e., 30-200 Hz) which can be too coarse to capture features that appear in narrow bands. An ideal option is to find the optimal frequency bands for specific tasks in real-time and dynamically. To tackle this problem, we propose an adaptive band filter that selects the useful frequency band in a data-driven way. Specifically, we leverage the phase-amplitude coupling (PAC) of the coupled working mechanism of synchronizing neuron and pyramidal neurons in neuronal oscillations, in which the phase of slower oscillations modulates the amplitude of faster ones, to help locate the fine frequency bands from the gamma range, in a task-specific and individual-specific way. Thus, the information can be more precisely extracted from ECoG signals to improve neural decoding performance. Based on this, an end-to-end decoder (PACNet) is proposed to construct a neural decoding application with adaptive filter banks in a uniform framework. Experiments show that PACNet can improve neural decoding performance universally with different tasks.

Advances in human intracranial electroencephalography research, guidelines and good practices

Since the second-half of the twentieth century, intracranial electroencephalography (iEEG), including both electrocorticography (ECoG) and stereo-electroencephalography (sEEG), has provided an intimate view into the human brain. At the interface between fundamental research and the clinic, iEEG provides both high temporal resolution and high spatial specificity but comes with constraints, such as the individual's tailored sparsity of electrode sampling. Over the years, researchers in neuroscience developed their practices to make the most of the iEEG approach. Here we offer a critical review of iEEG research practices in a didactic framework for newcomers, as well addressing issues encountered by proficient researchers. The scope is threefold: (i) review common practices in iEEG research, (ii) suggest potential guidelines for working with iEEG data and answer frequently asked questions based on the most widespread practices, and (iii) based on current neurophysiological knowledge and methodologies, pave the way to good practice standards in iEEG research. The organization of this paper follows the steps of iEEG data processing. The first section contextualizes iEEG data collection. The second section focuses on localization of intracranial electrodes. The third section highlights the main pre-processing steps. The fourth section presents iEEG signal analysis methods. The fifth section discusses statistical approaches. The sixth section draws some unique perspectives on iEEG research. Finally, to ensure a consistent nomenclature throughout the manuscript and to align with other guidelines, e.g., Brain Imaging Data Structure (BIDS) and the OHBM Committee on Best Practices in Data Analysis and Sharing (COBIDAS), we provide a glossary to disambiguate terms related to iEEG research.

Decoding Continuous Kinetic Information of Grasp from Stereo-electroencephalographic (SEEG) Recordings

OBJECTIVE

Brain-computer interfaces (BCIs) have the potential to bypass damaged neural pathways and restore functionality lost due to injury or disease. Approaches to decoding kinematic information are well documented; however, the decoding of kinetic information has received less attention. Additionally, the possibility of using stereo-electroencephalography (SEEG) for kinetic decoding during hand grasping tasks is still largely unknown. Thus, the objective of this paper is to demonstrate kinetic parameter decoding using SEEG in patients performing a grasping task with two different force levels under two different ascending rates.

APPROACH

Temporal-spectral representations were studied to investigate frequency modulation under different force tasks. Then, force amplitude was decoded from SEEG recordings using multiple decoders, including a linear model, a partial least squares (PLS) model, an unscented Kalman filter, and three deep learning models (shallow convolutional neural network, deep convolutional neural network and the proposed CNN+RNN neural network).

MAIN RESULTS

The current study showed that: 1) for some channel, both low-frequency modulation (event-related desynchronization (ERD)) and high-frequency modulation (event-related synchronization (ERS)) were sustained during prolonged force holding periods; 2) continuously changing grasp force can be decoded from the SEEG signals; 3) the novel CNN+RNN deep learning model achieved the best decoding performance, with the predicted force magnitude closely aligned to the ground truth under different force amplitudes and changing rates.

SIGNIFICANCE

This work verified the possibility of decoding continuously changing grasp force using SEEG recordings. The result presented in this study demonstrated the potential of SEEG recordings for future BCI application.

Decoding ECoG signal into 3D hand translation using deep learning

Objective.Motor brain-computer interfaces (BCIs) are a promising technology that may enable motor-impaired people to interact with their environment. BCIs would potentially compensate for arm and hand function loss, which is the top priority for individuals with tetraplegia. Designing real-time and accurate BCI is crucial to make such devices useful, safe, and easy to use by patients in a real-life environment. Electrocorticography (ECoG)-based BCIs emerge as a good compromise between invasiveness of the recording device and good spatial and temporal resolution of the recorded signal. However, most ECoG signal decoders used to predict continuous hand movements are linear models. These models have a limited representational capacity and may fail to capture the relationship between ECoG signal features and continuous hand movements. Deep learning (DL) models, which are state-of-the-art in many problems, could be a solution to better capture this relationship.Approach.In this study, we tested several DL-based architectures to predict imagined 3D continuous hand translation using time-frequency features extracted from ECoG signals. The dataset used in the analysis is a part of a long-term clinical trial (ClinicalTrials.gov identifier: NCT02550522) and was acquired during a closed-loop experiment with a tetraplegic subject. The proposed architectures include multilayer perceptron, convolutional neural networks (CNNs), and long short-term memory networks (LSTM). The accuracy of the DL-based and multilinear models was compared offline using cosine similarity.Main results.Our results show that CNN-based architectures outperform the current state-of-the-art multilinear model. The best architecture exploited the spatial correlation between neighboring electrodes with CNN and benefited from the sequential character of the desired hand trajectory by using LSTMs. Overall, DL increased the average cosine similarity, compared to the multilinear model, by up to 60%, from 0.189 to 0.302 and from 0.157 to 0.249 for the left and right hand, respectively.Significance.This study shows that DL-based models could increase the accuracy of BCI systems in the case of 3D hand translation prediction in a tetraplegic subject.

Recent Advances and Current Trends in Brain-Computer Interface (BCI) Research and Their Applications

Brain-Computer Interface (BCI) provides direct communication between the brain and an external device. BCI systems have become a trendy field of research in recent years. These systems can be used in a variety of applications to help both disabled and healthy people. Concerning significant BCI progress, we may assume that these systems are not very far from real-world applications. This review has taken into account current trends in BCI research. In this survey, one hundred most cited articles from the WOS database were selected over the last four years. This survey is divided into several sectors. These sectors are Medicine, Communication and Control, Entertainment, and Other BCI applications. The application area, recording method, signal acquisition types, and countries of origin have been identified in each article. This survey provides an overview of the BCI articles published from 2016 to 2020 and their current trends and advances in different application areas.

Decoding of Hand Gestures from Electrocorticography with LSTM Based Deep Neural Network

Hand gesture decoding is a key component of controlling prosthesis in the area of Brain Computer Interface (BCI). This study is concerned with classification of hand gestures, based on Electrocorticography (ECoG) recordings. Recent studies have utilized the temporal information in ECoG signals for robust hand gesture decoding. In our preliminary analysis on ECoG recordings of hand gestures, we observed different power variations in six frequency bands ranging from 4 to 200 Hz. Therefore, the current trend of including temporal information in the classifier was extended to provide equal importance to power variations in each of these frequency bands. Statistical and Principal Component Analysis (PCA) based feature reduction was implemented for each frequency band separately, and classification was performed with a Long Short-Term Memory (LSTM) based neural network to utilize both temporal and spatial information of each frequency band. The proposed architecture along with each feature reduction method was tested on ECoG recordings of five finger flexions performed by seven subjects from the publicly available 'fingerflex' dataset. An average classification accuracy of 82.4% was achieved with the statistical based channel selection method which is an improvement compared to state-of-the-art methods.

Investigating Data Cleaning Methods to Improve Performance of Brain-Computer Interfaces Based on Stereo-Electroencephalography

Stereo-electroencephalography (SEEG) utilizes localized and penetrating depth electrodes to directly measure electrophysiological brain activity. The implanted electrodes generally provide a sparse sampling of multiple brain regions, including both cortical and subcortical structures, making the SEEG neural recordings a potential source for the brain–computer interface (BCI) purpose in recent years. For SEEG signals, data cleaning is an essential preprocessing step in removing excessive noises for further analysis. However, little is known about what kinds of effect that different data cleaning methods may exert on BCI decoding performance and, moreover, what are the reasons causing the differentiated effects. To address these questions, we adopted five different data cleaning methods, including common average reference, gray–white matter reference, electrode shaft reference, bipolar reference, and Laplacian reference, to process the SEEG data and evaluated the effect of these methods on improving BCI decoding performance. Additionally, we also comparatively investigated the changes of SEEG signals induced by these different methods from multiple-domain (e.g., spatial, spectral, and temporal domain). The results showed that data cleaning methods could improve the accuracy of gesture decoding, where the Laplacian reference produced the best performance. Further analysis revealed that the superiority of the data cleaning method with excellent performance might be attributed to the increased distinguishability in the low-frequency band. The findings of this work highlighted the importance of applying proper data clean methods for SEEG signals and proposed the application of Laplacian reference for SEEG-based BCI.

Decoding four hand gestures with a single bipolar pair of electrocorticography electrodes

Objective.Electrocorticography (ECoG) based brain-computer interfaces (BCIs) can be used to restore communication in individuals with locked-in syndrome. In motor-based BCIs, the number of degrees-of-freedom, and thus the speed of the BCI, directly depends on the number of classes that can be discriminated from the neural activity in the sensorimotor cortex. When considering minimally invasive BCI implants, the size of the subdural ECoG implant must be minimized without compromising the number of degrees-of-freedom.Approach.Here we investigated if four hand gestures could be decoded using a single ECoG strip of four consecutive electrodes spaced 1 cm apart and compared the performance between a unipolar and a bipolar montage. For that we collected data of seven individuals with intractable epilepsy implanted with ECoG grids, covering the hand region of the sensorimotor cortex. Based on the implanted grids, we generated virtual ECoG strips and compared the decoding accuracy between (a) a single unipolar electrode (Unipolar Electrode), (b) a combination of four unipolar electrodes (Unipolar Strip), (c) a single bipolar pair (Bipolar Pair) and (d) a combination of six bipolar pairs (Bipolar Strip).Main results.We show that four hand gestures can be equally well decoded using 'Unipolar Strips' (mean 67.4 ± 11.7%), 'Bipolar Strips' (mean 66.6 ± 12.1%) and 'Bipolar Pairs' (mean 67.6 ± 9.4%), while 'Unipolar Electrodes' (61.6 ± 5.9%) performed significantly worse compared to 'Unipolar Strips' and 'Bipolar Pairs'.Significance.We conclude that a single bipolar pair is a potential candidate for minimally invasive motor-based BCIs and encourage the use of ECoG as a robust and reliable BCI platform for multi-class movement decoding.

Decoding attempted phantom hand movements from ipsilateral sensorimotor areas after amputation

Objective.The sensorimotor cortex is often selected as target in the development of a Brain-Computer Interface, as activation patterns from this region can be robustly decoded to discriminate between different movements the user executes. Up until recently, such BCIs were primarily based on activity in the contralateral hemisphere, where decoding movements still works even years after denervation. However, there is increasing evidence for a role of the sensorimotor cortex in controlling the ipsilateral body. The aim of this study is to investigate the effects of denervation on the movement representation on the ipsilateral sensorimotor cortex.Approach.Eight subjects with acquired above-elbow arm amputation and nine controls performed a task in which they made (or attempted to make with their phantom hand) six different gestures from the American Manual Alphabet. Brain activity was measured using 7T functional MRI, and a classifier was trained to discriminate between activation patterns on four different regions of interest (ROIs) on the ipsilateral sensorimotor cortex.Main results.Classification scores showed that decoding was possible and significantly better than chance level for both the phantom and intact hands from all ROIs. Decoding both the left (intact) and right (phantom) hand from the same hemisphere was also possible with above-chance level classification score.Significance.The possibility to decode both hands from the same hemisphere, even years after denervation, indicates that implantation of motor-electrodes for BCI control possibly need only cover a single hemisphere, making surgery less invasive, and increasing options for people with lateralized damage to motor cortex like after stroke.

Typical somatomotor physiology of the hand is preserved in a patient with an amputated arm: An ECoG case study

Electrophysiological signals in the human motor system may change in different ways after deafferentation, with some studies emphasizing reorganization while others propose retained physiology. Understanding whether motor electrophysiology is retained over longer periods of time can be invaluable for patients with paralysis (e.g. ALS or brainstem stroke) when signals from sensorimotor areas may be used for communication or control over neural prosthetic devices. In addition, a maintained electrophysiology can potentially benefit the treatment of phantom limb pains through prolonged use of these signals in a brain-machine interface (BCI). Here, we were presented with the unique opportunity to investigate the physiology of the sensorimotor cortex in a patient with an amputated arm using electrocorticographic (ECoG) measurements. While implanted with an ECoG grid for clinical evaluation of electrical stimulation for phantom limb pain, the patient performed attempted finger movements with the contralateral (lost) hand and executed finger movements with the ipsilateral (healthy) hand. The electrophysiology of the sensorimotor cortex contralateral to the amputated hand remained very similar to that of hand movement in healthy people, with a spatially focused increase of high-frequency band (65-175 Hz; HFB) power over the hand region and a distributed decrease in low-frequency band (15-28 Hz; LFB) power. The representation of the three different fingers (thumb, index and little) remained intact and HFB patterns could be decoded using support vector learning at single-trial classification accuracies of >90%, based on the first 1-3 s of the HFB response. These results indicate that hand representations are largely retained in the motor cortex. The intact physiological response of the amputated hand, the high distinguishability of the fingers and fast temporal peak are encouraging for neural prosthetic devices that target the sensorimotor cortex.

Self-Related Stimuli Decoding With Auditory and Visual Modalities Using Stereo-Electroencephalography

Name recognition plays important role in self-related cognitive processes and also contributes to a variety of clinical applications, such as autism spectrum disorder diagnosis and consciousness disorder analysis. However, most previous name-related studies usually adopted noninvasive EEG or fMRI recordings, which were limited by low spatial resolution and temporal resolution, respectively, and thus millisecond-level response latencies in precise brain regions could not be measured using these noninvasive recordings. By invasive stereo-electroencephalography (SEEG) recordings that have high resolution in both the spatial and temporal domain, the current study distinguished the neural response to one's own name or a stranger's name, and explored common active brain regions in both auditory and visual modalities. The neural activities were classified using spatiotemporal features of high-gamma, beta, and alpha band. Results showed that different names could be decoded using multi-region SEEG signals, and the best classification performance was achieved at high gamma (60–145 Hz) band. In this case, auditory and visual modality-based name classification accuracies were 84.5 ± 8.3 and 79.9 ± 4.6%, respectively. Additionally, some single regions such as the supramarginal gyrus, middle temporal gyrus, and insula could also achieve remarkable accuracies for both modalities, supporting their roles in the processing of self-related information. The average latency of the difference between the two responses in these precise regions was 354 ± 63 and 285 ± 59 ms in the auditory and visual modality, respectively. This study suggested that name recognition was attributed to a distributed brain network, and the subsets with decoding capabilities might be potential implanted regions for awareness detection and cognition evaluation.

The Riemannian Spatial Pattern method: mapping and clustering movement imagery using Riemannian geometry

OBJECTIVE

Over the last decade, Riemannian geometry has shown promising results for motor imagery classification. However, extracting the underlying spatial features is not as straightforward as for applying Common Spatial Pattern (CSP) filtering prior to classification. In this article, we propose a simple way to extract the spatial patterns obtained from Riemannian classification: the Riemannian Spatial Pattern (RSP) method, which is based on the backward channel selection procedure.

APPROACH

The RSP method was compared to the CSP approach on ECoG data obtained from a quadriplegic patient while performing imagined movements of arm articulations and fingers.

MAIN RESULTS

Similar results were found between the RSP and CSP methods for mapping each motor imagery task with activations following the classical somatotopic organization. Clustering obtained by pairwise comparisons of imagined motor movements however, revealed higher differentiation for the RSP method compared to the CSP approach. Importantly, the RSP approach could provide a precise comparison of the imagined finger flexions which added supplementary information to the mapping results.

SIGNIFICANCE

Our new RSP method illustrates the interest of the Riemannian framework in the spatial domain and as such offers new avenues for the neuroimaging community. This study is part of an ongoing clinical trial registered with ClinicalTrials.gov, NCT02550522.

The Neural Representation of Force across Grasp Types in Motor Cortex of Humans with Tetraplegia

Intracortical brain-computer interfaces (iBCIs) have the potential to restore hand grasping and object interaction to individuals with tetraplegia. Optimal grasping and object interaction require simultaneous production of both force and grasp outputs. However, since overlapping neural populations are modulated by both parameters, grasp type could affect how well forces are decoded from motor cortex in a closed-loop force iBCI. Therefore, this work quantified the neural representation and offline decoding performance of discrete hand grasps and force levels in two human participants with tetraplegia. Participants attempted to produce three discrete forces (light, medium, hard) using up to five hand grasp configurations. A two-way Welch ANOVA was implemented on multiunit neural features to assess their modulation to force and grasp Demixed principal component analysis (dPCA) was used to assess for population-level tuning to force and grasp and to predict these parameters from neural activity. Three major findings emerged from this work: (1) force information was neurally represented and could be decoded across multiple hand grasps (and, in one participant, across attempted elbow extension as well); (2) grasp type affected force representation within multiunit neural features and offline force classification accuracy; and (3) grasp was classified more accurately and had greater population-level representation than force. These findings suggest that force and grasp have both independent and interacting representations within cortex, and that incorporating force control into real-time iBCI systems is feasible across multiple hand grasps if the decoder also accounts for grasp type.

Classification of Individual Finger Movements Using Intracortical Recordings in Human Motor Cortex

BACKGROUND

Intracortical microelectrode arrays have enabled people with tetraplegia to use a brain-computer interface for reaching and grasping. In order to restore dexterous movements, it will be necessary to control individual fingers.

OBJECTIVE

To predict which finger a participant with hand paralysis was attempting to move using intracortical data recorded from the motor cortex.

METHODS

A 31-yr-old man with a C5/6 ASIA B spinal cord injury was implanted with 2 88-channel microelectrode arrays in left motor cortex. Across 3 d, the participant observed a virtual hand flex in each finger while neural firing rates were recorded. A 6-class linear discriminant analysis (LDA) classifier, with 10 × 10-fold cross-validation, was used to predict which finger movement was being performed (flexion/extension of all 5 digits and adduction/abduction of the thumb).

RESULTS

The mean overall classification accuracy was 67% (range: 65%-76%, chance: 17%), which occurred at an average of 560 ms (range: 420-780 ms) after movement onset. Individually, thumb flexion and thumb adduction were classified with the highest accuracies at 92% and 93%, respectively. The index, middle, ring, and little achieved an accuracy of 65%, 59%, 43%, and 56%, respectively, and, when incorrectly classified, were typically marked as an adjacent finger. The classification accuracies were reflected in a low-dimensional projection of the neural data into LDA space, where the thumb-related movements were most separable from the finger movements.

CONCLUSION

Classification of intention to move individual fingers was accurately predicted by intracortical recordings from a human participant with the thumb being particularly independent.

Functional MRI based simulations of ECoG grid configurations for optimal measurement of spatially distributed hand-gesture information

Objective In electrocorticography (ECoG), the physical characteristics of the electrode grid determine which aspect of the neurophysiology is measured. For particular cases, the ECoG grid may be tailored to capture specific features, such as in the development and use of brain-computer-interfaces (BCI). Neural representations of hand movement are increasingly used to control ECoG based BCIs. However, it remains unclear which grid configurations are the most optimal to capture the dynamics of hand gesture information. Here, we investigate how the design and surgical placement of grids would affect the usability of ECoG measurements. Approach High resolution 7T functional MRI was used as a proxy for neural activity in ten healthy participants to simulate various grid configurations, and evaluated the performance of each configuration for decoding hand gestures. The grid configurations varied in number of electrodes, electrode distance and electrode size. Main results Optimal decoding of hand gestures occurred in grid configurations with a higher number of densely-packed, large-size, electrodes up to a grid of ~5x5 electrodes. When restricting the grid placement to a highly informative region of primary sensorimotor cortex, optimal parameters converged to about 3x3 electrodes, an inter-electrode distance of 8mm, and an electrode size of 3mm radius (performing at ~70% 3-class classification accuracy). Significance Our approach might be used to identify the most informative region, find the optimal grid configuration and assist in positioning of the grid to achieve high BCI performance for the decoding of hand-gestures prior to surgical implantation.

Distributed mixed signal demultiplexer for electrocorticography electrodes

This work presents a novel architecture, exemplified for electrophysiological applications like ECoG that can be used to detect Epilepsy. The new ECoG is based on a mixed analog-digital architecture (Pulse Amplitude Modulation PAM), that allows the use of thousands of electrodes for recording. Whilst the increased number of electrodes helps to refine the spatial resolution of the medical application, the transmission of the signals from the electrodes to an external analysing device appears to be a bottleneck. To overcoming this, our work presents a hardware architecture and corresponding protocol for a mixed architecture that improves the information density between channels and their signal-to-noise ratio. This is shown by the correlation between the input and the transmitted signals in comparison to a classical digital transmission (Pulse Code Modulation PCM) system. We show in this work that it is possible to transmit the signals of 10 channels with a analog-digital architecture with the same quality of a full digital architecture.

A state-based probabilistic method for decoding hand position during movement from ECoG signals in non-human primate

OBJECTIVE

In this study, we proposed a state-based probabilistic method for decoding hand positions during unilateral and bilateral movements using the ECoG signals recorded from the brain of Rhesus monkey.

APPROACH

A customized electrode array was implanted subdurally in the right hemisphere of the brain covering from the primary motor cortex to the frontal cortex. Three different experimental paradigms were considered: ipsilateral, contralateral, and bilateral movements. During unilateral movement, the monkey was trained to get food with one hand, while during bilateral movement, the monkey used its left and right hands alternately to get food. To estimate the hand positions, a state-based probabilistic method was introduced which was based on the conditional probability of the hand movement state (i.e. idle, right hand movement, and left hand movement) and the conditional expectation of the hand position for each state. Moreover, a hybrid feature extraction method based on linear discriminant analysis and partial least squares (PLS) was introduced.

MAIN RESULTS

The proposed method could successfully decode the hand positions during ipsilateral, contralateral, and bilateral movements and significantly improved the decoding performance compared to the conventional Kalman and PLS regression methods [Formula: see text]. The proposed hybrid feature extraction method was found to outperform both the PLS and PCA methods [Formula: see text]. Investigating the kinematic information of each frequency band shows that more informative frequency bands were [Formula: see text] (15-30 Hz) and [Formula: see text](50-100 Hz) for ipsilateral and [Formula: see text] and [Formula: see text] (100-200 Hz) for contralateral movements. It is observed that ipsilateral movement was decoded better than contralateral movement for [Formula: see text] (5-15 Hz) and [Formula: see text] bands, while contralateral movements was decoded better for [Formula: see text] (30-200 Hz) and hfECoG (200-400 Hz) bands.

SIGNIFICANCE

Accurate decoding the bilateral movement using the ECoG recorded from one brain hemisphere is an important issue toward real-life applications of the brain-machine interface technologies.

Context-dependent relationship in high-resolution micro-ECoG studies during finger movements

OBJECTIVE

The activation of the sensorimotor cortex as measured by electrocorticographic (ECoG) signals has been correlated with contralateral hand movements in humans, as precisely as the level of individual digits. However, the relationship between individual and multiple synergistic finger movements and the neural signal as detected by ECoG has not been fully explored. The authors used intraoperative high-resolution micro-ECoG (µECoG) on the sensorimotor cortex to link neural signals to finger movements across several context-specific motor tasks.

METHODS

Three neurosurgical patients with cortical lesions over eloquent regions participated. During awake craniotomy, a sensorimotor cortex area of hand movement was localized by high-frequency responses measured by an 8 × 8 µECoG grid of 3-mm interelectrode spacing. Patients performed a flexion movement of the thumb or index finger, or a pinch movement of both, based on a visual cue. High-gamma (HG; 70-230 Hz) filtered µECoG was used to identify dominant electrodes associated with thumb and index movement. Hand movements were recorded by a dataglove simultaneously with µECoG recording.

RESULTS

In all 3 patients, the electrodes controlling thumb and index finger movements were identifiable approximately 3-6-mm apart by the HG-filtered µECoG signal. For HG power of cortical activation measured with µECoG, the thumb and index signals in the pinch movement were similar to those observed during thumb-only and index-only movement, respectively (all p > 0.05). Index finger movements, measured by the dataglove joint angles, were similar in both the index-only and pinch movements (p > 0.05). However, despite similar activation across the conditions, markedly decreased thumb movement was observed in pinch relative to independent thumb-only movement (all p < 0.05).

CONCLUSIONS

HG-filtered µECoG signals effectively identify dominant regions associated with thumb and index finger movement. For pinch, the µECoG signal comprises a combination of the signals from individual thumb and index movements. However, while the relationship between the index finger joint angle and HG-filtered signal remains consistent between conditions, there is not a fixed relationship for thumb movement. Although the HG-filtered µECoG signal is similar in both thumb-only and pinch conditions, the actual thumb movement is markedly smaller in the pinch condition than in the thumb-only condition. This implies a nonlinear relationship between the cortical signal and the motor output for some, but importantly not all, movement types. This analysis provides insight into the tuning of the motor cortex toward specific types of motor behaviors.

Real-time fMRI for brain-computer interfacing

Brain-computer interfaces (BCIs) based on functional magnetic resonance imaging (fMRI) provide an important complement to other noninvasive BCIs. While fMRI has several disadvantages (being nonportable, methodologically challenging, costly, and noisy), it is the only method providing high spatial resolution whole-brain coverage of brain activation. These properties allow relating mental activities to specific brain regions and networks providing a transparent scheme for BCI users to encode information and for real-time fMRI BCI systems to decode the intents of the user. Various mental activities have been used successfully in fMRI BCIs so far that can be classified into the four categories: (a) higher-order cognitive tasks (e.g., mental calculation), (b) covert language-related tasks (e.g., mental speech and mental singing), (c) imagery tasks (motor, visual, auditory, tactile, and emotion imagery), and (d) selective attention tasks (visual, auditory, and tactile attention). While the ultimate spatial and temporal resolution of fMRI BCIs is limited by the physiologic properties of the hemodynamic response, technical and analytical advances will likely lead to substantially improved fMRI BCIs in the future using, for example, decoding of imagined letter shapes at 7T as the basis for more "natural" communication BCIs.

Brain-computer interfaces for communication

Locked-in syndrome (LIS) is characterized by an inability to move or speak in the presence of intact cognition and can be caused by brainstem trauma or neuromuscular disease. Quality of life (QoL) in LIS is strongly impaired by the inability to communicate, which cannot always be remedied by traditional augmentative and alternative communication (AAC) solutions if residual muscle activity is insufficient to control the AAC device. Brain-computer interfaces (BCIs) may offer a solution by employing the person's neural signals instead of relying on muscle activity. Here, we review the latest communication BCI research using noninvasive signal acquisition approaches (electroencephalography, functional magnetic resonance imaging, functional near-infrared spectroscopy) and subdural and intracortical implanted electrodes, and we discuss current efforts to translate research knowledge into usable BCI-enabled communication solutions that aim to improve the QoL of individuals with LIS.

Human brain function and brain-computer interfaces

Human brain function research has evolved dramatically in the last decades. In this chapter the role of modern methods of recording brain activity in understanding human brain function is explained. Current knowledge of brain function relevant to brain-computer interface (BCI) research is detailed, with an emphasis on the motor system which provides an exceptional level of detail to decoding of intended or attempted movements in paralyzed beneficiaries of BCI technology and translation to computer-mediated actions. BCI technologies that stand to benefit the most of the detailed organization of the human cortex are, and for the foreseeable future are likely to be, reliant on intracranial electrodes. These evolving technologies are expected to enable severely paralyzed people to regain the faculty of movement and speech in the coming decades.

Decoding Movement From Electrocorticographic Activity: A Review

Electrocorticography (ECoG) holds promise to provide efficient neuroprosthetic solutions for people suffering from neurological disabilities. This recording technique combines adequate temporal and spatial resolution with the lower risks of medical complications compared to the other invasive methods. ECoG is routinely used in clinical practice for preoperative cortical mapping in epileptic patients. During the last two decades, research utilizing ECoG has considerably grown, including the paradigms where behaviorally relevant information is extracted from ECoG activity with decoding algorithms of different complexity. Several research groups have advanced toward the development of assistive devices driven by brain-computer interfaces (BCIs) that decode motor commands from multichannel ECoG recordings. Here we review the evolution of this field and its recent tendencies, and discuss the potential areas for future development.

Time-Variant Linear Discriminant Analysis Improves Hand Gesture and Finger Movement Decoding for Invasive Brain-Computer Interfaces

Invasive brain-computer interfaces yield remarkable performance in a multitude of applications. For classification experiments, high-gamma bandpower features and linear discriminant analysis (LDA) are commonly used due to simplicity and robustness. However, LDA is inherently static and not suited to account for transient information that is typically present in high-gamma features. To resolve this issue, we here present an extension of LDA to the time-variant feature space. We call this method time-variant linear discriminant analysis (TVLDA). It intrinsically provides a feature reduction stage, which makes external approaches thereto obsolete, such as feature selection techniques or common spatial patterns (CSPs). As well, we propose a time-domain whitening stage which equalizes the pronounced 1/f-shape of the typical brain-wave spectrum. We evaluated our proposed architecture based on recordings from 15 epilepsy patients with temporarily implanted subdural grids, who participated in additional research experiments besides clinical treatment. The experiments featured two different motor tasks involving three high-level gestures and individual finger movement. We used log-transformed bandpower features from the high-gamma band (50-300 Hz, excluding power-line harmonics) for classification. On average, whitening improved the classification performance by about 11%. On whitened data, TVLDA outperformed LDA with feature selection by 11.8%, LDA with CSPs by 13.9%, and regularized LDA with vectorized features by 16.4%. At the same time, TVLDA only required one or two internal features to achieve this. TVLDA provides stable results even if very few trials are available. It is easy to implement, fully automatic and deterministic. Due to its low complexity, TVLDA is suited for real-time brain-computer interfaces. Training is done in less than a second. TVLDA performed particularly well in experiments with data from high-density electrode arrays. For example, the three high-level gestures were correctly identified at a rate of 99% over all subjects. Similarly, the decoding accuracy of individual fingers was 96% on average over all subjects. To our knowledge, these mean accuracies are the highest ever reported for three-class and five-class motor-control BCIs.

Human motor decoding from neural signals: a review

Many people suffer from movement disability due to amputation or neurological diseases. Fortunately, with modern neurotechnology now it is possible to intercept motor control signals at various points along the neural transduction pathway and use that to drive external devices for communication or control. Here we will review the latest developments in human motor decoding. We reviewed the various strategies to decode motor intention from human and their respective advantages and challenges. Neural control signals can be intercepted at various points in the neural signal transduction pathway, including the brain (electroencephalography, electrocorticography, intracortical recordings), the nerves (peripheral nerve recordings) and the muscles (electromyography). We systematically discussed the sites of signal acquisition, available neural features, signal processing techniques and decoding algorithms in each of these potential interception points. Examples of applications and the current state-of-the-art performance were also reviewed. Although great strides have been made in human motor decoding, we are still far away from achieving naturalistic and dexterous control like our native limbs. Concerted efforts from material scientists, electrical engineers, and healthcare professionals are needed to further advance the field and make the technology widely available in clinical use.

Encoding of kinetic and kinematic movement parameters in the sensorimotor cortex: A Brain-Computer Interface perspective

For severely paralyzed people, Brain-Computer Interfaces (BCIs) can potentially replace lost motor output and provide a brain-based control signal for augmentative and alternative communication devices or neuroprosthetics. Many BCIs focus on neuronal signals acquired from the hand area of the sensorimotor cortex, employing changes in the patterns of neuronal firing or spectral power associated with one or more types of hand movement. Hand and finger movement can be described by two groups of movement features, namely kinematics (spatial and motion aspects) and kinetics (muscles and forces). Despite extensive primate and human research, it is not fully understood how these features are represented in the SMC and how they lead to the appropriate movement. Yet, the available information may provide insight into which features are most suitable for BCI control. To that purpose, the current paper provides an in-depth review on the movement features encoded in the SMC. Even though there is no consensus on how exactly the SMC generates movement, we conclude that some parameters are well represented in the SMC and can be accurately used for BCI control with discrete as well as continuous feedback. However, the vast evidence also suggests that movement should be interpreted as a combination of multiple parameters rather than isolated ones, pleading for further exploration of sensorimotor control models for accurate BCI control.

Epidural recordings in cochlear implant users

OBJECTIVE

In the long term it is desirable for CI users to control their device via brain signals. A possible strategy is the use of auditory evoked potentials (AEPs). Several studies have shown the suitability of auditory paradigms for such an approach. However, these investigations are based on non-invasive recordings. When thinking about everyday life applications, it would be more convenient to use implanted electrodes for signal acquisition. Ideally, the electrodes would be directly integrated into the CI. Further it is to be expected that invasively recorded signals have higher signal quality and are less affected by artifacts.

APPROACH

In this project we investigated the feasibility of implanting epidural electrodes temporarily during CI surgery and the possibility to record AEPs in the course of several days after implantation. Intraoperatively, auditory brainstem responses were recorded, whereas various kinds of AEPs were recorded postoperatively. After a few days the epidural electrodes were removed.

MAIN RESULTS

Data sets of ten subjects were obtained. Invasively recorded potentials were compared subjectively and objectively to clinical standard recordings using surface electrodes. Especially the cortical evoked response audiometry depicted clearer N1 waves for the epidural electrodes which were also visible at lower stimulation intensities compared to scalp electrodes. Furthermore the signal was less disturbed by artifacts. The objective quality measure (based on data sets of six patients) showed a significant better signal quality for the epidural compared to the scalp recordings.

SIGNIFICANCE

Altogether the approach revealed to be feasible and well tolerated by the patients. The epidural recordings showed a clearly better signal quality than the scalp recordings with AEPs being clearer recognizable. The results of the present study suggest that including epidural recording electrodes in future CI systems will improve the everyday life applicability of auditory closed loop systems for CI subjects.

Progress in the Field of Micro-Electrocorticography

Since the 1940s electrocorticography (ECoG) devices and, more recently, in the last decade, micro-electrocorticography (µECoG) cortical electrode arrays were used for a wide set of experimental and clinical applications, such as epilepsy localization and brain⁻computer interface (BCI) technologies. Miniaturized implantable µECoG devices have the advantage of providing greater-density neural signal acquisition and stimulation capabilities in a minimally invasive fashion. An increased spatial resolution of the µECoG array will be useful for greater specificity diagnosis and treatment of neuronal diseases and the advancement of basic neuroscience and BCI research. In this review, recent achievements of ECoG and µECoG are discussed. The electrode configurations and varying material choices used to design µECoG arrays are discussed, including advantages and disadvantages of µECoG technology compared to electroencephalography (EEG), ECoG, and intracortical electrode arrays. Electrode materials that are the primary focus include platinum, iridium oxide, poly(3,4-ethylenedioxythiophene) (PEDOT), indium tin oxide (ITO), and graphene. We discuss the biological immune response to µECoG devices compared to other electrode array types, the role of µECoG in clinical pathology, and brain⁻computer interface technology. The information presented in this review will be helpful to understand the current status, organize available knowledge, and guide future clinical and research applications of µECoG technologies.

The influence of prior pronunciations on sensorimotor cortex activity patterns during vowel production

OBJECTIVE

In recent years, brain-computer interface (BCI) systems have been investigated for their potential as a communication device to assist people with severe paralysis. Decoding speech sensorimotor cortex activity is a promising avenue for the generation of BCI control signals, but is complicated by variability in neural patterns, leading to suboptimal decoding. We investigated whether neural pattern variability associated with sound pronunciation can be explained by prior pronunciations and determined to what extent prior speech affects BCI decoding accuracy.

APPROACH

Neural patterns in speech motor areas were evaluated with electrocorticography in five epilepsy patients, who performed a simple speech task that involved pronunciation of the /i/ sound, preceded by either silence, the /a/ sound or the /u/ sound.

MAIN RESULTS

The neural pattern related to the /i/ sound depends on previous sounds and is therefore associated with multiple distinct sensorimotor patterns, which is likely to reflect differences in the movements towards this sound. We also show that these patterns still contain a commonality that is distinct from the other vowel sounds (/a/ and /u/). Classification accuracies for the decoding of different sounds do increase, however, when the multiple patterns for the /i/ sound are taken into account. Simply including multiple forms of the /i/ vowel in the training set for the creation of a single /i/ model performs as well as training individual models for each /i/ variation.

SIGNIFICANCE

Our results are of interest for the development of BCIs that aim to decode speech sounds from the sensorimotor cortex, since they argue that a multitude of cortical activity patterns associated with speech movements can be reduced to a basis set of models which reflect meaningful language units (vowels), yet it is important to account for the variety of neural patterns associated with a single sound in the training process.

Rapid Decoding of Hand Gestures in Electrocorticography Using Recurrent Neural Networks

Brain-computer interface (BCI) is a direct communication pathway between brain and external devices, and BCI-based prosthetic devices are promising to provide new rehabilitation options for people with motor disabilities. Electrocorticography (ECoG) signals contain rich information correlated with motor activities, and have great potential in hand gesture decoding. However, most existing decoders use long time windows, thus ignore the temporal dynamics within the period. In this study, we propose to use recurrent neural networks (RNNs) to exploit the temporal information in ECoG signals for robust hand gesture decoding. With RNN's high nonlinearity modeling ability, our method can effectively capture the temporal information in ECoG time series for robust gesture recognition. In the experiments, we decode three hand gestures using ECoG signals of two participants, and achieve an accuracy of 90%. Specially, we investigate the possibility of recognizing the gestures in a time interval as short as possible after motion onsets. Our method rapidly recognizes gestures within 0.5 s after motion onsets with an accuracy of about 80%. Experimental results also indicate that the temporal dynamics is especially informative for effective and rapid decoding of hand gestures.

Optimization of sampling rate and smoothing improves classification of high frequency power in electrocorticographic brain signals

Objective

High-frequency band (HFB) activity, measured using implanted sensors over the cortex, is increasingly considered as a feature for the study of brain function and the design of neural-implants, such as Brain-Computer Interfaces (BCIs). One common way of extracting these power signals is using a wavelet dictionary, which involves the selection of different temporal sampling and temporal smoothing parameters, such that the resulting HFB signal best represents the temporal features of the neuronal event of interest. Typically, the use of neuro-electrical signals for closed-loop BCI control requires a certain level of signal downsampling and smoothing in order to remove uncorrelated noise, optimize performance and provide fast feedback. However, a fixed setting of the sampling and smoothing parameters may lead to a suboptimal representation of the underlying neural responses and poor BCI control. This problem can be resolved with a systematic assessment of parameter settings.

Approach

With classification of HFB power responses as performance measure, different combinations of temporal sampling and temporal smoothing values were applied to data from sensory and motor tasks recorded with high-density and standard clinical electrocorticography (ECoG) grids in 12 epilepsy patients.

Main results

The results suggest that HFB ECoG responses are best performed with high sampling and subsequent smoothing. For the paradigms used in this study, optimal temporal sampling ranged from 29 Hz to 50 Hz. Regarding optimal smoothing, values were similar between tasks (0.1-0.9 s), except for executed complex hand gestures, for which two optimal possible smoothing windows were found (0.4-0.6 s and 0.9-2.7 s).

Significance

The range of optimal values indicates that parameter optimization depends on the functional paradigm and may be subject-specific. Our results advocate a methodical assessment of parameter settings for optimal decodability of ECoG signals.

Sub-millimeter ECoG pitch in human enables higher fidelity cognitive neural state estimation

Electrocorticography (ECoG), electrophysiological recording from the pial surface of the brain, is a critical measurement technique for clinical neurophysiology, basic neurophysiology studies, and demonstrates great promise for the development of neural prosthetic devices for assistive applications and the treatment of neurological disorders. Recent advances in device engineering are poised to enable orders of magnitude increase in the resolution of ECoG without comprised measurement quality. This enhancement in cortical sensing enables the observation of neural dynamics from the cortical surface at the micrometer scale. While these technical capabilities may be enabling, the extent to which finer spatial scale recording enhances functionally relevant neural state inference is unclear. We examine this question by employing a high-density and low impedance 400 μm pitch microECoG (μECoG) grid to record neural activity from the human cortical surface during cognitive tasks. By applying machine learning techniques to classify task conditions from the envelope of high-frequency band (70-170Hz) neural activity collected from two study participants, we demonstrate that higher density grids can lead to more accurate binary task condition classification. When controlling for grid area and selecting task informative sub-regions of the complete grid, we observed a consistent increase in mean classification accuracy with higher grid density; in particular, 400 μm pitch grids outperforming spatially sub-sampled lower density grids up to 23%. We also introduce a modeling framework to provide intuition for how spatial properties of measurements affect the performance gap between high and low density grids. To our knowledge, this work is the first quantitative demonstration of human sub-millimeter pitch cortical surface recording yielding higher-fidelity state estimation relative to devices at the millimeter-scale, motivating the development and testing of μECoG for basic and clinical neurophysiology as well as towards the realization of high-performance neural prostheses.

Dexterous Control of Seven Functional Hand Movements Using Cortically-Controlled Transcutaneous Muscle Stimulation in a Person With Tetraplegia

Individuals with tetraplegia identify restoration of hand function as a critical, unmet need to regain their independence and improve quality of life. Brain-Computer Interface (BCI)-controlled Functional Electrical Stimulation (FES) technology addresses this need by reconnecting the brain with paralyzed limbs to restore function. In this study, we quantified performance of an intuitive, cortically-controlled, transcutaneous FES system on standardized object manipulation tasks from the Grasp and Release Test (GRT). We found that a tetraplegic individual could use the system to control up to seven functional hand movements, each with >95% individual accuracy. He was able to select one movement from the possible seven movements available to him and use it to appropriately manipulate all GRT objects in real-time using naturalistic grasps. With the use of the system, the participant not only improved his GRT performance over his baseline, demonstrating an increase in number of transfers for all objects except the Block, but also significantly improved transfer times for the heaviest objects (videocassette (VHS), Can). Analysis of underlying motor cortex neural representations associated with the hand grasp states revealed an overlap or non-separability in neural activation patterns for similarly shaped objects that affected BCI-FES performance. These results suggest that motor cortex neural representations for functional grips are likely more related to hand shape and force required to hold objects, rather than to the objects themselves. These results, demonstrating multiple, naturalistic functional hand movements with the BCI-FES, constitute a further step toward translating BCI-FES technologies from research devices to clinical neuroprosthetics.

A piecewise probabilistic regression model to decode hand movement trajectories from epidural and subdural ECoG signals

OBJECTIVE

The primary concern of this study is to develop a probabilistic regression method that would improve the decoding of the hand movement trajectories from epidural ECoG as well as from subdural ECoG signals.

APPROACH

The model is characterized by the conditional expectation of the hand position given the ECoG signals. The conditional expectation of the hand position is then modeled by a linear combination of the conditional probability density functions defined for each segment of the movement. Moreover, a spatial linear filter is proposed for reducing the dimension of the feature space. The spatial linear filter is applied to each frequency band of the ECoG signals and extract the features with highest decoding performance.

MAIN RESULTS

For evaluating the proposed method, a dataset including 28 ECoG recordings from four adult Japanese macaques is used. The results show that the proposed decoding method outperforms the results with respect to the state of the art methods using this dataset. The relative kinematic information of each frequency band is also investigated using mutual information and decoding performance. The decoding performance shows that the best performance was obtained for high gamma bands from 50 to 200 Hz as well as high frequency ECoG band from 200 to 400 Hz for subdural recordings. However, the decoding performance was decreased for these frequency bands using epidural recordings. The mutual information shows that, on average, the high gamma band from 50 to 200 Hz and high frequency ECoG band from 200 to 400 Hz contain significantly more information than the average of the rest of the frequency bands [Formula: see text] for both subdural and epidural recordings. The results of high resolution time-frequency analysis show that ERD/ERS patterns in all frequency bands could reveal the dynamics of the ECoG responses during the movement. The onset and offset of the movement can be clearly identified by the ERD/ERS patterns.

SIGNIFICANCE

Reliable decoding the kinematic information from the brain signals paves the way for robust control of external devices.

Characterization of Hand Clenching in Human Sensorimotor Cortex Using High-, and Ultra-High Frequency Band Modulations of Electrocorticogram

Functional mapping of eloquent cortex before the resection of a tumor is a critical procedure for optimizing survival and quality of life. In order to locate the hand area of the motor cortex in two patients with low-grade gliomas (LGG), we recorded electrocorticogram (ECoG) from a 113 channel hybrid high-density grid (64 large contacts with diameter of 2.7 mm and 49 small contacts with diameter of 1 mm) while they executed hand clenching movements. We investigated the spatio-spectral characteristics of the neural oscillatory activity and observed that, in both patients, the hand movements were consistently associated with a wide spread power decrease in the low frequency band (LFB: 8–32 Hz) and a more localized power increase in the high frequency band (HFB: 60–280 Hz) within the sensorimotor region. Importantly, we observed significant power increase in the ultra-high frequency band (UFB: 300–800 Hz) during hand movements of both patients within a restricted cortical region close to the central sulcus, and the motor cortical “hand knob.” Among all frequency bands we studied, the UFB modulations were closest to the central sulcus and direct cortical stimulation (DCS) positive site. Both HFB and UFB modulations exhibited different timing characteristics at different locations. Power increase in HFB and UFB starting before movement onset was observed mostly at the anterior part of the activated cortical region. In addition, the spatial patterns in HFB and UFB indicated a probable postcentral shift of the hand motor function in one of the patients. We also compared the task related subband modulations captured by the small and large contacts in our hybrid grid. We did not find any significant difference in terms of band power changes. This study shows initial evidence that event-driven neural oscillatory activity recorded from ECoG can reach up to 800 Hz. The spatial distribution of UFB oscillations was found to be more focalized and closer to the central sulcus compared to LFB and HFB. More studies are needed to characterize further the functional significance of UFB relative to LFB and HFB.

EEG-Based Brain-Computer Interfaces for Communication and Rehabilitation of People with Motor Impairment: A Novel Approach of the 21 st Century

People with severe neurological impairments face many challenges in sensorimotor functions and communication with the environment; therefore they have increased demand for advanced, adaptive and personalized rehabilitation. During the last several decades, numerous studies have developed brain-computer interfaces (BCIs) with the goals ranging from providing means of communication to functional rehabilitation. Here we review the research on non-invasive, electroencephalography (EEG)-based BCI systems for communication and rehabilitation. We focus on the approaches intended to help severely paralyzed and locked-in patients regain communication using three different BCI modalities: slow cortical potentials, sensorimotor rhythms and P300 potentials, as operational mechanisms. We also review BCI systems for restoration of motor function in patients with spinal cord injury and chronic stroke. We discuss the advantages and limitations of these approaches and the challenges that need to be addressed in the future.

Preservation of hand movement representation in the sensorimotor areas of amputees

Denervation due to amputation is known to induce cortical reorganization in the sensorimotor cortex. Although there is evidence that reorganization does not lead to a complete loss of the representation of the phantom limb, it is unclear to what extent detailed, finger-specific activation patterns are preserved in motor cortex, an issue that is also relevant for development of brain-computer interface solutions for paralysed people. We applied machine learning to obtain a quantitative measure for the functional organization within the motor and adjacent cortices in amputees, using high resolution functional MRI and attempted hand gestures. Subjects with above-elbow arm amputation (n = 8) and non-amputated controls (n = 9) made several gestures with either their right or left hand. Amputees attempted to make gestures with their amputated hand. Images were acquired using 7 T functional MRI. The sensorimotor cortex was divided into four regions, and activity patterns were classified in individual subjects using a support vector machine. Classification scores were significantly above chance for all subjects and all hands, and were highly similar between amputees and controls in most regions. Decodability of phantom movements from primary motor cortex reached the levels of right hand movements in controls. Attempted movements were successfully decoded from primary sensory cortex in amputees, albeit lower than in controls but well above chance level despite absence of somatosensory feedback. There was no significant correlation between decodability and years since amputation, or age. The ability to decode attempted gestures demonstrates that the detailed hand representation is preserved in motor cortex and adjacent regions after denervation. This encourages targeting sensorimotor activity patterns for development of brain-computer interfaces.

Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids

For people who cannot communicate due to severe paralysis or involuntary movements, technology that decodes intended speech from the brain may offer an alternative means of communication. If decoding proves to be feasible, intracranial Brain-Computer Interface systems can be developed which are designed to translate decoded speech into computer generated speech or to instructions for controlling assistive devices. Recent advances suggest that such decoding may be feasible from sensorimotor cortex, but it is not clear how this challenge can be approached best. One approach is to identify and discriminate elements of spoken language, such as phonemes. We investigated feasibility of decoding four spoken phonemes from the sensorimotor face area, using electrocorticographic signals obtained with high-density electrode grids. Several decoding algorithms including spatiotemporal matched filters, spatial matched filters and support vector machines were compared. Phonemes could be classified correctly at a level of over 75% with spatiotemporal matched filters. Support Vector machine analysis reached a similar level, but spatial matched filters yielded significantly lower scores. The most informative electrodes were clustered along the central sulcus. Highest scores were achieved from time windows centered around voice onset time, but a 500 ms window before onset time could also be classified significantly. The results suggest that phoneme production involves a sequence of robust and reproducible activity patterns on the cortical surface. Importantly, decoding requires inclusion of temporal information to capture the rapid shifts of robust patterns associated with articulator muscle group contraction during production of a phoneme. The high classification scores are likely to be enabled by the use of high density grids, and by the use of discrete phonemes. Implications for use in Brain-Computer Interfaces are discussed.

Movement-Related Sensorimotor High-Gamma Activity Mainly Represents Somatosensory Feedback

Somatosensation plays pivotal roles in the everyday motor control of humans. During active movement, there exists a prominent high-gamma (HG >50 Hz) power increase in the primary somatosensory cortex (S1), and this provides an important feature in relation to the decoding of movement in a brain-machine interface (BMI). However, one concern of BMI researchers is the inflation of the decoding performance due to the activation of somatosensory feedback, which is not elicited in patients who have lost their sensorimotor function. In fact, it is unclear as to how much the HG component activated in S1 contributes to the overall sensorimotor HG power during voluntary movement. With regard to other functional roles of HG in S1, recent findings have reported that these HG power levels increase before the onset of actual movement, which implies neural activation for top-down movement preparation or sensorimotor interaction, i.e., an efference copy. These results are promising for BMI applications but remain inconclusive. Here, we found using electrocorticography (ECoG) from eight patients that HG activation in S1 is stronger and more informative than it is in the primary motor cortex (M1) regardless of the type of movement. We also demonstrate by means of electromyography (EMG) that the onset timing of the HG power in S1 is later (49 ms) than that of the actual movement. Interestingly, we show that the HG power fluctuations in S1 are closely related to subtle muscle contractions, even during the pre-movement period. These results suggest the following: (1) movement-related HG activity in S1 strongly affects the overall sensorimotor HG power, and (2) HG activity in S1 during voluntary movement mainly represents cortical neural processing for somatosensory feedback.

Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation

Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain's body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.