Give me a sign: decoding four complex hand gestures based on high-density ECoG

Workshops of the Sixth International Brain-Computer Interface Meeting: brain-computer interfaces past, present, and future

The Sixth International Brain-Computer Interface (BCI) Meeting was held 30 May-3 June 2016 at the Asilomar Conference Grounds, Pacific Grove, California, USA. The conference included 28 workshops covering topics in BCI and brain-machine interface research. Topics included BCI for specific populations or applications, advancing BCI research through use of specific signals or technological advances, and translational and commercial issues to bring both implanted and non-invasive BCIs to market. BCI research is growing and expanding in the breadth of its applications, the depth of knowledge it can produce, and the practical benefit it can provide both for those with physical impairments and the general public. Here we provide summaries of each workshop, illustrating the breadth and depth of BCI research and highlighting important issues and calls for action to support future research and development.

Decoding hand gestures from primary somatosensory cortex using high-density ECoG

Electrocorticography (ECoG) based Brain-Computer Interfaces (BCIs) have been proposed as a way to restore and replace motor function or communication in severely paralyzed people. To date, most motor-based BCIs have either focused on the sensorimotor cortex as a whole or on the primary motor cortex (M1) as a source of signals for this purpose. Still, target areas for BCI are not confined to M1, and more brain regions may provide suitable BCI control signals. A logical candidate is the primary somatosensory cortex (S1), which not only shares similar somatotopic organization to M1, but also has been suggested to have a role beyond sensory feedback during movement execution. Here, we investigated whether four complex hand gestures, taken from the American sign language alphabet, can be decoded exclusively from S1 using both spatial and temporal information. For decoding, we used the signal recorded from a small patch of cortex with subdural high-density (HD) grids in five patients with intractable epilepsy. Notably, we introduce a new method of trial alignment based on the increase of the electrophysiological response, which virtually eliminates the confounding effects of systematic and non-systematic temporal differences within and between gestures execution. Results show that S1 classification scores are high (76%), similar to those obtained from M1 (74%) and sensorimotor cortex as a whole (85%), and significantly above chance level (25%). We conclude that S1 offers characteristic spatiotemporal neuronal activation patterns that are discriminative between gestures, and that it is possible to decode gestures with high accuracy from a very small patch of cortex using subdurally implanted HD grids. The feasibility of decoding hand gestures using HD-ECoG grids encourages further investigation of implantable BCI systems for direct interaction between the brain and external devices with multiple degrees of freedom.

Continuous decoding of human grasp kinematics using epidural and subdural signals

OBJECTIVE

Restoring or replacing function in paralyzed individuals will one day be achieved through the use of brain-machine interfaces. Regaining hand function is a major goal for paralyzed patients. Two competing prerequisites for the widespread adoption of any hand neuroprosthesis are accurate control over the fine details of movement, and minimized invasiveness. Here, we explore the interplay between these two goals by comparing our ability to decode hand movements with subdural and epidural field potentials (EFPs).

APPROACH

We measured the accuracy of decoding continuous hand and finger kinematics during naturalistic grasping motions in five human subjects. We recorded subdural surface potentials (electrocorticography; ECoG) as well as with EFPs, with both standard- and high-resolution electrode arrays.

MAIN RESULTS

In all five subjects, decoding of continuous kinematics significantly exceeded chance, using either EGoG or EFPs. ECoG decoding accuracy compared favorably with prior investigations of grasp kinematics (mean ± SD grasp aperture variance accounted for was 0.54 ± 0.05 across all subjects, 0.75 ± 0.09 for the best subject). In general, EFP decoding performed comparably to ECoG decoding. The 7-20 Hz and 70-115 Hz spectral bands contained the most information about grasp kinematics, with the 70-115 Hz band containing greater information about more subtle movements. Higher-resolution recording arrays provided clearly superior performance compared to standard-resolution arrays.

SIGNIFICANCE

To approach the fine motor control achieved by an intact brain-body system, it will be necessary to execute motor intent on a continuous basis with high accuracy. The current results demonstrate that this level of accuracy might be achievable not just with ECoG, but with EFPs as well. Epidural placement of electrodes is less invasive, and therefore may incur less risk of encephalitis or stroke than subdural placement of electrodes. Accurately decoding motor commands at the epidural level may be an important step towards a clinically viable brain-machine interface.

Individual finger control of a modular prosthetic limb using high-density electrocorticography in a human subject

OBJECTIVE

We used native sensorimotor representations of fingers in a brain-machine interface (BMI) to achieve immediate online control of individual prosthetic fingers.

APPROACH

Using high gamma responses recorded with a high-density electrocorticography (ECoG) array, we rapidly mapped the functional anatomy of cued finger movements. We used these cortical maps to select ECoG electrodes for a hierarchical linear discriminant analysis classification scheme to predict: (1) if any finger was moving, and, if so, (2) which digit was moving. To account for sensory feedback, we also mapped the spatiotemporal activation elicited by vibrotactile stimulation. Finally, we used this prediction framework to provide immediate online control over individual fingers of the Johns Hopkins University Applied Physics Laboratory modular prosthetic limb.

MAIN RESULTS

The balanced classification accuracy for detection of movements during the online control session was 92% (chance: 50%). At the onset of movement, finger classification was 76% (chance: 20%), and 88% (chance: 25%) if the pinky and ring finger movements were coupled. Balanced accuracy of fully flexing the cued finger was 64%, and 77% had we combined pinky and ring commands. Offline decoding yielded a peak finger decoding accuracy of 96.5% (chance: 20%) when using an optimized selection of electrodes. Offline analysis demonstrated significant finger-specific activations throughout sensorimotor cortex. Activations either prior to movement onset or during sensory feedback led to discriminable finger control.

SIGNIFICANCE

Our results demonstrate the ability of ECoG-based BMIs to leverage the native functional anatomy of sensorimotor cortical populations to immediately control individual finger movements in real time.

Comparison of decoding resolution of standard and high-density electrocorticogram electrodes

OBJECTIVE

Electrocorticography (ECoG)-based brain-computer interface (BCI) is a promising platform for controlling arm prostheses. To restore functional independence, a BCI must be able to control arm prostheses along at least six degrees-of-freedoms (DOFs). Prior studies suggest that standard ECoG grids may be insufficient to decode multi-DOF arm movements. This study compared the ability of standard and high-density (HD) ECoG grids to decode the presence/absence of six elementary arm movements and the type of movement performed.

APPROACH

Three subjects implanted with standard grids (4 mm diameter, 10 mm spacing) and three with HD grids (2 mm diameter, 4 mm spacing) had ECoG signals recorded while performing the following movements: (1) pincer grasp/release, (2) wrist flexion/extension, (3) pronation/supination, (4) elbow flexion/extension, (5) shoulder internal/external rotation, and (6) shoulder forward flexion/extension. Data from the primary motor cortex were used to train a state decoder to detect the presence/absence of movement, and a six-class decoder to distinguish between these movements.

MAIN RESULTS

The average performances of the state decoders trained on HD ECoG data were superior (p = 3.05 × 10(-5)) to those of their standard grid counterparts across all combinations of the μ, β, low-γ, and high-γ frequency bands. The average best decoding error for HD grids was 2.6%, compared to 8.5% of standard grids (chance 50%). The movement decoders trained on HD ECoG data were superior (p = 3.05 × 10(-5)) to those based on standard ECoG across all band combinations. The average best decoding errors of 11.9% and 33.1% were obtained for HD and standard grids, respectively (chance error 83.3%). These improvements can be attributed to higher electrode density and signal quality of HD grids.

SIGNIFICANCE

Commonly used ECoG grids are inadequate for multi-DOF BCI arm prostheses. The performance gains by HD grids may eventually lead to independence-restoring BCI arm prosthesis.

Current Challenges Facing the Translation of Brain Computer Interfaces from Preclinical Trials to Use in Human Patients

Current research in brain computer interface (BCI) technology is advancing beyond preclinical studies, with trials beginning in human patients. To date, these trials have been carried out with several different types of recording interfaces. The success of these devices has varied widely, but different factors such as the level of invasiveness, timescale of recorded information, and ability to maintain stable functionality of the device over a long period of time all must be considered in addition to accuracy in decoding intent when assessing the most practical type of device moving forward. Here, we discuss various approaches to BCIs, distinguishing between devices focusing on control of operations extrinsic to the subject (e.g., prosthetic limbs, computer cursors) and those focusing on control of operations intrinsic to the brain (e.g., using stimulation or external feedback), including closed-loop or adaptive devices. In this discussion, we consider the current challenges facing the translation of various types of BCI technology to eventual human application.

Classifying multiple types of hand motions using electrocorticography during intraoperative awake craniotomy and seizure monitoring processes-case studies

In this work, some case studies were conducted to classify several kinds of hand motions from electrocorticography (ECoG) signals during intraoperative awake craniotomy & extraoperative seizure monitoring processes. Four subjects (P1, P2 with intractable epilepsy during seizure monitoring and P3, P4 with brain tumor during awake craniotomy) participated in the experiments. Subjects performed three types of hand motions (Grasp, Thumb-finger motion and Index-finger motion) contralateral to the motor cortex covered with ECoG electrodes. Two methods were used for signal processing. Method I: autoregressive (AR) model with burg method was applied to extract features, and additional waveform length (WL) feature has been considered, finally the linear discriminative analysis (LDA) was used as the classifier. Method II: stationary subspace analysis (SSA) was applied for data preprocessing, and the common spatial pattern (CSP) was used for feature extraction before LDA decoding process. Applying method I, the three-class accuracy of P1~P4 were 90.17, 96.00, 91.77, and 92.95% respectively. For method II, the three-class accuracy of P1~P4 were 72.00, 93.17, 95.22, and 90.36% respectively. This study verified the possibility of decoding multiple hand motion types during an awake craniotomy, which is the first step toward dexterous neuroprosthetic control during surgical implantation, in order to verify the optimal placement of electrodes. The accuracy during awake craniotomy was comparable to results during seizure monitoring. This study also indicated that ECoG was a promising approach for precise identification of eloquent cortex during awake craniotomy, and might form a promising BCI system that could benefit both patients and neurosurgeons.