Point-and-click cursor control with an intracortical neural interface system by humans with tetraplegia

A brain machine interface framework for exploring proactive control of smart environments

Brain machine interfaces (BMIs) can substantially improve the quality of life of elderly or disabled people. However, performing complex action sequences with a BMI system is onerous because it requires issuing commands sequentially. Fundamentally different from this, we have designed a BMI system that reads out mental planning activity and issues commands in a proactive manner. To demonstrate this, we recorded brain activity from freely-moving monkeys performing an instructed task and decoded it with an energy-efficient, small and mobile field-programmable gate array hardware decoder triggering real-time action execution on smart devices. Core of this is an adaptive decoding algorithm that can compensate for the day-by-day neuronal signal fluctuations with minimal re-calibration effort. We show that open-loop planning-ahead control is possible using signals from primary and pre-motor areas leading to significant time-gain in the execution of action sequences. This novel approach provides, thus, a stepping stone towards improved and more humane control of different smart environments with mobile brain machine interfaces.

Reconnecting the Hand and Arm to the Brain: Efficacy of Neural Interfaces for Sensorimotor Restoration After Tetraplegia

BACKGROUND AND OBJECTIVES

Paralysis after spinal cord injury involves damage to pathways that connect neurons in the brain to peripheral nerves in the limbs. Re-establishing this communication using neural interfaces has the potential to bridge the gap and restore upper extremity function to people with high tetraplegia. We report a novel approach for restoring upper extremity function using selective peripheral nerve stimulation controlled by intracortical microelectrode recordings from sensorimotor networks, along with restoration of tactile sensation of the hand using intracortical microstimulation.

METHODS

A 27-year-old right-handed man with AIS-B (motor-complete, sensory-incomplete) C3-C4 tetraplegia was enrolled into the clinical trial. Six 64-channel intracortical microelectrode arrays were implanted into left hemisphere regions involved in upper extremity function, including primary motor and sensory cortices, inferior frontal gyrus, and anterior intraparietal area. Nine 16-channel extraneural peripheral nerve electrodes were implanted to allow targeted stimulation of right median, ulnar (2), radial, axillary, musculocutaneous, suprascapular, lateral pectoral, and long thoracic nerves, to produce selective muscle contractions on demand. Proof-of-concept studies were performed to demonstrate feasibility of using a brain-machine interface to read from and write to the brain for restoring motor and sensory functions of the participant's own arm and hand.

RESULTS

Multiunit neural activity that correlated with intended motor action was successfully recorded from intracortical arrays. Microstimulation of electrodes in somatosensory cortex produced repeatable sensory percepts of individual fingers for restoration of touch sensation. Selective electrical activation of peripheral nerves produced antigravity muscle contractions, resulting in functional movements that the participant was able to command under brain control to perform virtual and actual arm and hand movements. The system was well tolerated with no operative complications.

CONCLUSION

The combination of implanted cortical electrodes and nerve cuff electrodes has the potential to create bidirectional restoration of motor and sensory functions of the arm and hand after neurological injury.

An Investigation of Manifold-Based Direct Control for a Brain-to-Body Neural Bypass

Objective: Brain-body interfaces (BBIs) have emerged as a very promising solution for restoring voluntary hand control in people with upper-limb paralysis. The BBI module decoding motor commands from brain signals should provide the user with intuitive, accurate, and stable control. Here, we present a preliminary investigation in a monkey of a brain decoding strategy based on the direct coupling between the activity of intrinsic neural ensembles and output variables, aiming at achieving ease of learning and long-term robustness. Results: We identified an intrinsic low-dimensional space (called manifold) capturing the co-variation patterns of the monkey's neural activity associated to reach-to-grasp movements. We then tested the animal's ability to directly control a computer cursor using cortical activation along the manifold axes. By daily recalibrating only scaling factors, we achieved rapid learning and stable high performance in simple, incremental 2D tasks over more than 12 weeks of experiments. Finally, we showed that this brain decoding strategy can be effectively coupled to peripheral nerve stimulation to trigger voluntary hand movements. Conclusions: These results represent a proof of concept of manifold-based direct control for BBI applications.

Measuring instability in chronic human intracortical neural recordings towards stable, long-term brain-computer interfaces

Intracortical brain-computer interfaces (iBCIs) enable people with tetraplegia to gain intuitive cursor control from movement intentions. To translate to practical use, iBCIs should provide reliable performance for extended periods of time. However, performance begins to degrade as the relationship between kinematic intention and recorded neural activity shifts compared to when the decoder was initially trained. In addition to developing decoders to better handle long-term instability, identifying when to recalibrate will also optimize performance. We propose a method to measure instability in neural data without needing to label user intentions. Longitudinal data were analyzed from two BrainGate2 participants with tetraplegia as they used fixed decoders to control a computer cursor spanning 142 days and 28 days, respectively. We demonstrate a measure of instability that correlates with changes in closed-loop cursor performance solely based on the recorded neural activity (Pearson r = 0.93 and 0.72, respectively). This result suggests a strategy to infer online iBCI performance from neural data alone and to determine when recalibration should take place for practical long-term use.

Comparative analysis of energy transfer mechanisms for neural implants

As neural implant technologies advance rapidly, a nuanced understanding of their powering mechanisms becomes indispensable, especially given the long-term biocompatibility risks like oxidative stress and inflammation, which can be aggravated by recurrent surgeries, including battery replacements. This review delves into a comprehensive analysis, starting with biocompatibility considerations for both energy storage units and transfer methods. The review focuses on four main mechanisms for powering neural implants: Electromagnetic, Acoustic, Optical, and Direct Connection to the Body. Among these, Electromagnetic Methods include techniques such as Near-Field Communication (RF). Acoustic methods using high-frequency ultrasound offer advantages in power transmission efficiency and multi-node interrogation capabilities. Optical methods, although still in early development, show promising energy transmission efficiencies using Near-Infrared (NIR) light while avoiding electromagnetic interference. Direct connections, while efficient, pose substantial safety risks, including infection and micromotion disturbances within neural tissue. The review employs key metrics such as specific absorption rate (SAR) and energy transfer efficiency for a nuanced evaluation of these methods. It also discusses recent innovations like the Sectored-Multi Ring Ultrasonic Transducer (S-MRUT), Stentrode, and Neural Dust. Ultimately, this review aims to help researchers, clinicians, and engineers better understand the challenges of and potentially create new solutions for powering neural implants.

Decoding the Spike-Band Subthreshold Motor Cortical Activity

Intracortical Brain-Computer Interfaces (iBCI) use single-unit activity (SUA), multiunit activity (MUA) and local field potentials (LFP) to control neuroprosthetic devices. SUA and MUA are usually extracted from the bandpassed recording through amplitude thresholding, while subthreshold data are ignored. Here, we show that subthreshold data can actually be decoded to determine behavioral variables with test set accuracy of up to 100%. Although the utility of SUA, MUA and LFP for decoding behavioral variables has been explored previously, this study investigates the utility of spike-band subthreshold activity exclusively. We provide evidence suggesting that this activity can be used to keep decoding performance at acceptable levels even when SUA quality is reduced over time. To the best of our knowledge, the signals that we derive from the subthreshold activity may be the weakest neural signals that have ever been extracted from extracellular neural recordings, while still being decodable with test set accuracy of up to 100%. These results are relevant for the development of fully data-driven and automated methods for amplitude thresholding spike-band extracellular neural recordings in iBCIs containing thousands of electrodes.

Using principles of motor control to analyze performance of human machine interfaces

There have been significant advances in biosignal extraction techniques to drive external biomechatronic devices or to use as inputs to sophisticated human machine interfaces. The control signals are typically derived from biological signals such as myoelectric measurements made either from the surface of the skin or subcutaneously. Other biosignal sensing modalities are emerging. With improvements in sensing modalities and control algorithms, it is becoming possible to robustly control the target position of an end-effector. It remains largely unknown to what extent these improvements can lead to naturalistic human-like movement. In this paper, we sought to answer this question. We utilized a sensing paradigm called sonomyography based on continuous ultrasound imaging of forearm muscles. Unlike myoelectric control strategies which measure electrical activation and use the extracted signals to determine the velocity of an end-effector; sonomyography measures muscle deformation directly with ultrasound and uses the extracted signals to proportionally control the position of an end-effector. Previously, we showed that users were able to accurately and precisely perform a virtual target acquisition task using sonomyography. In this work, we investigate the time course of the control trajectories derived from sonomyography. We show that the time course of the sonomyography-derived trajectories that users take to reach virtual targets reflect the trajectories shown to be typical for kinematic characteristics observed in biological limbs. Specifically, during a target acquisition task, the velocity profiles followed a minimum jerk trajectory shown for point-to-point arm reaching movements, with similar time to target. In addition, the trajectories based on ultrasound imaging result in a systematic delay and scaling of peak movement velocity as the movement distance increased. We believe this is the first evaluation of similarities in control policies in coordinated movements in jointed limbs, and those based on position control signals extracted at the individual muscle level. These results have strong implications for the future development of control paradigms for assistive technologies.

Using Principles of Motor Control to Analyze Performance of Human Machine Interfaces

There have been significant advances in biosignal extraction techniques to drive external biomechatronic devices or to use as inputs to sophisticated human machine interfaces. The control signals are typically derived from biological signals such as myoelectric measurements made either from the surface of the skin or subcutaneously. Other biosignal sensing modalities are emerging. With improvements in sensing modalities and control algorithms, it is becoming possible to robustly control the target position of a end effector. It remains largely unknown to what extent these improvements can lead to naturalistic human-like movement. In this paper, we sought to answer this question. We utilized a sensing paradigm called sonomyography based on continuous ultrasound imaging of forearm muscles. Unlike myoelectric control strategies which measure electrical activation and use the extracted signals to determine the velocity of an end-effector; sonomyography measures muscle deformation directly with ultrasound and uses the extracted signals to proportionally control the position of an end-effector. Previously, we showed that users were able to accurately and precisely perform a virtual target acquisition task using sonomyography. In this work, we investigate the time course of the control trajectories derived from sonomyography. We show that the time course of the sonomyography-derived trajectories that users take to reach virtual targets reflect the trajectories shown to be typical for kinematic characteristics observed in biological limbs. Specifically, during a target acquisition task, the velocity profiles followed a minimum jerk trajectory shown for point-to-point arm reaching movements, with similar time to target. In addition, the trajectories based on ultrasound imaging result in a systematic delay and scaling of peak movement velocity as the movement distance increased. We believe this is the first evaluation of similarities in control policies in coordinated movements in jointed limbs, and those based on position control signals extracted at the individual muscle level. These results have strong implications for the future development of control paradigms for assistive technologies.

Reconnecting the Hand and Arm to the Brain: Efficacy of Neural Interfaces for Sensorimotor Restoration after Tetraplegia

Background

Paralysis after spinal cord injury involves damage to pathways that connect neurons in the brain to peripheral nerves in the limbs. Re-establishing this communication using neural interfaces has the potential to bridge the gap and restore upper extremity function to people with high tetraplegia.

Objective

We report a novel approach for restoring upper extremity function using selective peripheral nerve stimulation controlled by intracortical microelectrode recordings from sensorimotor networks, along with restoration of tactile sensation of the hand using intracortical microstimulation.

Methods

A right-handed man with motor-complete C3-C4 tetraplegia was enrolled into the clinical trial. Six 64-channel intracortical microelectrode arrays were implanted into left hemisphere regions involved in upper extremity function, including primary motor and sensory cortices, inferior frontal gyrus, and anterior intraparietal area. Nine 16-channel extraneural peripheral nerve electrodes were implanted to allow targeted stimulation of right median, ulnar (2), radial, axillary, musculocutaneous, suprascapular, lateral pectoral, and long thoracic nerves, to produce selective muscle contractions on demand. Proof-of-concept studies were performed to demonstrate feasibility of a bidirectional brain-machine interface to restore function of the participant's own arm and hand.

Results

Multi-unit neural activity that correlated with intended motor action was successfully recorded from intracortical arrays. Microstimulation of electrodes in somatosensory cortex produced repeatable sensory percepts of individual fingers for restoration of touch sensation. Selective electrical activation of peripheral nerves produced antigravity muscle contractions. The system was well tolerated with no operative complications.

Conclusion

The combination of implanted cortical electrodes and nerve cuff electrodes has the potential to allow restoration of motor and sensory functions of the arm and hand after neurological injury.

Translational opportunities and challenges of invasive electrodes for neural interfaces

Invasive brain-machine interfaces can restore motor, sensory and cognitive functions. However, their clinical adoption has been hindered by the surgical risk of implantation and by suboptimal long-term reliability. In this Review, we highlight the opportunities and challenges of invasive technology for clinically relevant electrophysiology. Specifically, we discuss the characteristics of neural probes that are most likely to facilitate the clinical translation of invasive neural interfaces, describe the neural signals that can be acquired or produced by intracranial electrodes, the abiotic and biotic factors that contribute to their failure, and emerging neural-interface architectures.

Learning to control a BMI-driven wheelchair for people with severe tetraplegia

Mind-controlled wheelchairs are an intriguing assistive mobility solution applicable in complete paralysis. Despite progress in brain-machine interface (BMI) technology, its translation remains elusive. The primary objective of this study is to probe the hypothesis that BMI skill acquisition by end-users is fundamental to control a non-invasive brain-actuated intelligent wheelchair in real-world settings. We demonstrate that three tetraplegic spinal-cord injury users could be trained to operate a non-invasive, self-paced thought-controlled wheelchair and execute complex navigation tasks. However, only the two users exhibiting increasing decoding performance and feature discriminancy, significant neuroplasticity changes and improved BMI command latency, achieved high navigation performance. In addition, we show that dexterous, continuous control of robots is possible through low-degree of freedom, discrete and uncertain control channels like a motor imagery BMI, by blending human and artificial intelligence through shared-control methodologies. We posit that subject learning and shared-control are the key components paving the way for translational non-invasive BMI.

Validation of a non-invasive, real-time, human-in-the-loop model of intracortical brain-computer interfaces

Objective. Despite the tremendous promise of invasive brain-computer interfaces (iBCIs), the associated study costs, risks, and ethical considerations limit the opportunity to develop and test the algorithms that decode neural activity into a user's intentions. Our goal was to address this challenge by designing an iBCI model capable of testing many human subjects in closed-loop.Approach. We developed an iBCI model that uses artificial neural networks (ANNs) to translate human finger movements into realistic motor cortex firing patterns, which can then be decoded in real time. We call the model the joint angle BCI, or jaBCI. jaBCI allows readily recruited, healthy subjects to perform closed-loop iBCI tasks using any neural decoder, preserving subjects' control-relevant short-latency error correction and learning dynamics.Main results. We validated jaBCI offline through emulated neuron firing statistics, confirming that emulated neural signals have firing rates, low-dimensional PCA geometry, and rotational jPCA dynamics that are quite similar to the actual neurons (recorded in monkey M1) on which we trained the ANN. We also tested jaBCI in closed-loop experiments, our single study examining roughly as many subjects as have been tested world-wide with iBCIs (n= 25). Performance was consistent with that of the paralyzed, human iBCI users with implanted intracortical electrodes. jaBCI allowed us to imitate the experimental protocols (e.g. the same velocity Kalman filter decoder and center-out task) and compute the same seven behavioral measures used in three critical studies.Significance. These encouraging results suggest the jaBCI's real-time firing rate emulation is a useful means to provide statistically robust sample sizes for rapid prototyping and optimization of decoding algorithms, the study of bi-directional learning in iBCIs, and improving iBCI control.

Design-development of an at-home modular brain-computer interface (BCI) platform in a case study of cervical spinal cord injury

OBJECTIVE

The objective of this study was to develop a portable and modular brain-computer interface (BCI) software platform independent of input and output devices. We implemented this platform in a case study of a subject with cervical spinal cord injury (C5 ASIA A).

BACKGROUND

BCIs can restore independence for individuals with paralysis by using brain signals to control prosthetics or trigger functional electrical stimulation. Though several studies have successfully implemented this technology in the laboratory and the home, portability, device configuration, and caregiver setup remain challenges that limit deployment to the home environment. Portability is essential for transitioning BCI from the laboratory to the home.

METHODS

The BCI platform implementation consisted of an Activa PC + S generator with two subdural four-contact electrodes implanted over the dominant left hand-arm region of the sensorimotor cortex, a minicomputer fixed to the back of the subject's wheelchair, a custom mobile phone application, and a mechanical glove as the end effector. To quantify the performance for this at-home implementation of the BCI, we quantified system setup time at home, chronic (14-month) decoding accuracy, hardware and software profiling, and Bluetooth communication latency between the App and the minicomputer. We created a dataset of motor-imagery labeled signals to train a binary motor imagery classifier on a remote computer for online, at-home use.

RESULTS

Average bluetooth data transmission delay between the minicomputer and mobile App was 23 ± 0.014 ms. The average setup time for the subject's caregiver was 5.6 ± 0.83 min. The average times to acquire and decode neural signals and to send those decoded signals to the end-effector were respectively 404.1 ms and 1.02 ms. The 14-month median accuracy of the trained motor imagery classifier was 87.5 ± 4.71% without retraining.

CONCLUSIONS

The study presents the feasibility of an at-home BCI system that subjects can seamlessly operate using a friendly mobile user interface, which does not require daily calibration nor the presence of a technical person for at-home setup. The study also describes the portability of the BCI system and the ability to plug-and-play multiple end effectors, providing the end-user the flexibility to choose the end effector to accomplish specific motor tasks for daily needs. Trial registration ClinicalTrials.gov: NCT02564419. First posted on 9/30/2015.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

Cyclic, Condition-Independent Activity in Primary Motor Cortex Predicts Corrective Movement Behavior

Reaching movements are known to have large condition-independent (CI) neural activity and cyclic neural dynamics. A new precision center-out task was performed by rhesus macaques to test the hypothesis that cyclic, CI neural activity in the primary motor cortex (M1) occurs not only during initial reaching movements but also during subsequent corrective movements. Corrective movements were observed to be discrete with time courses and bell-shaped speed profiles similar to the initial movements. CI cyclic neural trajectories were similar and repeated for initial and each additional corrective submovement. The phase of the cyclic CI neural activity predicted the time of peak movement speed more accurately than regression of instantaneous firing rate, even when the subject made multiple corrective movements. Rather than being controlled as continuations of the initial reach, a discrete cycle of motor cortex activity encodes each corrective submovement.

Generalizable cursor click decoding using grasp-related neural transients

Objective.Intracortical brain-computer interfaces (iBCI) have the potential to restore independence for individuals with significant motor or communication impairments. One of the most realistic avenues for clinical translation of iBCI technology is enabling control of a computer cursor-i.e. movement-related neural activity is interpreted (decoded) and used to drive cursor function. Here we aim to improve cursor click decoding to allow for both point-and-click and click-and-drag control.Approach.Using chronic microelectrode arrays implanted in the motor cortex of two participants with tetraplegia, we identified prominent neural responses related to attempted hand grasp. We then developed a new approach for decoding cursor click (hand grasp) based on the most salient responses.Main results.We found that the population-wide response contained three dominant components related to hand grasp: an onset transient response, a sustained response, and an offset transient response. The transient responses were larger in magnitude-and thus more reliably detected-than the sustained response, and a click decoder based on these transients outperformed the standard approach of binary state classification.Significance.A transient-based approach for identifying hand grasp can provide a high degree of cursor click control for both point-and-click and click-and-drag applications. This generalized click functionality is an important step toward high-performance cursor control and eventual clinical translation of iBCI technology.

Long-term intracortical microelectrode array performance in a human: a 5 year retrospective analysis

Objective. Brain-computer interfaces (BCIs) that record neural activity using intracortical microelectrode arrays (MEAs) have shown promise for mitigating disability associated with neurological injuries and disorders. While the chronic performance and failure modes of MEAs have been well studied and systematically described in non-human primates, there is far less reported about long-term MEA performance in humans. Our group has collected one of the largest neural recording datasets from a Utah MEA in a human subject, spanning over 5 years (2014-2019). Here we present both long-term signal quality and BCI performance as well as highlight several acute signal disruption events observed during the clinical study.Approach. Long-term Utah array performance was evaluated by analyzing neural signal metric trends and decoding accuracy for tasks regularly performed across 448 clinical recording sessions. For acute signal disruptions, we identify or hypothesize the root cause of the disruption, show how the disruption manifests in the collected data, and discuss potential identification and mitigation strategies for the disruption.Main results. Neural signal quality metrics deteriorated rapidly within the first year, followed by a slower decline through the remainder of the study. Nevertheless, BCI performance remained high 5 years after implantation, which is encouraging for the translational potential of this technology as an assistive device. We also present examples of unanticipated signal disruptions during chronic MEA use, which are critical to detect as BCI technology progresses toward home usage.Significance. Our work fills a gap in knowledge around long-term MEA performance in humans, providing longevity and efficacy data points to help characterize the performance of implantable neural sensors in a human population. The trial was registered on ClinicalTrials.gov (Identifier NCT01997125) and conformed to institutional requirements for the conduct of human subjects research.

Home Use of a Percutaneous Wireless Intracortical Brain-Computer Interface by Individuals With Tetraplegia

OBJECTIVE

Individuals with neurological disease or injury such as amyotrophic lateral sclerosis, spinal cord injury or stroke may become tetraplegic, unable to speak or even locked-in. For people with these conditions, current assistive technologies are often ineffective. Brain-computer interfaces are being developed to enhance independence and restore communication in the absence of physical movement. Over the past decade, individuals with tetraplegia have achieved rapid on-screen typing and point-and-click control of tablet apps using intracortical brain-computer interfaces (iBCIs) that decode intended arm and hand movements from neural signals recorded by implanted microelectrode arrays. However, cables used to convey neural signals from the brain tether participants to amplifiers and decoding computers and require expert oversight, severely limiting when and where iBCIs could be available for use. Here, we demonstrate the first human use of a wireless broadband iBCI.

METHODS

Based on a prototype system previously used in pre-clinical research, we replaced the external cables of a 192-electrode iBCI with wireless transmitters and achieved high-resolution recording and decoding of broadband field potentials and spiking activity from people with paralysis. Two participants in an ongoing pilot clinical trial completed on-screen item selection tasks to assess iBCI-enabled cursor control.

RESULTS

Communication bitrates were equivalent between cabled and wireless configurations. Participants also used the wireless iBCI to control a standard commercial tablet computer to browse the web and use several mobile applications. Within-day comparison of cabled and wireless interfaces evaluated bit error rate, packet loss, and the recovery of spike rates and spike waveforms from the recorded neural signals. In a representative use case, the wireless system recorded intracortical signals from two arrays in one participant continuously through a 24-hour period at home.

SIGNIFICANCE

Wireless multi-electrode recording of broadband neural signals over extended periods introduces a valuable tool for human neuroscience research and is an important step toward practical deployment of iBCI technology for independent use by individuals with paralysis. On-demand access to high-performance iBCI technology in the home promises to enhance independence and restore communication and mobility for individuals with severe motor impairment.

The Neural Representation of Force across Grasp Types in Motor Cortex of Humans with Tetraplegia

Intracortical brain-computer interfaces (iBCIs) have the potential to restore hand grasping and object interaction to individuals with tetraplegia. Optimal grasping and object interaction require simultaneous production of both force and grasp outputs. However, since overlapping neural populations are modulated by both parameters, grasp type could affect how well forces are decoded from motor cortex in a closed-loop force iBCI. Therefore, this work quantified the neural representation and offline decoding performance of discrete hand grasps and force levels in two human participants with tetraplegia. Participants attempted to produce three discrete forces (light, medium, hard) using up to five hand grasp configurations. A two-way Welch ANOVA was implemented on multiunit neural features to assess their modulation to force and grasp Demixed principal component analysis (dPCA) was used to assess for population-level tuning to force and grasp and to predict these parameters from neural activity. Three major findings emerged from this work: (1) force information was neurally represented and could be decoded across multiple hand grasps (and, in one participant, across attempted elbow extension as well); (2) grasp type affected force representation within multiunit neural features and offline force classification accuracy; and (3) grasp was classified more accurately and had greater population-level representation than force. These findings suggest that force and grasp have both independent and interacting representations within cortex, and that incorporating force control into real-time iBCI systems is feasible across multiple hand grasps if the decoder also accounts for grasp type.

Auditory cues reveal intended movement information in middle frontal gyrus neuronal ensemble activity of a person with tetraplegia

Intracortical brain-computer interfaces (iBCIs) allow people with paralysis to directly control assistive devices using neural activity associated with the intent to move. Realizing the full potential of iBCIs critically depends on continued progress in understanding how different cortical areas contribute to movement control. Here we present the first comparison between neuronal ensemble recordings from the left middle frontal gyrus (MFG) and precentral gyrus (PCG) of a person with tetraplegia using an iBCI. As expected, PCG was more engaged in selecting and generating intended movements than in earlier perceptual stages of action planning. By contrast, MFG displayed movement-related information during the sensorimotor processing steps preceding the appearance of the action plan in PCG, but only when the actions were instructed using auditory cues. These results describe a previously unreported function for neurons in the human left MFG in auditory processing contributing to motor control.

Decoding hand kinematics from population responses in sensorimotor cortex during grasping

OBJECTIVE

The hand-a complex effector comprising dozens of degrees of freedom of movement-endows us with the ability to flexibly, precisely, and effortlessly interact with objects. The neural signals associated with dexterous hand movements in primary motor cortex (M1) and somatosensory cortex (SC) have received comparatively less attention than have those associated with proximal upper limb control.

APPROACH

To fill this gap, we trained two monkeys to grasp objects varying in size and shape while tracking their hand postures and recording single-unit activity from M1 and SC. We then decoded their hand kinematics across tens of joints from population activity in these areas.

MAIN RESULTS

We found that we could accurately decode kinematics with a small number of neural signals and that different cortical fields carry different amounts of information about hand kinematics. In particular, neural signals in rostral M1 led to better performance than did signals in caudal M1, whereas Brodmann's area 3a outperformed areas 1 and 2 in SC. Moreover, decoding performance was higher for joint angles than joint angular velocities, in contrast to what has been found with proximal limb decoders.

SIGNIFICANCE

We conclude that cortical signals can be used for dexterous hand control in brain machine interface applications and that postural representations in SC may be exploited via intracortical stimulation to close the sensorimotor loop.

Mapping of primary somatosensory cortex of the hand area using a high-density electrocorticography grid for closed-loop brain computer interface

The ideal modality for generating sensation in sensorimotor brain computer interfaces (BCI) has not been determined. Here we report the feasibility of using a high-density "mini"-electrocorticography (mECoG) grid in a somatosensory BCI system. Thirteen subjects with intractable epilepsy underwent standard clinical implantation of subdural electrodes for the purpose of seizure localization. An additional high-density mECoG grid was placed (Adtech, 8 by 8, 1.2-mm exposed, 3-mm center-to-center spacing) over the hand area of primary somatosensory cortex. Following implantation, cortical mapping was performed with stimulation parameters of frequency: 50 Hz, pulse-width: 250 µs, pulse duration: 4 s, polarity: alternating, and current that ranged from 0.5 mA to 12 mA at the discretion of the epileptologist. Location of the evoked sensory percepts was recorded along with a description of the sensation. The hand was partitioned into 48 distinct boxes. A box was included if sensation was felt anywhere within the box. The percentage of the hand covered was 63.9% (± 34.4%) (mean ± s.d.). Mean redundancy, measured as electrode pairs stimulating the same box, was 1.9 (± 2.2) electrodes per box; and mean resolution, measured as boxes included per electrode pair stimulation, was 11.4 (± 13.7) boxes with 8.1 (± 10.7) boxes in the digits and 3.4 (± 6.0) boxes in the palm. Functional utility of the system was assessed by quantifying usable percepts. Under the strictest classification, "dermatomally exclusive" percepts, the mean was 2.8 usable percepts per grid. Allowing "perceptually unique" percepts at the same anatomical location, the mean was 5.5 usable percepts per grid. Compared to the small area of coverage and redundancy of a microelectrode system, or the poor resolution of a standard ECoG grid, a mECoG is likely the best modality for a somatosensory BCI system with good coverage of the hand and minimal redundancy.

Postural control of arm and fingers through integration of movement commands

Every movement ends in a period of stillness. Current models assume that commands that hold the limb at a target location do not depend on the commands that moved the limb to that location. Here, we report a surprising relationship between movement and posture in primates: on a within-trial basis, the commands that hold the arm and finger at a target location depend on the mathematical integration of the commands that moved the limb to that location. Following damage to the corticospinal tract, both the move and hold period commands become more variable. However, the hold period commands retain their dependence on the integral of the move period commands. Thus, our data suggest that the postural controller possesses a feedforward module that uses move commands to calculate a component of hold commands. This computation may arise within an unknown subcortical system that integrates cortical commands to stabilize limb posture.

Neural Representation of Observed, Imagined, and Attempted Grasping Force in Motor Cortex of Individuals with Chronic Tetraplegia

Hybrid kinetic and kinematic intracortical brain-computer interfaces (iBCIs) have the potential to restore functional grasping and object interaction capabilities in individuals with tetraplegia. This requires an understanding of how kinetic information is represented in neural activity, and how this representation is affected by non-motor parameters such as volitional state (VoS), namely, whether one observes, imagines, or attempts an action. To this end, this work investigates how motor cortical neural activity changes when three human participants with tetraplegia observe, imagine, and attempt to produce three discrete hand grasping forces with the dominant hand. We show that force representation follows the same VoS-related trends as previously shown for directional arm movements; namely, that attempted force production recruits more neural activity compared to observed or imagined force production. Additionally, VoS-modulated neural activity to a greater extent than grasping force. Neural representation of forces was lower than expected, possibly due to compromised somatosensory pathways in individuals with tetraplegia, which have been shown to influence motor cortical activity. Nevertheless, attempted forces (but not always observed or imagined forces) could be decoded significantly above chance, thereby potentially providing relevant information towards the development of a hybrid kinetic and kinematic iBCI.

Neurorestoration: Advances in human brain–computer interface using microelectrode arrays

Neural damage has been a great challenge to the medical field for a very long time. The emergence of brain–computer interfaces (BCIs) offered a new possibility to enhance the activity of daily living and provide a new formation of entertainment for those with disabilities. Intracortical BCIs, which require the implantation of microelectrodes, can receive neuronal signals with a high spatial and temporal resolution from the individual’s cortex. When BCI decoded cortical signals and mapped them to external devices, it displayed the ability not only to replace part of the human motor function but also to help individuals restore certain neurological functions. In this review, we focus on human intracortical BCI research using microelectrode arrays and summarize the main directions and the latest results in this field. In general, we found that intracortical BCI research based on motor neuroprosthetics and functional electrical stimulation have already achieved some simple functional replacement and treatment of motor function. Pioneering work in the posterior parietal cortex has given us a glimpse of the potential that intracortical BCIs have to control external devices and receive various sensory information.

Longitudinal multimodal assessment of neurodegeneration and vascular remodeling correlated with signal degradation in chronic cortical silicon microelectrodes

Significance: Cortically implanted microelectrode arrays provide a direct interface with neuronal populations and are used to restore movement capabilities and provide sensory feedback to patients with paralysis or amputation. Penetrating electrodes experience high rates of signal degradation within the first year that limit effectiveness and lead to eventual device failure. Aim: To assess vascular and neuronal changes over time in mice with implanted electrodes and examine the contribution of the brain tissue response to electrode performance. Approach: We used a multimodal approach combining in vivo electrophysiology and subcellular-level optical imaging. Results: At acute timescales, we observed structural damage from the mechanical trauma of electrode insertion, evidenced by severed dendrites in the electrode path and local hypofluorescence. Superficial vessel growth and remodeling occurred within the first few weeks in both electrode-implanted and window-only animals, but the deeper capillary growth evident in window-only animals was suppressed in electrode-implanted animals. After longer implantation periods, there was evidence of degeneration of transected dendrites superficial to the electrode path and localized neuronal cell body loss, along with deep vascular velocity changes near the electrode. Total spike rate (SR) across all animals reached a peak between 3 and 9 months postimplantation, then decreased. The local field potential signal remained relatively constant for up to 6 months, particularly in the high-gamma band, indicating long-term electrode viability and neuronal functioning at further distances from the electrode, but it showed a reduction in some animals at later time points. Most importantly, we found that progressive high-gamma and SR reductions both correlate positively with localized cell loss and decreasing capillary density within 100    μ m of the electrode. Conclusions: This multifaceted approach provided a more comprehensive picture of the ongoing biological response at the brain-electrode interface than can be achieved with postmortem histology alone and established a real-time relationship between electrophysiology and tissue damage.

Brain-computer interfaces for communication

Locked-in syndrome (LIS) is characterized by an inability to move or speak in the presence of intact cognition and can be caused by brainstem trauma or neuromuscular disease. Quality of life (QoL) in LIS is strongly impaired by the inability to communicate, which cannot always be remedied by traditional augmentative and alternative communication (AAC) solutions if residual muscle activity is insufficient to control the AAC device. Brain-computer interfaces (BCIs) may offer a solution by employing the person's neural signals instead of relying on muscle activity. Here, we review the latest communication BCI research using noninvasive signal acquisition approaches (electroencephalography, functional magnetic resonance imaging, functional near-infrared spectroscopy) and subdural and intracortical implanted electrodes, and we discuss current efforts to translate research knowledge into usable BCI-enabled communication solutions that aim to improve the QoL of individuals with LIS.

Towards Intelligent Intracortical BMI (i 2BMI): Low-Power Neuromorphic Decoders That Outperform Kalman Filters

Fully-implantable wireless intracortical Brain Machine Interfaces (iBMI) is one of the most promising next frontiers in the nascent field of neurotechnology. However, scaling the number of channels in such systems by another 10× is difficult due to power and bandwidth requirements of the wireless transmitter. One promising solution for that is to include more processing, up to the decoder, in the implant so that transmission data-rate is reduced drastically. Earlier work on neuromorphic decoder chips only showed classification of discrete states. We present results for continuous state decoding using a low-power neuromorphic decoder chip termed Spike-input Extreme Learning Machine (SELMA) that implements a nonlinear decoder without memory and its memory-based version with time-delayed bins, SELMA-bins. We have compared SELMA, SELMA-bins against state-of-the-art Steady-State Kalman Filter (SSKF), a linear decoder with memory, across two different datasets involving a total of 4 non-human primates (NHPs). Results show at least a 10% (20%) or more increase in the fraction of variance accounted for (FVAF) by SELMA (SELMA-bins) over SSKF across the datasets. Estimated energy consumption comparison shows SELMA (SELMA-bins) consuming ≈ 9 nJ/update (23 nJ/update) against SSKF's ≈ 7.4 nJ/update for an iBMI with a 10 degree of freedom control. Thus, SELMA yields better performance against SSKF while consuming energy in the same range as SSKF whereas SELMA-bins performs the best with moderately increased energy consumption, albeit far less than energy required for raw data transmission. This paves the way for reducing transmission data rates in future scaled iBMI systems.

Human motor decoding from neural signals: a review

Many people suffer from movement disability due to amputation or neurological diseases. Fortunately, with modern neurotechnology now it is possible to intercept motor control signals at various points along the neural transduction pathway and use that to drive external devices for communication or control. Here we will review the latest developments in human motor decoding. We reviewed the various strategies to decode motor intention from human and their respective advantages and challenges. Neural control signals can be intercepted at various points in the neural signal transduction pathway, including the brain (electroencephalography, electrocorticography, intracortical recordings), the nerves (peripheral nerve recordings) and the muscles (electromyography). We systematically discussed the sites of signal acquisition, available neural features, signal processing techniques and decoding algorithms in each of these potential interception points. Examples of applications and the current state-of-the-art performance were also reviewed. Although great strides have been made in human motor decoding, we are still far away from achieving naturalistic and dexterous control like our native limbs. Concerted efforts from material scientists, electrical engineers, and healthcare professionals are needed to further advance the field and make the technology widely available in clinical use.

Decoding Native Cortical Representations for Flexion and Extension at Upper Limb Joints Using Electrocorticography

Brain-machine interface (BMI) researchers have traditionally focused on modeling endpoint reaching tasks to provide the control of neurally driven prosthetic arms. Most previous research has focused on achieving an endpoint control through a Cartesian-coordinate-centered approach. However, a joint-centered approach could potentially be used to intuitively control a wide range of limb movements. We systematically investigated the feasibility of discriminating between flexion and extension of different upper limb joints using electrocorticography(ECoG) recordings from sensorimotor cortex. Four subjects implanted with macro-ECoG (10-mm spacing), high-density ECoG (5-mm spacing), and/or micro-ECoG arrays (0.9-mm spacing and 4 mm × 4 mm coverage), performed randomly cued flexions or extensions of the fingers, wrist, or elbow contralateral to the implanted hemisphere. We trained a linear model to classify six movements using averaged high-gamma power (70-110 Hz) modulations at different latencies with respect to movement onset, and within a time interval restricted to flexion or extension at each joint. Offline decoding models for each subject classified these movements with accuracies of 62%-83%. Our results suggest that the widespread ECoG coverage of sensorimotor cortex could allow a whole limb BMI to sample native cortical representations in order to control flexion and extension at multiple joints.

Cortical control of a tablet computer by people with paralysis

General-purpose computers have become ubiquitous and important for everyday life, but they are difficult for people with paralysis to use. Specialized software and personalized input devices can improve access, but often provide only limited functionality. In this study, three research participants with tetraplegia who had multielectrode arrays implanted in motor cortex as part of the BrainGate2 clinical trial used an intracortical brain-computer interface (iBCI) to control an unmodified commercial tablet computer. Neural activity was decoded in real time as a point-and-click wireless Bluetooth mouse, allowing participants to use common and recreational applications (web browsing, email, chatting, playing music on a piano application, sending text messages, etc.). Two of the participants also used the iBCI to "chat" with each other in real time. This study demonstrates, for the first time, high-performance iBCI control of an unmodified, commercially available, general-purpose mobile computing device by people with tetraplegia.

Motor Control and Sensory Feedback Enhance Prosthesis Embodiment and Reduce Phantom Pain After Long-Term Hand Amputation

We quantified prosthesis embodiment and phantom pain reduction associated with motor control and sensory feedback from a prosthetic hand in one human with a long-term transradial amputation. Microelectrode arrays were implanted in the residual median and ulnar arm nerves and intramuscular electromyography recording leads were implanted in residual limb muscles to enable sensory feedback and motor control. Objective measures (proprioceptive drift) and subjective measures (survey answers) were used to assess prosthesis embodiment. For both measures, there was a significant level of embodiment of the physical prosthetic limb after open-loop motor control of the prosthesis (i.e., without sensory feedback), open-loop sensation from the prosthesis (i.e., without motor control), and closed-loop control of the prosthesis (i.e., motor control with sensory feedback). There was also a statistically significant reduction in reported phantom pain after experimental sessions that included open-loop nerve microstimulation, open-loop prosthesis motor control, or closed-loop prosthesis motor control. The closed-loop condition provided no additional significant improvements in phantom pain reduction or prosthesis embodiment relative to the open-loop sensory condition or the open-loop motor condition. This study represents the first long-term (14-month), systematic report of phantom pain reduction and prosthesis embodiment in a human amputee across a variety of prosthesis use cases.

Data-Driven Transducer Design and Identification for Internally-Paced Motor Brain Computer Interfaces: A Review

Brain-Computer Interfaces (BCIs) are systems that establish a direct communication pathway between the users' brain activity and external effectors. They offer the potential to improve the quality of life of motor-impaired patients. Motor BCIs aim to permit severely motor-impaired users to regain limb mobility by controlling orthoses or prostheses. In particular, motor BCI systems benefit patients if the decoded actions reflect the users' intentions with an accuracy that enables them to efficiently interact with their environment. One of the main challenges of BCI systems is to adapt the BCI's signal translation blocks to the user to reach a high decoding accuracy. This paper will review the literature of data-driven and user-specific transducer design and identification approaches and it focuses on internally-paced motor BCIs. In particular, continuous kinematic biomimetic and mental-task decoders are reviewed. Furthermore, static and dynamic decoding approaches, linear and non-linear decoding, offline and real-time identification algorithms are considered. The current progress and challenges related to the design of clinical-compatible motor BCI transducers are additionally discussed.

Extracting wavelet based neural features from human intracortical recordings for neuroprosthetics applications

Background

Understanding the long-term behavior of intracortically-recorded signals is essential for improving the performance of Brain Computer Interfaces. However, few studies have systematically investigated chronic neural recordings from an implanted microelectrode array in the human brain.

Methods

In this study, we show the applicability of wavelet decomposition method to extract and demonstrate the utility of long-term stable features in neural signals obtained from a microelectrode array implanted in the motor cortex of a human with tetraplegia. Wavelet decomposition was applied to the raw voltage data to generate mean wavelet power (MWP) features, which were further divided into three sub-frequency bands, low-frequency MWP (lf-MWP, 0–234 Hz), mid-frequency MWP (mf-MWP, 234 Hz–3.75 kHz) and high-frequency MWP (hf-MWP, >3.75 kHz). We analyzed these features using data collected from two experiments that were repeated over the course of about 3 years and compared their signal stability and decoding performance with the more standard threshold crossings, local field potentials (LFP), multi-unit activity (MUA) features obtained from the raw voltage recordings.

Results

All neural features could stably track neural information for over 3 years post-implantation and were less prone to signal degradation compared to threshold crossings. Furthermore, when used as an input to support vector machine based decoding algorithms, the mf-MWP and MUA demonstrated significantly better performance, respectively, in classifying imagined motor tasks than using the lf-MWP, hf-MWP, LFP, or threshold crossings.

Conclusions

Our results suggest that using MWP features in the appropriate frequency bands can provide an effective neural feature for brain computer interface intended for chronic applications.

Trial registration

This study was approved by the U.S. Food and Drug Administration (Investigational Device Exemption) and the Ohio State University Medical Center Institutional Review Board (Columbus, Ohio). The study conformed to institutional requirements for the conduct of human subjects and was filed on ClinicalTrials.gov (Identifier NCT01997125).

Electronic supplementary material

The online version of this article (10.1186/s42234-018-0011-x) contains supplementary material, which is available to authorized users.

Engineering Artificial Somatosensation Through Cortical Stimulation in Humans

Sensory feedback is a critical aspect of motor control rehabilitation following paralysis or amputation. Current human studies have demonstrated the ability to deliver some of this sensory information via brain-machine interfaces, although further testing is needed to understand the stimulation parameters effect on sensation. Here, we report a systematic evaluation of somatosensory restoration in humans, using cortical stimulation with subdural mini-electrocorticography (mini-ECoG) grids. Nine epilepsy patients undergoing implantation of cortical electrodes for seizure localization were also implanted with a subdural 64-channel mini-ECoG grid over the hand area of the primary somatosensory cortex (S1). We mapped the somatotopic location and size of receptive fields evoked by stimulation of individual channels of the mini-ECoG grid. We determined the effects on perception by varying stimulus parameters of pulse width, current amplitude, and frequency. Finally, a target localization task was used to demonstrate the use of artificial sensation in a behavioral task. We found a replicable somatotopic representation of the hand on the mini-ECoG grid across most subjects during electrical stimulation. The stimulus-evoked sensations were usually of artificial quality, but in some cases were more natural and of a cutaneous or proprioceptive nature. Increases in pulse width, current strength and frequency generally produced similar quality sensations at the same somatotopic location, but with a perception of increased intensity. The subjects produced near perfect performance when using the evoked sensory information in target acquisition tasks. These findings indicate that electrical stimulation of somatosensory cortex through mini-ECoG grids has considerable potential for restoring useful sensation to patients with paralysis and amputation.

A piecewise probabilistic regression model to decode hand movement trajectories from epidural and subdural ECoG signals

OBJECTIVE

The primary concern of this study is to develop a probabilistic regression method that would improve the decoding of the hand movement trajectories from epidural ECoG as well as from subdural ECoG signals.

APPROACH

The model is characterized by the conditional expectation of the hand position given the ECoG signals. The conditional expectation of the hand position is then modeled by a linear combination of the conditional probability density functions defined for each segment of the movement. Moreover, a spatial linear filter is proposed for reducing the dimension of the feature space. The spatial linear filter is applied to each frequency band of the ECoG signals and extract the features with highest decoding performance.

MAIN RESULTS

For evaluating the proposed method, a dataset including 28 ECoG recordings from four adult Japanese macaques is used. The results show that the proposed decoding method outperforms the results with respect to the state of the art methods using this dataset. The relative kinematic information of each frequency band is also investigated using mutual information and decoding performance. The decoding performance shows that the best performance was obtained for high gamma bands from 50 to 200 Hz as well as high frequency ECoG band from 200 to 400 Hz for subdural recordings. However, the decoding performance was decreased for these frequency bands using epidural recordings. The mutual information shows that, on average, the high gamma band from 50 to 200 Hz and high frequency ECoG band from 200 to 400 Hz contain significantly more information than the average of the rest of the frequency bands [Formula: see text] for both subdural and epidural recordings. The results of high resolution time-frequency analysis show that ERD/ERS patterns in all frequency bands could reveal the dynamics of the ECoG responses during the movement. The onset and offset of the movement can be clearly identified by the ERD/ERS patterns.

SIGNIFICANCE

Reliable decoding the kinematic information from the brain signals paves the way for robust control of external devices.

Restoration of Hindlimb Movements after Complete Spinal Cord Injury Using Brain-Controlled Functional Electrical Stimulation

Single neuron and local field potential signals recorded in the primary motor cortex have been repeatedly demonstrated as viable control signals for multi-degree-of-freedom actuators. Although the primary source of these signals has been fore/upper limb motor regions, recent evidence suggests that neural adaptation underlying neuroprosthetic control is generalizable across cortex, including hindlimb sensorimotor cortex. Here, adult rats underwent a longitudinal study that included a hindlimb pedal press task in response to cues for specific durations, followed by brain machine interface (BMI) tasks in healthy rats, after rats received a complete spinal transection and after the BMI signal controls epidural stimulation (BMI-FES). Over the course of the transition from learned behavior to BMI task, fewer neurons were responsive after the cue, the proportion of neurons selective for press duration increased and these neurons carried more information. After a complete, mid-thoracic spinal lesion that completely severed both ascending and descending connections to the lower limbs, there was a reduction in task-responsive neurons followed by a reacquisition of task selectivity in recorded populations. This occurred due to a change in pattern of neuronal responses not simple changes in firing rate. Finally, during BMI-FES, additional information about the intended press duration was produced. This information was not dependent on the stimulation, which was the same for short and long duration presses during the early phase of stimulation, but instead was likely due to sensory feedback to sensorimotor cortex in response to movement along the trunk during the restored pedal press. This post-cue signal could be used as an error signal in a continuous decoder providing information about the position of the limb to optimally control a neuroprosthetic device.

An Intention-Driven Semi-autonomous Intelligent Robotic System for Drinking

In this study, an intention-driven semi-autonomous intelligent robotic (ID-SIR) system is designed and developed to assist the severely disabled patients to live independently. The system mainly consists of a non-invasive brain–machine interface (BMI) subsystem, a robot manipulator and a visual detection and localization subsystem. Different from most of the existing systems remotely controlled by joystick, head- or eye tracking, the proposed ID-SIR system directly acquires the intention from users’ brain. Compared with the state-of-art system only working for a specific object in a fixed place, the designed ID-SIR system can grasp any desired object in a random place chosen by a user and deliver it to his/her mouth automatically. As one of the main advantages of the ID-SIR system, the patient is only required to send one intention command for one drinking task and the autonomous robot would finish the rest of specific controlling tasks, which greatly eases the burden on patients. Eight healthy subjects attended our experiment, which contained 10 tasks for each subject. In each task, the proposed ID-SIR system delivered the desired beverage container to the mouth of the subject and then put it back to the original position. The mean accuracy of the eight subjects was 97.5%, which demonstrated the effectiveness of the ID-SIR system.

Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.

High performance communication by people with paralysis using an intracortical brain-computer interface

Brain-computer interfaces (BCIs) have the potential to restore communication for people with tetraplegia and anarthria by translating neural activity into control signals for assistive communication devices. While previous pre-clinical and clinical studies have demonstrated promising proofs-of-concept (Serruya et al., 2002; Simeral et al., 2011; Bacher et al., 2015; Nuyujukian et al., 2015; Aflalo et al., 2015; Gilja et al., 2015; Jarosiewicz et al., 2015; Wolpaw et al., 1998; Hwang et al., 2012; Spüler et al., 2012; Leuthardt et al., 2004; Taylor et al., 2002; Schalk et al., 2008; Moran, 2010; Brunner et al., 2011; Wang et al., 2013; Townsend and Platsko, 2016; Vansteensel et al., 2016; Nuyujukian et al., 2016; Carmena et al., 2003; Musallam et al., 2004; Santhanam et al., 2006; Hochberg et al., 2006; Ganguly et al., 2011; O'Doherty et al., 2011; Gilja et al., 2012), the performance of human clinical BCI systems is not yet high enough to support widespread adoption by people with physical limitations of speech. Here we report a high-performance intracortical BCI (iBCI) for communication, which was tested by three clinical trial participants with paralysis. The system leveraged advances in decoder design developed in prior pre-clinical and clinical studies (Gilja et al., 2015; Kao et al., 2016; Gilja et al., 2012). For all three participants, performance exceeded previous iBCIs (Bacher et al., 2015; Jarosiewicz et al., 2015) as measured by typing rate (by a factor of 1.4-4.2) and information throughput (by a factor of 2.2-4.0). This high level of performance demonstrates the potential utility of iBCIs as powerful assistive communication devices for people with limited motor function.Clinical Trial No: NCT00912041.

Retrospectively supervised click decoder calibration for self-calibrating point-and-click brain-computer interfaces

Brain-computer interfaces (BCIs) aim to restore independence to people with severe motor disabilities by allowing control of acursor on a computer screen or other effectors with neural activity. However, physiological and/or recording-related nonstationarities in neural signals can limit long-term decoding stability, and it would be tedious for users to pause use of the BCI whenever neural control degrades to perform decoder recalibration routines. We recently demonstrated that a kinematic decoder (i.e. a decoder that controls cursor movement) can be recalibrated using data acquired during practical point-and-click control of the BCI by retrospectively inferring users' intended movement directions based on their subsequent selections. Here, we extend these methods to allow the click decoder to also be recalibrated using data acquired during practical BCI use. We retrospectively labeled neural data patterns as corresponding to "click" during all time bins in which the click log-likelihood (decoded using linear discriminant analysis, or LDA) had been above the click threshold that was used during real-time neural control. We labeled as "non-click" those periods that the kinematic decoder's retrospective target inference (RTI) heuristics determined to be consistent with intended cursor movement. Once these neural activity patterns were labeled, the click decoder was calibrated using standard supervised classifier training methods. Combined with real-time bias correction and baseline firing rate tracking, this set of "retrospectively labeled" decoder calibration methods enabled a BrainGate participant with amyotrophic lateral sclerosis (T9) to type freely across 11 research sessions spanning 29days, maintaining high-performance neural control over cursor movement and click without needing to interrupt virtual keyboard use for explicit calibration tasks. By eliminating the need for tedious calibration tasks with prescribed targets and pre-specified click times, this approach advances the potential clinical utility of intracortical BCIs for individuals with severe motor disability.

Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface

Brain-computer interfaces (BCIs) promise to restore independence for people with severe motor disabilities by translating decoded neural activity directly into the control of a computer. However, recorded neural signals are not stationary (that is, can change over time), degrading the quality of decoding. Requiring users to pause what they are doing whenever signals change to perform decoder recalibration routines is time-consuming and impractical for everyday use of BCIs. We demonstrate that signal nonstationarity in an intracortical BCI can be mitigated automatically in software, enabling long periods (hours to days) of self-paced point-and-click typing by people with tetraplegia, without degradation in neural control. Three key innovations were included in our approach: tracking the statistics of the neural activity during self-timed pauses in neural control, velocity bias correction during neural control, and periodically recalibrating the decoder using data acquired during typing by mapping neural activity to movement intentions that are inferred retrospectively based on the user's self-selected targets. These methods, which can be extended to a variety of neurally controlled applications, advance the potential for intracortical BCIs to help restore independent communication and assistive device control for people with paralysis.

A High-Performance Neural Prosthesis Incorporating Discrete State Selection With Hidden Markov Models

Communication neural prostheses aim to restore efficient communication to people with motor neurological injury or disease by decoding neural activity into control signals. These control signals are both analog (e.g., the velocity of a computer mouse) and discrete (e.g., clicking an icon with a computer mouse) in nature. Effective, high-performing, and intuitive-to-use communication prostheses should be capable of decoding both analog and discrete state variables seamlessly. However, to date, the highest-performing autonomous communication prostheses rely on precise analog decoding and typically do not incorporate high-performance discrete decoding. In this report, we incorporated a hidden Markov model (HMM) into an intracortical communication prosthesis to enable accurate and fast discrete state decoding in parallel with analog decoding. In closed-loop experiments with nonhuman primates implanted with multielectrode arrays, we demonstrate that incorporating an HMM into a neural prosthesis can increase state-of-the-art achieved bitrate by 13.9% and 4.2% in two monkeys ( ). We found that the transition model of the HMM is critical to achieving this performance increase. Further, we found that using an HMM resulted in the highest achieved peak performance we have ever observed for these monkeys, achieving peak bitrates of 6.5, 5.7, and 4.7 bps in Monkeys J, R, and L, respectively. Finally, we found that this neural prosthesis was robustly controllable for the duration of entire experimental sessions. These results demonstrate that high-performance discrete decoding can be beneficially combined with analog decoding to achieve new state-of-the-art levels of performance.

Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves

OBJECTIVE

An important goal of neuroprosthetic research is to establish bidirectional communication between the user and new prosthetic limbs that are capable of controlling >20 different movements. One strategy for achieving this goal is to interface the prosthetic limb directly with efferent and afferent fibres in the peripheral nervous system using an array of intrafascicular microelectrodes. This approach would provide access to a large number of independent neural pathways for controlling high degree-of-freedom prosthetic limbs, as well as evoking multiple-complex sensory percepts.

APPROACH

Utah Slanted Electrode Arrays (USEAs, 96 recording/stimulating electrodes) were implanted for 30 days into the median (Subject 1-M, 31 years post-amputation) or ulnar (Subject 2-U, 1.5 years post-amputation) nerves of two amputees. Neural activity was recorded during intended movements of the subject's phantom fingers and a linear Kalman filter was used to decode the neural data. Microelectrode stimulation of varying amplitudes and frequencies was delivered via single or multiple electrodes to investigate the number, size and quality of sensory percepts that could be evoked. Device performance over time was assessed by measuring: electrode impedances, signal-to-noise ratios (SNRs), stimulation thresholds, number and stability of evoked percepts.

MAIN RESULTS

The subjects were able to proportionally, control individual fingers of a virtual robotic hand, with 13 different movements decoded offline (r = 0.48) and two movements decoded online. Electrical stimulation across one USEA evoked >80 sensory percepts. Varying the stimulation parameters modulated percept quality. Devices remained intrafascicularly implanted for the duration of the study with no significant changes in the SNRs or percept thresholds.

SIGNIFICANCE

This study demonstrated that an array of 96 microelectrodes can be implanted into the human peripheral nervous system for up to 1 month durations. Such an array could provide intuitive control of a virtual prosthetic hand with broad sensory feedback.

Brain-state classification and a dual-state decoder dramatically improve the control of cursor movement through a brain-machine interface

OBJECTIVE

It is quite remarkable that brain machine interfaces (BMIs) can be used to control complex movements with fewer than 100 neurons. Success may be due in part to the limited range of dynamical conditions under which most BMIs are tested. Achieving high-quality control that spans these conditions with a single linear mapping will be more challenging. Even for simple reaching movements, existing BMIs must reduce the stochastic noise of neurons by averaging the control signals over time, instead of over the many neurons that normally control movement. This forces a compromise between a decoder with dynamics allowing rapid movement and one that allows postures to be maintained with little jitter. Our current work presents a method for addressing this compromise, which may also generalize to more highly varied dynamical situations, including movements with more greatly varying speed.

APPROACH

We have developed a system that uses two independent Wiener filters as individual components in a single decoder, one optimized for movement, and the other for postural control. We computed an LDA classifier using the same neural inputs. The decoder combined the outputs of the two filters in proportion to the likelihood assigned by the classifier to each state.

MAIN RESULTS

We have performed online experiments with two monkeys using this neural-classifier, dual-state decoder, comparing it to a standard, single-state decoder as well as to a dual-state decoder that switched states automatically based on the cursor's proximity to a target. The performance of both monkeys using the classifier decoder was markedly better than that of the single-state decoder and comparable to the proximity decoder.

SIGNIFICANCE

We have demonstrated a novel strategy for dealing with the need to make rapid movements while also maintaining precise cursor control when approaching and stabilizing within targets. Further gains can undoubtedly be realized by optimizing the performance of the individual movement and posture decoders.

Clinical translation of a high-performance neural prosthesis

Neural prostheses have the potential to improve the quality of life of individuals with paralysis by directly mapping neural activity to limb- and computer-control signals. We translated a neural prosthetic system previously developed in animal model studies for use by two individuals with amyotrophic lateral sclerosis who had intracortical microelectrode arrays placed in motor cortex. Measured more than 1 year after implant, the neural cursor-control system showed the highest published performance achieved by a person to date, more than double that of previous pilot clinical trial participants.