Decoding grasp and speech signals from the cortical grasp circuit in a tetraplegic human

Restoring Speech Using Brain-Computer Interfaces

People who have lost the ability to speak due to neurological injuries would greatly benefit from assistive technology that provides a fast, intuitive, and naturalistic means of communication. This need can be met with brain-computer interfaces (BCIs): medical devices that bypass injured parts of the nervous system and directly transform neural activity into outputs such as text or sound. BCIs for restoring movement and typing have progressed rapidly in recent clinical trials; speech BCIs are the next frontier. This review covers the clinical need for speech BCIs, surveys foundational studies that point to where and how speech can be decoded in the brain, describes recent progress in both discrete and continuous speech decoding and closed-loop speech BCIs, provides metrics for assessing these systems' performance, and highlights key remaining challenges on the road toward clinically useful speech neuroprostheses.

Multi-gesture drag-and-drop decoding in a 2D iBCI control task

Objective. Intracortical brain-computer interfaces (iBCIs) have demonstrated the ability to enable point and click as well as reach and grasp control for people with tetraplegia. However, few studies have investigated iBCIs during long-duration discrete movements that would enable common computer interactions such as 'click-and-hold' or 'drag-and-drop'.Approach. Here, we examined the performance of multi-class and binary (attempt/no-attempt) classification of neural activity in the left precentral gyrus of two BrainGate2 clinical trial participants performing hand gestures for 1, 2, and 4 s in duration. We then designed a novel 'latch decoder' that utilizes parallel multi-class and binary decoding processes and evaluated its performance on data from isolated sustained gesture attempts and a multi-gesture drag-and-drop task.Main results. Neural activity during sustained gestures revealed a marked decrease in the discriminability of hand gestures sustained beyond 1 s. Compared to standard direct decoding methods, the Latch decoder demonstrated substantial improvement in decoding accuracy for gestures performed independently or in conjunction with simultaneous 2D cursor control.Significance. This work highlights the unique neurophysiologic response patterns of sustained gesture attempts in human motor cortex and demonstrates a promising decoding approach that could enable individuals with tetraplegia to intuitively control a wider range of consumer electronics using an iBCI.

Neuronal responses in the human primary motor cortex coincide with the subjective onset of movement intention in brain-machine interface-mediated actions

Self-initiated behavior is accompanied by the experience of intending our actions. Here, we leverage the unique opportunity to examine the full intentional chain-from intention to action to environmental effects-in a tetraplegic person outfitted with a primary motor cortex (M1) brain-machine interface (BMI) generating real hand movements via neuromuscular electrical stimulation (NMES). This combined BMI-NMES approach allowed us to selectively manipulate each element of the intentional chain (intention, action, effect) while probing subjective experience and performing extra-cellular recordings in human M1. Behaviorally, we reveal a novel form of intentional binding: motor intentions are reflected in a perceived temporal attraction between the onset of intentions and that of actions. Neurally, we demonstrate that evoked spiking activity in M1 largely coincides in time with the onset of the experience of intention and that M1 spike counts and the onset of subjective intention may co-vary on a trial-by-trial basis. Further, population-level dynamics, as indexed by a decoder instantiating movement, reflect intention-action temporal binding. The results fill a significant knowledge gap by relating human spiking activity in M1 with the onset of subjective intention and complement prior human intracranial work examining pre-motor and parietal areas.

Invasive Brain-Computer Interface for Communication: A Scoping Review

BACKGROUND

The rapid expansion of the brain-computer interface for patients with neurological deficits has garnered significant interest, and for patients, it provides an additional route where conventional rehabilitation has its limits. This has particularly been the case for patients who lose the ability to communicate. Circumventing neural injuries by recording from the intact cortex and subcortex has the potential to allow patients to communicate and restore self-expression. Discoveries over the last 10-15 years have been possible through advancements in technology, neuroscience, and computing. By examining studies involving intracranial brain-computer interfaces that aim to restore communication, we aimed to explore the advances made and explore where the technology is heading.

METHODS

For this scoping review, we systematically searched PubMed and OVID Embase. After processing the articles, the search yielded 41 articles that we included in this review.

RESULTS

The articles predominantly assessed patients who had either suffered from amyotrophic lateral sclerosis, cervical cord injury, or brainstem stroke, resulting in tetraplegia and, in some cases, difficulty speaking. Of the intracranial implants, ten had ALS, six had brainstem stroke, and thirteen had a spinal cord injury. Stereoelectroencephalography was also used, but the results, whilst promising, are still in their infancy. Studies involving patients who were moving cursors on a screen could improve the speed of movement by optimising the interface and utilising better decoding methods. In recent years, intracortical devices have been successfully used for accurate speech-to-text and speech-to-audio decoding in patients who are unable to speak.

CONCLUSIONS

Here, we summarise the progress made by BCIs used for communication. Speech decoding directly from the cortex can provide a novel therapeutic method to restore full, embodied communication to patients suffering from tetraplegia who otherwise cannot communicate.

Decoding speech intent from non-frontal cortical areas

Objective. Brain machine interfaces (BMIs) that can restore speech have predominantly focused on decoding speech signals from the speech motor cortices. A few studies have shown some information outside the speech motor cortices, such as in parietal and temporal lobes, that also may be useful for BMIs. The ability to use information from outside the frontal lobe could be useful not only for people with locked-in syndrome, but also to people with frontal lobe damage, which can cause nonfluent aphasia or apraxia of speech. However, temporal and parietal lobes are predominantly involved in perceptive speech processing and comprehension. Therefore, to be able to use signals from these areas in a speech BMI, it is important to ascertain that they are related to production. Here, using intracranial recordings, we sought evidence for whether, when and where neural information related to speech intent could be found in the temporal and parietal corticesApproach. Using intracranial recordings, we examined neural activity across temporal and parietal cortices to identify signals associated with speech intent. We employed causal information to distinguish speech intent from resting states and other language-related processes, such as comprehension and working memory. Neural signals were analyzed for their spatial distribution and temporal dynamics to determine their relevance to speech production.Main results. Causal information enabled us to distinguish speech intent from resting state and other processes involved in language processing or working memory. Information related to speech intent was distributed widely across the temporal and parietal lobes, including superior temporal, medial temporal, angular, and supramarginal gyri.Significance. Loss of communication due to neurological diseases can be devastating. While speech BMIs have made strides in decoding speech from frontal lobe signals, our study reveals that the temporal and parietal cortices contain information about speech production intent that can be causally decoded prior to the onset of voice. This information is distributed across a large network. This information can be used to improve current speech BMIs and potentially expand the patient population for speech BMIs to include people with frontal lobe damage from stroke or traumatic brain injury.

Global motor dynamics - Invariant neural representations of motor behavior in distributed brain-wide recordings

Objective.Motor-related neural activity is more widespread than previously thought, as pervasive brain-wide neural correlates of motor behavior have been reported in various animal species. Brain-wide movement-related neural activity have been observed in individual brain areas in humans as well, but it is unknown to what extent global patterns exist.Approach.Here, we use a decoding approach to capture and characterize brain-wide neural correlates of movement. We recorded invasive electrophysiological data from stereotactic electroencephalographic electrodes implanted in eight epilepsy patients who performed both an executed and imagined grasping task. Combined, these electrodes cover the whole brain, including deeper structures such as the hippocampus, insula and basal ganglia. We extract a low-dimensional representation and classify movement from rest trials using a Riemannian decoder.Main results.We reveal global neural dynamics that are predictive across tasks and participants. Using an ablation analysis, we demonstrate that these dynamics remain remarkably stable under loss of information. Similarly, the dynamics remain stable across participants, as we were able to predict movement across participants using transfer learning.Significance.Our results show that decodable global motor-related neural dynamics exist within a low-dimensional space. The dynamics are predictive of movement, nearly brain-wide and present in all our participants. The results broaden the scope to brain-wide investigations, and may allow combining datasets of multiple participants with varying electrode locations or calibrationless neural decoder.

A direct spinal cord-computer interface enables the control of the paralysed hand in spinal cord injury

Paralysis of the muscles controlling the hand dramatically limits the quality of life for individuals living with spinal cord injury (SCI). Here, with a non-invasive neural interface, we demonstrate that eight motor complete SCI individuals (C5-C6) are still able to task-modulate in real-time the activity of populations of spinal motor neurons with residual neural pathways. In all SCI participants tested, we identified groups of motor units under voluntary control that encoded various hand movements. The motor unit discharges were mapped into more than 10 degrees of freedom, ranging from grasping to individual hand-digit flexion and extension. We then mapped the neural dynamics into a real-time controlled virtual hand. The SCI participants were able to match the cue hand posture by proportionally controlling four degrees of freedom (opening and closing the hand and index flexion/extension). These results demonstrate that wearable muscle sensors provide access to spared motor neurons that are fully under voluntary control in complete cervical SCI individuals. This non-invasive neural interface allows the investigation of motor neuron changes after the injury and has the potential to promote movement restoration when integrated with assistive devices.

A mosaic of whole-body representations in human motor cortex

Understanding how the body is represented in motor cortex is key to understanding how the brain controls movement. The precentral gyrus (PCG) has long been thought to contain largely distinct regions for the arm, leg and face (represented by the "motor homunculus"). However, mounting evidence has begun to reveal a more intermixed, interrelated and broadly tuned motor map. Here, we revisit the motor homunculus using microelectrode array recordings from 20 arrays that broadly sample PCG across 8 individuals, creating a comprehensive map of human motor cortex at single neuron resolution. We found whole-body representations throughout all sampled points of PCG, contradicting traditional leg/arm/face boundaries. We also found two speech-preferential areas with a broadly tuned, orofacial-dominant area in between them, previously unaccounted for by the homunculus. Throughout PCG, movement representations of the four limbs were interlinked, with homologous movements of different limbs (e.g., toe curl and hand close) having correlated representations. Our findings indicate that, while the classic homunculus aligns with each area's preferred body region at a coarse level, at a finer scale, PCG may be better described as a mosaic of functional zones, each with its own whole-body representation.

A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages

Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.

Transcranial focused ultrasound to V5 enhances human visual motion brain-computer interface by modulating feature-based attention

A brain-computer interface (BCI) enables users to control devices with their minds. Despite advancements, non-invasive BCIs still exhibit high error rates, prompting investigation into the potential reduction through concurrent targeted neuromodulation. Transcranial focused ultrasound (tFUS) is an emerging non-invasive neuromodulation technology with high spatiotemporal precision. This study examines whether tFUS neuromodulation can improve BCI outcomes, and explores the underlying mechanism of action using high-density electroencephalography (EEG) source imaging (ESI). As a result, V5-targeted tFUS significantly reduced the error in a BCI speller task. Source analyses revealed a significantly increase in theta and alpha activities in the tFUS condition at both V5 and downstream in the dorsal visual processing pathway. Correlation analysis indicated that the connection within the dorsal processing pathway was preserved during tFUS stimulation, while the ventral connection was weakened. These findings suggest that V5-targeted tFUS enhances feature-based attention to visual motion.

A Multimodal Assistive-Robotic-Arm Control System to Increase Independence After Tetraplegia

Following tetraplegia, independence for completing essential daily tasks, such as opening doors and eating, significantly declines. Assistive robotic manipulators (ARMs) could restore independence, but typically input devices for these manipulators require functional use of the hands. We created and validated a hands-free multimodal input system for controlling an ARM in virtual reality using combinations of a gyroscope, eye-tracking, and heterologous surface electromyography (sEMG). These input modalities are mapped to ARM functions based on the user's preferences and to maximize the utility of their residual volitional capabilities following tetraplegia. The two participants in this study with tetraplegia preferred to use the control mapping with sEMG button functions and disliked winking commands. Non-disabled participants were more varied in their preferences and performance, further suggesting that customizability is an advantageous component of the control system. Replacing buttons from a traditional handheld controller with sEMG did not substantively reduce performance. The system provided adequate control to all participants to complete functional tasks in virtual reality such as opening door handles, turning stove dials, eating, and drinking, all of which enable independence and improved quality of life for these individuals.

Representation of internal speech by single neurons in human supramarginal gyrus

Speech brain-machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.

Functional ultrasound imaging of human brain activity through an acoustically transparent cranial window

Visualization of human brain activity is crucial for understanding normal and aberrant brain function. Currently available neural activity recording methods are highly invasive, have low sensitivity, and cannot be conducted outside of an operating room. Functional ultrasound imaging (fUSI) is an emerging technique that offers sensitive, large-scale, high-resolution neural imaging; however, fUSI cannot be performed through the adult human skull. Here, we used a polymeric skull replacement material to create an acoustic window compatible with fUSI to monitor adult human brain activity in a single individual. Using an in vitro cerebrovascular phantom to mimic brain vasculature and an in vivo rodent cranial defect model, first, we evaluated the fUSI signal intensity and signal-to-noise ratio through polymethyl methacrylate (PMMA) cranial implants of different thicknesses or a titanium mesh implant. We found that rat brain neural activity could be recorded with high sensitivity through a PMMA implant using a dedicated fUSI pulse sequence. We then designed a custom ultrasound-transparent cranial window implant for an adult patient undergoing reconstructive skull surgery after traumatic brain injury. We showed that fUSI could record brain activity in an awake human outside of the operating room. In a video game "connect the dots" task, we demonstrated mapping and decoding of task-modulated cortical activity in this individual. In a guitar-strumming task, we mapped additional task-specific cortical responses. Our proof-of-principle study shows that fUSI can be used as a high-resolution (200 μm) functional imaging modality for measuring adult human brain activity through an acoustically transparent cranial window.

Text and image generation from intracranial electroencephalography using an embedding space for text and images

Objective.Invasive brain-computer interfaces (BCIs) are promising communication devices for severely paralyzed patients. Recent advances in intracranial electroencephalography (iEEG) coupled with natural language processing have enhanced communication speed and accuracy. It should be noted that such a speech BCI uses signals from the motor cortex. However, BCIs based on motor cortical activities may experience signal deterioration in users with motor cortical degenerative diseases such as amyotrophic lateral sclerosis. An alternative approach to using iEEG of the motor cortex is necessary to support patients with such conditions.Approach. In this study, a multimodal embedding of text and images was used to decode visual semantic information from iEEG signals of the visual cortex to generate text and images. We used contrastive language-image pretraining (CLIP) embedding to represent images presented to 17 patients implanted with electrodes in the occipital and temporal cortices. A CLIP image vector was inferred from the high-γpower of the iEEG signals recorded while viewing the images.Main results.Text was generated by CLIPCAP from the inferred CLIP vector with better-than-chance accuracy. Then, an image was created from the generated text using StableDiffusion with significant accuracy.Significance.The text and images generated from iEEG through the CLIP embedding vector can be used for improved communication.

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models

Objective.Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field.Approach.In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task.Main results.We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%-100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech.Significance.These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.

Transcranial Focused Ultrasound to V5 Enhances Human Visual Motion Brain-Computer Interface by Modulating Feature-Based Attention

Paralysis affects roughly 1 in 50 Americans. While there is no cure for the condition, brain-computer interfaces (BCI) can allow users to control a device with their mind, bypassing the paralyzed region. Non-invasive BCIs still have high error rates, which is hypothesized to be reduced with concurrent targeted neuromodulation. This study examines whether transcranial focused ultrasound (tFUS) modulation can improve BCI outcomes, and what the underlying mechanism of action might be through high-density electroencephalography (EEG)-based source imaging (ESI) analyses. V5-targeted tFUS significantly reduced the error for the BCI speller task. ESI analyses showed significantly increased theta activity in the tFUS condition at both V5 and downstream the dorsal visual processing pathway. Correlation analysis indicates that the dorsal processing pathway connection was preserved during tFUS stimulation, whereas extraneous connections were severed. These results suggest that V5-targeted tFUS' mechanism of action is to raise the brain's feature-based attention to visual motion.

A window to the brain: ultrasound imaging of human neural activity through a permanent acoustic window

Recording human brain activity is crucial for understanding normal and aberrant brain function. However, available recording methods are either highly invasive or have relatively low sensitivity. Functional ultrasound imaging (fUSI) is an emerging technique that offers sensitive, large-scale, high-resolution neural imaging. However, fUSI cannot be performed through adult human skull. Here, we use a polymeric skull replacement material to create an acoustic window allowing ultrasound to monitor brain activity in fully intact adult humans. We design the window through experiments in phantoms and rodents, then implement it in a participant undergoing reconstructive skull surgery. Subsequently, we demonstrate fully non-invasive mapping and decoding of cortical responses to finger movement, marking the first instance of high-resolution (200 μm) and large-scale (50 mmx38 mm) brain imaging through a permanent acoustic window.

Dataset of Speech Production in intracranial.Electroencephalography

Speech production is an intricate process involving a large number of muscles and cognitive processes. The neural processes underlying speech production are not completely understood. As speech is a uniquely human ability, it can not be investigated in animal models. High-fidelity human data can only be obtained in clinical settings and is therefore not easily available to all researchers. Here, we provide a dataset of 10 participants reading out individual words while we measured intracranial EEG from a total of 1103 electrodes. The data, with its high temporal resolution and coverage of a large variety of cortical and sub-cortical brain regions, can help in understanding the speech production process better. Simultaneously, the data can be used to test speech decoding and synthesis approaches from neural data to develop speech Brain-Computer Interfaces and speech neuroprostheses.

Measurement(s)	Brain activity
Technology Type(s)	Stereotactic electroencephalography
Sample Characteristic - Organism	Homo sapiens
Sample Characteristic - Environment	Epilepsy monitoring center
Sample Characteristic - Location	The Netherlands

Speech decoding from a small set of spatially segregated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network

Objective. Speech decoding, one of the most intriguing brain-computer interface applications, opens up plentiful opportunities from rehabilitation of patients to direct and seamless communication between human species. Typical solutions rely on invasive recordings with a large number of distributed electrodes implanted through craniotomy. Here we explored the possibility of creating speech prosthesis in a minimally invasive setting with a small number of spatially segregated intracranial electrodes.Approach. We collected one hour of data (from two sessions) in two patients implanted with invasive electrodes. We then used only the contacts that pertained to a single stereotactic electroencephalographic (sEEG) shaft or an electrocorticographic (ECoG) stripe to decode neural activity into 26 words and one silence class. We employed a compact convolutional network-based architecture whose spatial and temporal filter weights allow for a physiologically plausible interpretation.Mainresults. We achieved on average 55% accuracy using only six channels of data recorded with a single minimally invasive sEEG electrode in the first patient and 70% accuracy using only eight channels of data recorded for a single ECoG strip in the second patient in classifying 26+1 overtly pronounced words. Our compact architecture did not require the use of pre-engineered features, learned fast and resulted in a stable, interpretable and physiologically meaningful decision rule successfully operating over a contiguous dataset collected during a different time interval than that used for training. Spatial characteristics of the pivotal neuronal populations corroborate with active and passive speech mapping results and exhibit the inverse space-frequency relationship characteristic of neural activity. Compared to other architectures our compact solution performed on par or better than those recently featured in neural speech decoding literature.Significance. We showcase the possibility of building a speech prosthesis with a small number of electrodes and based on a compact feature engineering free decoder derived from a small amount of training data.

Getting a grasp on BMIs: Decoding prehension and speech signals

Wandelt et al. (2022) show that different grasps can be decoded from neural activity in the human supramarginal gyrus (SMG), ventral premotor cortex, and somatosensory cortex during motor imagery and speech, highlighting the attractiveness of higher-level areas such as the SMG for brain-machine interface applications.