Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication

Representation of internal speech by single neurons in human supramarginal gyrus

Speech brain-machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.

An accurate and rapidly calibrating speech neuroprosthesis

Brain-computer interfaces can enable rapid, intuitive communication for people with paralysis by transforming the cortical activity associated with attempted speech into text on a computer screen. Despite recent advances, communication with brain-computer interfaces has been restricted by extensive training data requirements and inaccurate word output. A man in his 40's with ALS with tetraparesis and severe dysarthria (ALSFRS-R = 23) was enrolled into the BrainGate2 clinical trial. He underwent surgical implantation of four microelectrode arrays into his left precentral gyrus, which recorded neural activity from 256 intracortical electrodes. We report a speech neuroprosthesis that decoded his neural activity as he attempted to speak in both prompted and unstructured conversational settings. Decoded words were displayed on a screen, then vocalized using text-to-speech software designed to sound like his pre-ALS voice. On the first day of system use, following 30 minutes of attempted speech training data, the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. On the second day, the size of the possible output vocabulary increased to 125,000 words, and, after 1.4 additional hours of training data, the neuroprosthesis achieved 90.2% accuracy. With further training data, the neuroprosthesis sustained 97.5% accuracy beyond eight months after surgical implantation. The participant has used the neuroprosthesis to communicate in self-paced conversations for over 248 hours. In an individual with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore naturalistic communication after a brief training period.

Global research trends and hotspots of artificial intelligence research in spinal cord neural injury and restoration-a bibliometrics and visualization analysis

Background

Artificial intelligence (AI) technology has made breakthroughs in spinal cord neural injury and restoration in recent years. It has a positive impact on clinical treatment. This study explores AI research's progress and hotspots in spinal cord neural injury and restoration. It also analyzes research shortcomings related to this area and proposes potential solutions.

Methods

We used CiteSpace 6.1.R6 and VOSviewer 1.6.19 to research WOS articles on AI research in spinal cord neural injury and restoration.

Results

A total of 1,502 articles were screened, in which the United States dominated; Kadone, Hideki (13 articles, University of Tsukuba, JAPAN) was the author with the highest number of publications; ARCH PHYS MED REHAB (IF = 4.3) was the most cited journal, and topics included molecular biology, immunology, neurology, sports, among other related areas.

Conclusion

We pinpointed three research hotspots for AI research in spinal cord neural injury and restoration: (1) intelligent robots and limb exoskeletons to assist rehabilitation training; (2) brain-computer interfaces; and (3) neuromodulation and noninvasive electrical stimulation. In addition, many new hotspots were discussed: (1) starting with image segmentation models based on convolutional neural networks; (2) the use of AI to fabricate polymeric biomaterials to provide the microenvironment required for neural stem cell-derived neural network tissues; (3) AI survival prediction tools, and transcription factor regulatory networks in the field of genetics were discussed. Although AI research in spinal cord neural injury and restoration has many benefits, the technology has several limitations (data and ethical issues). The data-gathering problem should be addressed in future research, which requires a significant sample of quality clinical data to build valid AI models. At the same time, research on genomics and other mechanisms in this field is fragile. In the future, machine learning techniques, such as AI survival prediction tools and transcription factor regulatory networks, can be utilized for studies related to the up-regulation of regeneration-related genes and the production of structural proteins for axonal growth.

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals

Objective

This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training.

Approach

We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes, by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train both subject-specific models using data from a single participant as well as multi-patient models exploiting data from multiple participants.

Main Results

The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation.

Significance

The proposed SwinTW decoder enables future speech neuroprostheses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests the exciting possibility of developing speech neuroprostheses for people with speech disability without relying on their own neural data for training, which is not always feasible.

[Ethical considerations for medical applications of implantable brain-computer interfaces]

Implantable brain-computer interfaces (BCIs) have potentially important clinical applications due to the high spatial resolution and signal-to-noise ratio of electrodes that are closer to or implanted in the cerebral cortex. However, the surgery and electrodes of implantable BCIs carry safety risks of brain tissue damage, and their medical applications face ethical challenges, with little literature to date systematically considering ethical norms for the medical applications of implantable BCIs. In order to promote the clinical translation of this type of BCI, we considered the ethics of practice for the medical application of implantable BCIs, including: reducing the risk of brain tissue damage from implantable BCI surgery and electrodes, providing patients with customized and personalized implantable BCI treatments, ensuring multidisciplinary collaboration in the clinical application of implantable BCIs, and the responsible use of implantable BCIs, among others. It is expected that this article will provide thoughts and references for the research and development of ethics of the medical application of implantable BCI.

Update on How to Approach a Patient with Locked-In Syndrome and Their Communication Ability

Locked-in syndrome (LIS) is a rare and challenging condition that results in tetraplegia and cranial nerve paralysis while maintaining consciousness and variable cognitive function. Once acute management is completed, it is important to work with the patient on developing a plan to maintain and improve their quality of life (QOL). A key component towards increasing or maintaining QOL within this population involves the establishment of a functional communication system. Evaluating cognition in patients with LIS is vital for evaluating patients' communication needs along with physical rehabilitation to maximize their QOL. In the past decade or so, there has been an increase in research surrounding brain-computer interfaces to improve communication abilities for paralyzed patients. This article provides an update on the available technology and the protocol for finding the best way for patients with this condition to communicate. This article aims to increase knowledge of how to enhance and manage communication among LIS patients.

Brain-computer interface paradigms and neural coding

Brain signal patterns generated in the central nervous system of brain-computer interface (BCI) users are closely related to BCI paradigms and neural coding. In BCI systems, BCI paradigms and neural coding are critical elements for BCI research. However, so far there have been few references that clearly and systematically elaborated on the definition and design principles of the BCI paradigm as well as the definition and modeling principles of BCI neural coding. Therefore, these contents are expounded and the existing main BCI paradigms and neural coding are introduced in the review. Finally, the challenges and future research directions of BCI paradigm and neural coding were discussed, including user-centered design and evaluation for BCI paradigms and neural coding, revolutionizing the traditional BCI paradigms, breaking through the existing techniques for collecting brain signals and combining BCI technology with advanced AI technology to improve brain signal decoding performance. It is expected that the review will inspire innovative research and development of the BCI paradigm and neural coding.

A Neural Speech Decoding Framework Leveraging Deep Learning and Speech Synthesis

Decoding human speech from neural signals is essential for brain-computer interface (BCI) technologies restoring speech function in populations with neurological deficits. However, it remains a highly challenging task, compounded by the scarce availability of neural signals with corresponding speech, data complexity, and high dimensionality, and the limited publicly available source code. Here, we present a novel deep learning-based neural speech decoding framework that includes an ECoG Decoder that translates electrocorticographic (ECoG) signals from the cortex into interpretable speech parameters and a novel differentiable Speech Synthesizer that maps speech parameters to spectrograms. We develop a companion audio-to-audio auto-encoder consisting of a Speech Encoder and the same Speech Synthesizer to generate reference speech parameters to facilitate the ECoG Decoder training. This framework generates natural-sounding speech and is highly reproducible across a cohort of 48 participants. Among three neural network architectures for the ECoG Decoder, the 3D ResNet model has the best decoding performance (PCC=0.804) in predicting the original speech spectrogram, closely followed by the SWIN model (PCC=0.796). Our experimental results show that our models can decode speech with high correlation even when limited to only causal operations, which is necessary for adoption by real-time neural prostheses. We successfully decode speech in participants with either left or right hemisphere coverage, which could lead to speech prostheses in patients with speech deficits resulting from left hemisphere damage. Further, we use an occlusion analysis to identify cortical regions contributing to speech decoding across our models. Finally, we provide open-source code for our two-stage training pipeline along with associated preprocessing and visualization tools to enable reproducible research and drive research across the speech science and prostheses communities.

Continuous synthesis of artificial speech sounds from human cortical surface recordings during silent speech production

Objective. Brain-computer interfaces can restore various forms of communication in paralyzed patients who have lost their ability to articulate intelligible speech. This study aimed to demonstrate the feasibility of closed-loop synthesis of artificial speech sounds from human cortical surface recordings during silent speech production.Approach. Ten participants with intractable epilepsy were temporarily implanted with intracranial electrode arrays over cortical surfaces. A decoding model that predicted audible outputs directly from patient-specific neural feature inputs was trained during overt word reading and immediately tested with overt, mimed and imagined word reading. Predicted outputs were later assessed objectively against corresponding voice recordings and subjectively through human perceptual judgments.Main results. Artificial speech sounds were successfully synthesized during overt and mimed utterances by two participants with some coverage of the precentral gyrus. About a third of these sounds were correctly identified by naïve listeners in two-alternative forced-choice tasks. A similar outcome could not be achieved during imagined utterances by any of the participants. However, neural feature contribution analyses suggested the presence of exploitable activation patterns during imagined speech in the postcentral gyrus and the superior temporal gyrus. In future work, a more comprehensive coverage of cortical surfaces, including posterior parts of the middle frontal gyrus and the inferior frontal gyrus, could improve synthesis performance during imagined speech.Significance.As the field of speech neuroprostheses is rapidly moving toward clinical trials, this study addressed important considerations about task instructions and brain coverage when conducting research on silent speech with non-target participants.

Impacts of Cortical Regions on EEG-based Classification of Lexical Tones and Vowels in Spoken Speech

Speech impairment is one of the most serious problems for patients with communication disorders, e.g., stroke survivors. The brain-computer interface (BCI) systems have shown the potential to alternatively control or rehabilitate the neurological damages in speech production. The effects of different cortical regions in speech-based BCI systems are essential to be studied, which are favorable for improving the performance of speech-based BCI systems. This work aimed to explore the impacts of different speech-related cortical regions in the electroencephalogram (EEG) based classification of seventy spoken Mandarin monosyllables carrying four vowels and four lexical tones. Seven audible speech production-related cortical regions were studied, involving Broca's and Wernicke's areas, auditory cortex, motor cortex, prefrontal cortex, sensory cortex, left brain, right brain, and whole brain. Following the previous studies in which EEG signals were collected from ten subjects during Mandarin speech production, the features of EEG signals were extracted by the Riemannian manifold method, and a linear discriminant analysis (LDA) was regarded as a classifier to classify different vowels and lexical tones. The results showed that when using electrodes from whole brain, the classifier reached the best performances, which were 48.5% for lexical tones and 70.0% for vowels, respectively. The vowel classification results under Broca's and Wernicke's areas, auditory cortex, or prefrontal cortex were higher than those under the motor cortex or sensory cortex. No such differences were observed in the lexical tone classification task.

Machine-Learning Methods for Speech and Handwriting Detection Using Neural Signals: A Review

Brain-Computer Interfaces (BCIs) have become increasingly popular in recent years due to their potential applications in diverse fields, ranging from the medical sector (people with motor and/or communication disabilities), cognitive training, gaming, and Augmented Reality/Virtual Reality (AR/VR), among other areas. BCI which can decode and recognize neural signals involved in speech and handwriting has the potential to greatly assist individuals with severe motor impairments in their communication and interaction needs. Innovative and cutting-edge advancements in this field have the potential to develop a highly accessible and interactive communication platform for these people. The purpose of this review paper is to analyze the existing research on handwriting and speech recognition from neural signals. So that the new researchers who are interested in this field can gain thorough knowledge in this research area. The current research on neural signal-based recognition of handwriting and speech has been categorized into two main types: invasive and non-invasive studies. We have examined the latest papers on converting speech-activity-based neural signals and handwriting-activity-based neural signals into text data. The methods of extracting data from the brain have also been discussed in this review. Additionally, this review includes a brief summary of the datasets, preprocessing techniques, and methods used in these studies, which were published between 2014 and 2022. This review aims to provide a comprehensive summary of the methodologies used in the current literature on neural signal-based recognition of handwriting and speech. In essence, this article is intended to serve as a valuable resource for future researchers who wish to investigate neural signal-based machine-learning methods in their work.

Therapeutic Effectiveness of Brain Computer Interfaces in Stroke Patients: A Systematic Review

Background

Brain-computer interfaces (BCIs) are a rapidly advancing field which utilizes brain activity to control external devices for a myriad of functions, including the restoration of motor function. Clinically, BCIs have been especially impactful in patients who suffer from stroke-mediated damage. However, due to the rapid advancement in the field, there is a lack of accepted standards of practice. Therefore, the aim of this systematic review is to summarize the current literature published regarding the efficacy of BCI-based rehabilitation of motor dysfunction in stroke patients.

Methodology

This systematic review was performed in accordance with the guidelines set forth by the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) 2020 statement. PubMed, Embase, and Cochrane Library were queried for relevant articles and screened for inclusion criteria by two authors. All discrepancies were resolved by discussion among both reviewers and subsequent consensus.

Results

11/12 (91.6%) of studies focused on upper extremity outcomes and reported larger initial improvements for participants in the treatment arm (using BCI) as compared to those in the control arm (no BCI). 2/2 studies focused on lower extremity outcomes reported improvements for the treatment arm compared to the control arm.

Discussion/Conclusion

This systematic review illustrates the utility BCI has for the restoration of upper extremity and lower extremity motor function in stroke patients and supports further investigation of BCI for other clinical indications.

Cutting-edge communication and learning assistive technologies for disabled children: An artificial intelligence perspective

In this study we provide an in-depth review and analysis of the impact of artificial intelligence (AI) components and solutions that support the development of cutting-edge assistive technologies for children with special needs. Various disabilities are addressed and the most recent assistive technologies that enhance communication and education of disabled children, as well as the AI technologies that have enabled their development, are presented. The paper summarizes with an AI perspective on future assistive technologies and ethical concerns arising from the use of such cutting-edge communication and learning technologies for children with disabilities.

Clinical neuroscience and neurotechnology: An amazing symbiosis

In the last decades, clinical neuroscience found a novel ally in neurotechnologies, devices able to record and stimulate electrical activity in the nervous system. These technologies improved the ability to diagnose and treat neural disorders. Neurotechnologies are concurrently enabling a deeper understanding of healthy and pathological dynamics of the nervous system through stimulation and recordings during brain implants. On the other hand, clinical neurosciences are not only driving neuroengineering toward the most relevant clinical issues, but are also shaping the neurotechnologies thanks to clinical advancements. For instance, understanding the etiology of a disease informs the location of a therapeutic stimulation, but also the way stimulation patterns should be designed to be more effective/naturalistic. Here, we describe cases of fruitful integration such as Deep Brain Stimulation and cortical interfaces to highlight how this symbiosis between clinical neuroscience and neurotechnology is closer to a novel integrated framework than to a simple interdisciplinary interaction.