Spatiotemporal dynamics of electrocorticographic high gamma activity during overt and covert word repetition

Revealing the spatiotemporal brain dynamics of covert speech compared with overt speech: A simultaneous EEG-fMRI study

Covert speech (CS) refers to speaking internally to oneself without producing any sound or movement. CS is involved in multiple cognitive functions and disorders. Reconstructing CS content by brain-computer interface (BCI) is also an emerging technique. However, it is still controversial whether CS is a truncated neural process of overt speech (OS) or involves independent patterns. Here, we performed a word-speaking experiment with simultaneous EEG-fMRI. It involved 32 participants, who generated words both overtly and covertly. By integrating spatial constraints from fMRI into EEG source localization, we precisely estimated the spatiotemporal dynamics of neural activity. During CS, EEG source activity was localized in three regions: the left precentral gyrus, the left supplementary motor area, and the left putamen. Although OS involved more brain regions with stronger activations, CS was characterized by an earlier event-locked activation in the left putamen (peak at 262 ms versus 1170 ms). The left putamen was also identified as the only hub node within the functional connectivity (FC) networks of both OS and CS, while showing weaker FC strength towards speech-related regions in the dominant hemisphere during CS. Path analysis revealed significant multivariate associations, indicating an indirect association between the earlier activation in the left putamen and CS, which was mediated by reduced FC towards speech-related regions. These findings revealed the specific spatiotemporal dynamics of CS, offering insights into CS mechanisms that are potentially relevant for future treatment of self-regulation deficits, speech disorders, and development of BCI speech applications.

Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech

Several attempts for speech brain-computer interfacing (BCI) have been made to decode phonemes, sub-words, words, or sentences using invasive measurements, such as the electrocorticogram (ECoG), during auditory speech perception, overt speech, or imagined (covert) speech. Decoding sentences from covert speech is a challenging task. Sixteen epilepsy patients with intracranially implanted electrodes participated in this study, and ECoGs were recorded during overt speech and covert speech of eight Japanese sentences, each consisting of three tokens. In particular, Transformer neural network model was applied to decode text sentences from covert speech, which was trained using ECoGs obtained during overt speech. We first examined the proposed Transformer model using the same task for training and testing, and then evaluated the model's performance when trained with overt task for decoding covert speech. The Transformer model trained on covert speech achieved an average token error rate (TER) of 46.6% for decoding covert speech, whereas the model trained on overt speech achieved a TER of 46.3% ( p > 0.05 ; d = 0.07 ) . Therefore, the challenge of collecting training data for covert speech can be addressed using overt speech. The performance of covert speech can improve by employing several overt speeches.

Representation of internal speech by single neurons in human supramarginal gyrus

Speech brain-machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.

Integrated sentence-level speech perception evokes strengthened language networks and facilitates early speech development

Natural poetic speeches (i.e., proverbs, nursery rhymes, and commercial ads) with strong prosodic regularities are easily memorized by children and the harmonious acoustic patterns are suggested to facilitate their integrated sentence processing. Do children have specific neural pathways for perceiving such poetic utterances, and does their speech development benefit from it? We recorded the task-induced hemodynamic changes of 94 children aged 2 to 12 years using functional near-infrared spectroscopy (fNIRS) while they listened to poetic and non-poetic natural sentences. Seventy-three adult as controls were recruited to investigate the developmental specificity of children group. The results indicated that poetic sentences perceiving is a highly integrated process featured by a lower brain workload in both groups. However, an early activated large-scale network was induced only in the child group, coordinated by hubs for connectivity diversity. Additionally, poetic speeches evoked activation in the phonological encoding regions in the children's group rather than adult controls which decreases with children's ages. The neural responses to poetic speeches were positively linked to children's speech communication performance, especially the fluency and semantic aspects. These results reveal children's neural sensitivity to integrated speech perception which facilitate early speech development by strengthening more sophisticated language networks and the perception-production circuit.

Distributed feedforward and feedback cortical processing supports human speech production

Speech production is a complex human function requiring continuous feedforward commands together with reafferent feedback processing. These processes are carried out by distinct frontal and temporal cortical networks, but the degree and timing of their recruitment and dynamics remain poorly understood. We present a deep learning architecture that translates neural signals recorded directly from the cortex to an interpretable representational space that can reconstruct speech. We leverage learned decoding networks to disentangle feedforward vs. feedback processing. Unlike prevailing models, we find a mixed cortical architecture in which frontal and temporal networks each process both feedforward and feedback information in tandem. We elucidate the timing of feedforward and feedback-related processing by quantifying the derived receptive fields. Our approach provides evidence for a surprisingly mixed cortical architecture of speech circuitry together with decoding advances that have important implications for neural prosthetics.

Spatio-temporal evolution of human neural activity during visually cued hand movements

Making hand movements in response to visual cues is common in daily life. It has been well known that this process activates multiple areas in the brain, but how these neural activations progress across space and time remains largely unknown. Taking advantage of intracranial electroencephalographic (iEEG) recordings using depth and subdural electrodes from 36 human subjects using the same task, we applied single-trial and cross-trial analyses to high-frequency iEEG activity. The results show that the neural activation was widely distributed across the human brain both within and on the surface of the brain, and focused specifically on certain areas in the parietal, frontal, and occipital lobes, where parietal lobes present significant left lateralization on the activation. We also demonstrate temporal differences across these brain regions. Finally, we evaluated the degree to which the timing of activity within these regions was related to sensory or motor function. The findings of this study promote the understanding of task-related neural processing of the human brain, and may provide important insights for translational applications.

Characterization of High-Gamma Activity in Electrocorticographic Signals

Introduction

Electrocorticographic (ECoG) high-gamma activity (HGA) is a widely recognized and robust neural correlate of cognition and behavior. However, fundamental signal properties of HGA, such as the high-gamma frequency band or temporal dynamics of HGA, have never been systematically characterized. As a result, HGA estimators are often poorly adjusted, such that they miss valuable physiological information.

Methods

To address these issues, we conducted a thorough qualitative and quantitative characterization of HGA in ECoG signals. Our study is based on ECoG signals recorded from 18 epilepsy patients while performing motor control, listening, and visual perception tasks. In this study, we first categorize HGA into HGA types based on the cognitive/behavioral task. For each HGA type, we then systematically quantify three fundamental signal properties of HGA: the high-gamma frequency band, the HGA bandwidth, and the temporal dynamics of HGA.

Results

The high-gamma frequency band strongly varies across subjects and across cognitive/behavioral tasks. In addition, HGA time courses have lowpass character, with transients limited to 10 Hz. The task-related rise time and duration of these HGA time courses depend on the individual subject and cognitive/behavioral task. Task-related HGA amplitudes are comparable across the investigated tasks.

Discussion

This study is of high practical relevance because it provides a systematic basis for optimizing experiment design, ECoG acquisition and processing, and HGA estimation. Our results reveal previously unknown characteristics of HGA, the physiological principles of which need to be investigated in further studies.

The nested hierarchy of overt, mouthed, and imagined speech activity evident in intracranial recordings

Recent studies have demonstrated that it is possible to decode and synthesize various aspects of acoustic speech directly from intracranial measurements of electrophysiological brain activity. In order to continue progressing toward the development of a practical speech neuroprosthesis for the individuals with speech impairments, better understanding and modeling of imagined speech processes are required. The present study uses intracranial brain recordings from participants that performed a speaking task with trials consisting of overt, mouthed, and imagined speech modes, representing various degrees of decreasing behavioral output. Speech activity detection models are constructed using spatial, spectral, and temporal brain activity features, and the features and model performances are characterized and compared across the three degrees of behavioral output. The results indicate the existence of a hierarchy in which the relevant channels for the lower behavioral output modes form nested subsets of the relevant channels from the higher behavioral output modes. This provides important insights for the elusive goal of developing more effective imagined speech decoding models with respect to the better-established overt speech decoding counterparts.

Opportunities, pitfalls and trade-offs in designing protocols for measuring the neural correlates of speech

Decoding speech and speech-related processes directly from the human brain has intensified in studies over recent years as such a decoder has the potential to positively impact people with limited communication capacity due to disease or injury. Additionally, it can present entirely new forms of human-computer interaction and human-machine communication in general and facilitate better neuroscientific understanding of speech processes. Here, we synthesize the literature on neural speech decoding pertaining to how speech decoding experiments have been conducted, coalescing around a necessity for thoughtful experimental design aimed at specific research goals, and robust procedures for evaluating speech decoding paradigms. We examine the use of different modalities for presenting stimuli to participants, methods for construction of paradigms including timings and speech rhythms, and possible linguistic considerations. In addition, novel methods for eliciting naturalistic speech and validating imagined speech task performance in experimental settings are presented based on recent research. We also describe the multitude of terms used to instruct participants on how to produce imagined speech during experiments and propose methods for investigating the effect of these terms on imagined speech decoding. We demonstrate that the range of experimental procedures used in neural speech decoding studies can have unintended consequences which can impact upon the efficacy of the knowledge obtained. The review delineates the strengths and weaknesses of present approaches and poses methodological advances which we anticipate will enhance experimental design, and progress toward the optimal design of movement independent direct speech brain-computer interfaces.

A voice without a mouth no more: The neurobiology of language and consciousness

Most research on the neurobiology of language ignores consciousness and vice versa. Here, language, with an emphasis on inner speech, is hypothesised to generate and sustain self-awareness, i.e., higher-order consciousness. Converging evidence supporting this hypothesis is reviewed. To account for these findings, a 'HOLISTIC' model of neurobiology of language, inner speech, and consciousness is proposed. It involves a 'core' set of inner speech production regions that initiate the experience of feeling and hearing words. These take on affective qualities, deriving from activation of associated sensory, motor, and emotional representations, involving a largely unconscious dynamic 'periphery', distributed throughout the whole brain. Responding to those words forms the basis for sustained network activity, involving 'default mode' activation and prefrontal and thalamic/brainstem selection of contextually relevant responses. Evidence for the model is reviewed, supporting neuroimaging meta-analyses conducted, and comparisons with other theories of consciousness made. The HOLISTIC model constitutes a more parsimonious and complete account of the 'neural correlates of consciousness' that has implications for a mechanistic account of mental health and wellbeing.

Virtual implantation using conventional scalp EEG delineates seizure onset and predicts surgical outcome in children with epilepsy

OBJECTIVE

Delineation of the seizure onset zone (SOZ) is required in children with drug resistant epilepsy (DRE) undergoing neurosurgery. Intracranial EEG (icEEG) serves as gold standard but has limitations. Here, we examine the utility of virtual implantation with electrical source imaging (ESI) on ictal scalp EEG for mapping the SOZ and predict surgical outcome.

METHODS

We retrospectively analyzed EEG data from 35 children with DRE who underwent surgery and dichotomized into seizure-free (SF) and non-seizure-free (NSF). We estimated virtual sensors (VSs) at brain locations that matched icEEG implantation and compared ictal patterns at VSs vs icEEG. We calculated the agreement between VSs SOZ and clinically defined SOZ and built receiver operating characteristic (ROC) curves to test whether it predicted outcome.

RESULTS

Twenty-one patients were SF after surgery. Moderate agreement between virtual and icEEG patterns was observed (kappa = 0.45, p < 0.001). Virtual SOZ agreement with clinically defined SOZ was higher in SF vs NSF patients (66.6% vs 41.6%, p = 0.01). Anatomical concordance of virtual SOZ with clinically defined SOZ predicted outcome (AUC = 0.73; 95% CI: 0.57-0.89; sensitivity = 66.7%; specificity = 78.6%; accuracy = 71.4%).

CONCLUSIONS

Virtual implantation on ictal scalp EEG can approximate the SOZ and predict outcome.

SIGNIFICANCE

SOZ mapping with VSs may contribute to tailoring icEEG implantation and predict outcome.

Sleep talking versus sleep moaning: electrophysiological patterns preceding linguistic vocalizations during sleep

STUDY OBJECTIVES

Sleep talking (ST) has been rarely studied as an isolated phenomenon. Late investigations over the psycholinguistic features of vocal production in ST pointed to coherence with wake language formal features. Therefore, we investigated the EEG correlates of Verbal ST as the overt manifestation of sleep-related language processing, with the hypothesis of shared electrophysiological correlates with wake language production.

METHODS

From a sample of 155 Highly frequent STs, we recorded 13 participants (age range 19-30 years, mean age 24.6 ± 3.3; 7F) via vPSG for at least two consecutive nights, and a total of 28 nights. We first investigated the sleep macrostructure of STs compared to 13 age and gender-matched subjects. We then compared the EEG signal before 21 Verbal STs versus 21 Nonverbal STs (moaning, laughing, crying, etc.) in six STs reporting both vocalization types in Stage 2 NREM sleep.

RESULTS

The 2 × 2 mixed analysis of variance Group × Night interaction showed no statistically significant effect for macrostructural variables, but significant main effects for Group with lower REM (%), total sleep time, total bedtime, sleep efficiency index, and greater NREM (%) for STs compared to controls. EEG statistical comparisons (paired-samples Student's t-test) showed a decrement in power spectra for Verbal STs versus Nonverbal STs within the theta and alpha EEG bands, strongly lateralized to the left hemisphere and localized on centro-parietal-occipitals channels. A single left parietal channel (P7) held significance after Bonferroni correction.

CONCLUSIONS

Our results suggest shared neural mechanisms between Verbal ST and language processing during wakefulness and a possible functional overlapping with linguistic planning in wakefulness.

A State-of-the-Art Review of EEG-Based Imagined Speech Decoding

Currently, the most used method to measure brain activity under a non-invasive procedure is the electroencephalogram (EEG). This is because of its high temporal resolution, ease of use, and safety. These signals can be used under a Brain Computer Interface (BCI) framework, which can be implemented to provide a new communication channel to people that are unable to speak due to motor disabilities or other neurological diseases. Nevertheless, EEG-based BCI systems have presented challenges to be implemented in real life situations for imagined speech recognition due to the difficulty to interpret EEG signals because of their low signal-to-noise ratio (SNR). As consequence, in order to help the researcher make a wise decision when approaching this problem, we offer a review article that sums the main findings of the most relevant studies on this subject since 2009. This review focuses mainly on the pre-processing, feature extraction, and classification techniques used by several authors, as well as the target vocabulary. Furthermore, we propose ideas that may be useful for future work in order to achieve a practical application of EEG-based BCI systems toward imagined speech decoding.

Left hemisphere dominance for bilateral kinematic encoding in the human brain

Neurophysiological studies in humans and nonhuman primates have revealed movement representations in both the contralateral and ipsilateral hemispheres. Inspired by clinical observations, we ask if this bilateral representation differs for the left and right hemispheres. Electrocorticography was recorded in human participants during an instructed-delay reaching task, with movements produced with either the contralateral or ipsilateral arm. Using a cross-validated kinematic encoding model, we found stronger bilateral encoding in the left hemisphere, an effect that was present during preparation and was amplified during execution. Consistent with this asymmetry, we also observed better across-arm generalization in the left hemisphere, indicating similar neural representations for right and left arm movements. Notably, these left hemisphere electrodes were centered over premotor and parietal regions. The more extensive bilateral encoding in the left hemisphere adds a new perspective to the pervasive neuropsychological finding that the left hemisphere plays a dominant role in praxis.

Imagined speech can be decoded from low- and cross-frequency intracranial EEG features

Reconstructing intended speech from neural activity using brain-computer interfaces holds great promises for people with severe speech production deficits. While decoding overt speech has progressed, decoding imagined speech has met limited success, mainly because the associated neural signals are weak and variable compared to overt speech, hence difficult to decode by learning algorithms. We obtained three electrocorticography datasets from 13 patients, with electrodes implanted for epilepsy evaluation, who performed overt and imagined speech production tasks. Based on recent theories of speech neural processing, we extracted consistent and specific neural features usable for future brain computer interfaces, and assessed their performance to discriminate speech items in articulatory, phonetic, and vocalic representation spaces. While high-frequency activity provided the best signal for overt speech, both low- and higher-frequency power and local cross-frequency contributed to imagined speech decoding, in particular in phonetic and vocalic, i.e. perceptual, spaces. These findings show that low-frequency power and cross-frequency dynamics contain key information for imagined speech decoding.

A comparison and classification of oscillatory characteristics in speech perception and covert speech

Covert speech, the mental imagery of speaking, has been studied increasingly to understand and decode thoughts in the context of brain-computer interfaces. In studies of speech comprehension, neural oscillations are thought to play a key role in the temporal encoding of speech. However, little is known about the role of oscillations in covert speech. In this study, we investigated the oscillatory involvements in covert speech and speech perception. Data were collected from 10 participants with 64 channel EEG. Participants heard the words, 'blue' and 'orange', and subsequently mentally rehearsed them. First, continuous wavelet transform was performed on epoched signals and subsequently two-tailed t-tests between two classes were conducted to determine statistical differences in frequency and time (t-CWT). Features were also extracted using t-CWT and subsequently classified using a support vector machine. θ and γ phase amplitude coupling (PAC) was also assessed within and between tasks. All binary classifications produced accuracies significantly greater (80-90%) than chance level, supporting the use of t-CWT in determining relative oscillatory involvements. While the perception task dynamically invoked all frequencies with more prominent θ and α activity, the covert task favoured higher frequencies with significantly higher γ activity than perception. Moreover, the perception condition produced significant θ-γ PAC, corroborating a reported linkage between syllabic and phonemic sampling. Although this coupling was found to be suppressed in the covert condition, we found significant cross-task coupling between perception θ and covert speech γ. Covert speech processing appears to be largely associated with higher frequencies of EEG. Importantly, the significant cross-task coupling between speech perception and covert speech, in the absence of within-task covert speech PAC, supports the notion that the γ- and θ-bands subserve, respectively, shared and unique encoding processes across tasks.

Six-dimensional dynamic tractography atlas of language connectivity in the developing brain

During a verbal conversation, our brain moves through a series of complex linguistic processing stages: sound decoding, semantic comprehension, retrieval of semantically coherent words, and overt production of speech outputs. Each process is thought to be supported by a network consisting of local and long-range connections bridging between major cortical areas. Both temporal and extratemporal lobe regions have functional compartments responsible for distinct language domains, including the perception and production of phonological and semantic components. This study provides quantitative evidence of how directly connected inter-lobar neocortical networks support distinct stages of linguistic processing across brain development. Novel six-dimensional tractography was used to intuitively visualize the strength and temporal dynamics of direct inter-lobar effective connectivity between cortical areas activated during each linguistic processing stage. We analysed 3401 non-epileptic intracranial electrode sites from 37 children with focal epilepsy (aged 5-20 years) who underwent extra-operative electrocorticography recording. Principal component analysis of auditory naming-related high-gamma modulations determined the relative involvement of each cortical area during each linguistic processing stage. To quantify direct effective connectivity, we delivered single-pulse electrical stimulation to 488 temporal and 1581 extratemporal lobe sites and measured the early cortico-cortical spectral responses at distant electrodes. Mixed model analyses determined the effects of naming-related high-gamma co-augmentation between connecting regions, age, and cerebral hemisphere on the strength of effective connectivity independent of epilepsy-related factors. Direct effective connectivity was strongest between extratemporal and temporal lobe site pairs, which were simultaneously activated between sentence offset and verbal response onset (i.e. response preparation period); this connectivity was approximately twice more robust than that with temporal lobe sites activated during stimulus listening or overt response. Conversely, extratemporal lobe sites activated during overt response were equally connected with temporal lobe language sites. Older age was associated with increased strength of inter-lobar effective connectivity especially between those activated during response preparation. The arcuate fasciculus supported approximately two-thirds of the direct effective connectivity pathways from temporal to extratemporal auditory language-related areas but only up to half of those in the opposite direction. The uncinate fasciculus consisted of <2% of those in the temporal-to-extratemporal direction and up to 6% of those in the opposite direction. We, for the first time, provided an atlas which quantifies and animates the strength, dynamics, and direction specificity of inter-lobar neural communications between language areas via the white matter pathways. Language-related effective connectivity may be strengthened in an age-dependent manner even after the age of 5.

The Effects of Classification Method and Electrode Configuration on EEG-based Silent Speech Classification

The effective classification for imagined speech and intended speech is of great help to the development of speech-based brain-computer interfaces (BCIs). This work distinguished imagined speech and intended speech by employing the cortical EEG signals recorded from scalp. EEG signals from eleven subjects were recorded when they produced Mandarin-Chinese monosyllables in imagined speech and intended speech, and EEG features were classified by the common spatial pattern, time-domain, frequency-domain and Riemannian manifold based methods. The classification results indicated that the Riemannian manifold based method yielded the highest classification accuracy of 85.9% among the four classification methods. Moreover, the classification accuracy with the left-only brain electrode configuration was close to that with the whole brain electrode configuration. The findings of this work have potential to extend the output commands of silent speech interfaces.

Imagined speech increases the hemodynamic response and functional connectivity of the dorsal motor cortex

Objective. Decoding imagined speech from brain signals could provide a more natural, user-friendly way for developing the next generation of the brain-computer interface (BCI). With the advantages of non-invasive, portable, relatively high spatial resolution and insensitivity to motion artifacts, the functional near-infrared spectroscopy (fNIRS) shows great potential for developing the non-invasive speech BCI. However, there is a lack of fNIRS evidence in uncovering the neural mechanism of imagined speech. Our goal is to investigate the specific brain regions and the corresponding cortico-cortical functional connectivity features during imagined speech with fNIRS.Approach. fNIRS signals were recorded from 13 subjects' bilateral motor and prefrontal cortex during overtly and covertly repeating words. Cortical activation was determined through the mean oxygen-hemoglobin concentration changes, and functional connectivity was calculated by Pearson's correlation coefficient.Main results. (a) The bilateral dorsal motor cortex was significantly activated during the covert speech, whereas the bilateral ventral motor cortex was significantly activated during the overt speech. (b) As a subregion of the motor cortex, sensorimotor cortex (SMC) showed a dominant dorsal response to covert speech condition, whereas a dominant ventral response to overt speech condition. (c) Broca's area was deactivated during the covert speech but activated during the overt speech. (d) Compared to overt speech, dorsal SMC(dSMC)-related functional connections were enhanced during the covert speech.Significance. We provide fNIRS evidence for the involvement of dSMC in speech imagery. dSMC is the speech imagery network's key hub and is probably involved in the sensorimotor information processing during the covert speech. This study could inspire the BCI community to focus on the potential contribution of dSMC during speech imagery.

Detection of human white matter activation and evaluation of its function in movement decoding using stereo-electroencephalography (SEEG)

Objective. White matter tissue takes up approximately 50% of the human brain volume and it is widely known as a messenger conducting information between areas of the central nervous system. However, the characteristics of white matter neural activity and whether white matter neural recordings can contribute to movement decoding are often ignored and still remain largely unknown. In this work, we make quantitative analyses to investigate these two important questions using invasive neural recordings.Approach. We recorded stereo-electroencephalography (SEEG) data from 32 human subjects during a visually-cued motor task, where SEEG recordings can tap into gray and white matter electrical activity simultaneously. Using the proximal tissue density method, we identified the location (i.e. gray or white matter) of each SEEG contact. Focusing on alpha oscillatory and high gamma activities, we compared the activation patterns between gray matter and white matter. Then, we evaluated the performance of such white matter activation in movement decoding.Main results. The results show that white matter also presents activation under the task, in a similar way with the gray matter but at a significantly lower amplitude. Additionally, this work also demonstrates that combing white matter neural activities together with that of gray matter significantly promotes the movement decoding accuracy than using gray matter signals only.Significance. Taking advantage of SEEG recordings from a large number of subjects, we reveal the response characteristics of white matter neural signals under the task and demonstrate its enhancing function in movement decoding. This study highlights the importance of taking white matter activities into consideration in further scientific research and translational applications.

Decoding imagined speech from EEG signals using hybrid-scale spatial-temporal dilated convolution network

Objective.Directly decoding imagined speech from electroencephalogram (EEG) signals has attracted much interest in brain-computer interface applications, because it provides a natural and intuitive communication method for locked-in patients. Several methods have been applied to imagined speech decoding, but how to construct spatial-temporal dependencies and capture long-range contextual cues in EEG signals to better decode imagined speech should be considered.Approach.In this study, we propose a novel model called hybrid-scale spatial-temporal dilated convolution network (HS-STDCN) for EEG-based imagined speech recognition. HS-STDCN integrates feature learning from temporal and spatial information into a unified end-to-end model. To characterize the temporal dependencies of the EEG sequences, we adopted a hybrid-scale temporal convolution layer to capture temporal information at multiple levels. A depthwise spatial convolution layer was then designed to construct intrinsic spatial relationships of EEG electrodes, which can produce a spatial-temporal representation of the input EEG data. Based on the spatial-temporal representation, dilated convolution layers were further employed to learn long-range discriminative features for the final classification.Main results.To evaluate the proposed method, we compared the HS-STDCN with other existing methods on our collected dataset. The HS-STDCN achieved an averaged classification accuracy of 54.31% for decoding eight imagined words, which is significantly better than other methods at a significance level of 0.05.Significance.The proposed HS-STDCN model provided an effective approach to make use of both the temporal and spatial dependencies of the input EEG signals for imagined speech recognition. We also visualized the word semantic differences to analyze the impact of word semantics on imagined speech recognition, investigated the important regions in the decoding process, and explored the use of fewer electrodes to achieve comparable performance.

Prediction of Speech Onset by Micro-Electrocorticography of the Human Brain

Recent technological advances show the feasibility of offline decoding speech from neuronal signals, paving the way to the development of chronically implanted speech brain computer interfaces (sBCI). Two key steps that still need to be addressed for the online deployment of sBCI are, on the one hand, the definition of relevant design parameters of the recording arrays, on the other hand, the identification of robust physiological markers of the patient's intention to speak, which can be used to online trigger the decoding process. To address these issues, we acutely recorded speech-related signals from the frontal cortex of two human patients undergoing awake neurosurgery for brain tumors using three different micro-electrocorticographic ([Formula: see text]ECoG) devices. First, we observed that, at the smallest investigated pitch (600[Formula: see text][Formula: see text]m), neighboring channels are highly correlated, suggesting that more closely spaced electrodes would provide some redundant information. Second, we trained a classifier to recognize speech-related motor preparation from high-gamma oscillations (70-150[Formula: see text]Hz), demonstrating that these neuronal signals can be used to reliably predict speech onset. Notably, our model generalized both across subjects and recording devices showing the robustness of its performance. These findings provide crucial information for the design of future online sBCI.

Within-subject reaction time variability: Role of cortical networks and underlying neurophysiological mechanisms

Variations in reaction time are a ubiquitous characteristic of human behavior. Extensively documented, they have been successfully modeled using parameters of the subject or the task, but the neural basis of behavioral reaction time that varies within the same subject and the same task has been minimally studied. In this paper, we investigate behavioral reaction time variance using 28 datasets of direct cortical recordings in humans who engaged in four different types of simple sensory-motor reaction time tasks. Using a previously described technique that can identify the onset of population-level cortical activity and a novel functional connectivity algorithm described herein, we show that the cumulative latency difference of population-level neural activity across the task-related cortical network can explain up to 41% of the trial-by-trial variance in reaction time. Furthermore, we show that reaction time variance may primarily be due to the latencies in specific brain regions and demonstrate that behavioral latency variance is accumulated across the whole task-related cortical network. Our results suggest that population-level neural activity monotonically increases prior to movement execution, and that trial-by-trial changes in that increase are, in part, accounted for by inhibitory activity indexed by low-frequency oscillations. This pre-movement neural activity explains 19% of the measured variance in neural latencies in our data. Thus, our study provides a mechanistic explanation for a sizable fraction of behavioral reaction time when the subject's task is the same from trial to trial.

Simulating lesion-dependent functional recovery mechanisms

Functional recovery after brain damage varies widely and depends on many factors, including lesion site and extent. When a neuronal system is damaged, recovery may occur by engaging residual (e.g., perilesional) components. When damage is extensive, recovery depends on the availability of other intact neural structures that can reproduce the same functional output (i.e., degeneracy). A system's response to damage may occur rapidly, require learning or both. Here, we simulate functional recovery from four different types of lesions, using a generative model of word repetition that comprised a default premorbid system and a less used alternative system. The synthetic lesions (i) completely disengaged the premorbid system, leaving the alternative system intact, (ii) partially damaged both premorbid and alternative systems, and (iii) limited the experience-dependent plasticity of both. The results, across 1000 trials, demonstrate that (i) a complete disconnection of the premorbid system naturally invoked the engagement of the other, (ii) incomplete damage to both systems had a much more devastating long-term effect on model performance and (iii) the effect of reducing learning capacity within each system. These findings contribute to formal frameworks for interpreting the effect of different types of lesions.

Sufficient sampling for kriging prediction of cortical potential in rat, monkey, and human µECoG

Objective. Large channel count surface-based electrophysiology arrays (e.g. µECoG) are high-throughput neural interfaces with good chronic stability. Electrode spacing remains ad hoc due to redundancy and nonstationarity of field dynamics. Here, we establish a criterion for electrode spacing based on the expected accuracy of predicting unsampled field potential from sampled sites.Approach. We applied spatial covariance modeling and field prediction techniques based on geospatial kriging to quantify sufficient sampling for thousands of 500 ms µECoG snapshots in human, monkey, and rat. We calculated a probably approximately correct (PAC) spacing based on kriging that would be required to predict µECoG fields at≤10% error for most cases (95% of observations).Main results. Kriging theory accurately explained the competing effects of electrode density and noise on predicting field potential. Across five frequency bands from 4-7 to 75-300 Hz, PAC spacing was sub-millimeter for auditory cortex in anesthetized and awake rats, and posterior superior temporal gyrus in anesthetized human. At 75-300 Hz, sub-millimeter PAC spacing was required in all species and cortical areas.Significance. PAC spacing accounted for the effect of signal-to-noise on prediction quality and was sensitive to the full distribution of non-stationary covariance states. Our results show that µECoG arrays should sample at sub-millimeter resolution for applications in diverse cortical areas and for noise resilience.

Neural representations of imagined speech revealed by frequency-tagged magnetoencephalography responses

Speech mental imagery is a quasi-perceptual experience that occurs in the absence of real speech stimulation. How imagined speech with higher-order structures such as words, phrases and sentences is rapidly organized and internally constructed remains elusive. To address this issue, subjects were tasked with imagining and perceiving poems along with a sequence of reference sounds with a presentation rate of 4 Hz while magnetoencephalography (MEG) recording was conducted. Giving that a sentence in a traditional Chinese poem is five syllables, a sentential rhythm was generated at a distinctive frequency of 0.8 Hz. Using the frequency tagging we concurrently tracked the neural processing timescale to the top-down generation of rhythmic constructs embedded in speech mental imagery and the bottom-up sensory-driven activity that were precisely tagged at the sentence-level rate of 0.8 Hz and a stimulus-level rate of 4 Hz, respectively. We found similar neural responses induced by the internal construction of sentences from syllables with both imagined and perceived poems and further revealed shared and distinct cohorts of cortical areas corresponding to the sentence-level rhythm in imagery and perception. This study supports the view of a common mechanism between imagery and perception by illustrating the neural representations of higher-order rhythmic structures embedded in imagined and perceived speech.

Inner speech

Inner speech travels under many aliases: the inner voice, verbal thought, thinking in words, internal verbalization, "talking in your head," the "little voice in the head," and so on. It is both a familiar element of first-person experience and a psychological phenomenon whose complex cognitive components and distributed neural bases are increasingly well understood. There is evidence that inner speech plays a variety of cognitive roles, from enabling abstract thought, to supporting metacognition, memory, and executive function. One active area of controversy concerns the relation of inner speech to auditory verbal hallucinations (AVHs) in schizophrenia, with a common proposal being that sufferers of AVH misidentify their own inner speech as being generated by someone else. Recently, researchers have used artificial intelligence to translate the neural and neuromuscular signatures of inner speech into corresponding outer speech signals, laying the groundwork for a variety of new applications and interventions. This article is categorized under: Philosophy > Foundations of Cognitive Science Linguistics > Language in Mind and Brain Philosophy > Consciousness Philosophy > Psychological Capacities.

Prefrontal High Gamma in ECoG Tags Periodicity of Musical Rhythms in Perception and Imagination

Rhythmic auditory stimuli are known to elicit matching activity patterns in neural populations. Furthermore, recent research has established the particular importance of high-gamma brain activity in auditory processing by showing its involvement in auditory phrase segmentation and envelope tracking. Here, we use electrocorticographic (ECoG) recordings from eight human listeners to see whether periodicities in high-gamma activity track the periodicities in the envelope of musical rhythms during rhythm perception and imagination. Rhythm imagination was elicited by instructing participants to imagine the rhythm to continue during pauses of several repetitions. To identify electrodes whose periodicities in high-gamma activity track the periodicities in the musical rhythms, we compute the correlation between the autocorrelations (ACCs) of both the musical rhythms and the neural signals. A condition in which participants listened to white noise was used to establish a baseline. High-gamma autocorrelations in auditory areas in the superior temporal gyrus and in frontal areas on both hemispheres significantly matched the autocorrelations of the musical rhythms. Overall, numerous significant electrodes are observed on the right hemisphere. Of particular interest is a large cluster of electrodes in the right prefrontal cortex that is active during both rhythm perception and imagination. This indicates conscious processing of the rhythms’ structure as opposed to mere auditory phenomena. The autocorrelation approach clearly highlights that high-gamma activity measured from cortical electrodes tracks both attended and imagined rhythms.

Multiclass covert speech classification using extreme learning machine

The objective of the proposed research is to classify electroencephalography (EEG) data of covert speech words. Six subjects were asked to perform covert speech tasks i.e mental repetition of four different words i.e 'left', 'right', 'up' and 'down'. Fifty trials for each word recorded for every subject. Kernel-based Extreme Learning Machine (kernel ELM) was used for multiclass and binary classification of EEG signals of covert speech words. We achieved a maximum multiclass and binary classification accuracy of (49.77%) and (85.57%) respectively. The kernel ELM achieves significantly higher accuracy compared to some of the most commonly used classification algorithms in Brain-Computer Interfaces (BCIs). Our findings suggested that covert speech EEG signals could be successfully classified using kernel ELM. This research involving the classification of covert speech words potentially leading to real-time silent speech BCI research.

Electrocorticography reveals spatiotemporal neuronal activation patterns of verbal fluency in patients with epilepsy

Verbal fluency is commonly used to evaluate cognitive dysfunction in a variety of neuropsychiatric diseases, yet the neurobiology underlying performance of this task is incompletely understood. Electrocorticography (ECoG) provides a unique opportunity to investigate temporal activation patterns during cognitive tasks with high spatial and temporal precision. We used ECoG to study high gamma activity (HGA) patterns in patients undergoing presurgical evaluation for intractable epilepsy as they completed an overt, free-recall verbal fluency task. We examined regions demonstrating changes in HGA during specific timeframes relative to speech onset. Early pre-speech high gamma activity was present in left frontal regions during letter fluency and in bifrontal regions during category fluency. During timeframes typically associated with word planning, a distributed network was engaged including left inferior frontal, orbitofrontal and posterior temporal regions. Peri-Rolandic activation was observed during speech onset, and there was post-speech activation in the bilateral posterior superior temporal regions. Based on these observations in the context of prior studies, we propose a model of neocortical activity patterns underlying verbal fluency.

Human cortical networking by probabilistic and frequency-specific coupling

Large-scale cortical networking patterns have been established based on the correlation of slow fluctuations of resting fMRI signals. However, the electrophysiological mechanism of cortical networking remained to be elucidated. With large-scale human ECoG recording, we developed a novel approach for functional network parcellation on the basis of probabilistic co-activation of cortical sites in spatio-temporal microstates. The parcellated networks were verified by electrical cortical stimulation (ECS) and somatosensory evoked potentials recording, which showed significantly higher accuracy than the traditional long-term correlation method. This provides direct electrophysiological evidence supporting the dynamic nature of cortical networking. Further analysis revealed that the brain-wide connectivity is likely established on the coupling of ECoG power envelop over a common carrier frequency ranging from alpha to low-beta (8-32Hz). Surprisingly, the cortical networking pattern over this specific frequency was found to be consistent across various tasks, which resembles the resting networks. The high similarity between the above functional network parcellation and the fMRI resting network atlas in individuals also suggested the slow power-envelope coupling of band-limited neural oscillations as the electrophysiological basis of spontaneous BOLD signals. Collectively, our findings on direct human recording revealed a probabilistic and frequency specific coupling mechanism for large-scale cortical networking shared by task and resting brain.

Transformation of speech sequences in human sensorimotor circuits

After we listen to a series of words, we can silently replay them in our mind. Does this mental replay involve a reactivation of our original perceptual dynamics? We recorded electrocorticographic (ECoG) activity across the lateral cerebral cortex as people heard and then mentally rehearsed spoken sentences. For each region, we tested whether silent rehearsal of sentences involved reactivation of sentence-specific representations established during perception or transformation to a distinct representation. In sensorimotor and premotor cortex, we observed reliable and temporally precise responses to speech; these patterns transformed to distinct sentence-specific representations during mental rehearsal. In contrast, we observed less reliable and less temporally precise responses in prefrontal and temporoparietal cortex; these higher-order representations, which were sensitive to sentence semantics, were shared across perception and rehearsal of the same sentence. The mental rehearsal of natural speech involves the transformation of stimulus-locked speech representations in sensorimotor and premotor cortex, combined with diffuse reactivation of higher-order semantic representations.

Comparison between covert sound-production task (sound-imagery) vs. motor-imagery for onset detection in real-life online self-paced BCIs

BACKGROUND

Even though the BCI field has quickly grown in the last few years, it is still mainly investigated as a research area. Increased practicality and usability are required to move BCIs to the real-world. Self-paced (SP) systems would reduce the problem but there is still the big challenge of what is known as the 'onset detection problem'.

METHODS

Our previous studies showed how a new sound-imagery (SI) task, high-tone covert sound production, is very effective for onset detection scenarios and we expect there are several advantages over most common asynchronous approaches used thus far, i.e., motor-imagery (MI): 1) Intuitiveness; 2) benefits to people with motor disabilities and, especially, those with lesions on cortical motor areas; and 3) no significant overlap with other common, spontaneous cognitive states, making it easier to use in daily-life situations. The approach was compared with MI tasks in online real-life scenarios, i.e., during activities such as watching videos and reading text. In our scenario, when a new message prompt from a messenger program appeared on the screen, participants watching a video (or reading text, browsing images) were asked to open the message by executing the SI or MI tasks, respectively, for each experimental condition.

RESULTS

The results showed the SI task performed statistically significantly better than the MI approach: 84.04% (SI) vs 66.79 (MI) True-False positive rate for the sliding image scenario, 80.84% vs 61.07% for watching video. The classification performance difference between SI and MI was found not to be significant in the text-reading scenario. Furthermore, the onset response speed showed SI (4.08 s) being significantly faster than MI (5.46 s). In terms of basic usability, 75% of subjects found SI easier to use.

CONCLUSIONS

Our novel SI task outperforms typical MI for SP onset detection BCIs, therefore it would be more easily used in daily-life situations. This could be a significant step forward for the BCI field which has so far been mainly restricted to research-oriented indoor laboratory settings.

Multimaterial and multifunctional neural interfaces: from surface-type and implantable electrodes to fiber-based devices

Development of neural interfaces from surface electrodes to fibers with various type, functionality, and materials.

Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis

Speaking is a sensorimotor behavior whose neural basis is difficult to study with single neuron resolution due to the scarcity of human intracortical measurements. We used electrode arrays to record from the motor cortex 'hand knob' in two people with tetraplegia, an area not previously implicated in speech. Neurons modulated during speaking and during non-speaking movements of the tongue, lips, and jaw. This challenges whether the conventional model of a 'motor homunculus' division by major body regions extends to the single-neuron scale. Spoken words and syllables could be decoded from single trials, demonstrating the potential of intracortical recordings for brain-computer interfaces to restore speech. Two neural population dynamics features previously reported for arm movements were also present during speaking: a component that was mostly invariant across initiating different words, followed by rotatory dynamics during speaking. This suggests that common neural dynamical motifs may underlie movement of arm and speech articulators.

A Penny for Your Thoughts: Children's Inner Speech and Its Neuro-Development

Inner speech emerges in early childhood, in parallel with the maturation of the dorsal language stream. To date, the developmental relations between these two processes have not been examined. We review evidence that the dorsal language stream has a role in supporting the psychological phenomenon of inner speech, before considering pediatric studies of the dorsal stream's anatomical development and evidence for its emerging functional roles. We examine possible causal accounts of the relations between these two developmental processes and consider their implications for phylogenetic theories about the evolution of inner speech and the accounts of the ontogenetic relations between language and cognition.

Low-Frequency Oscillations Code Speech during Verbal Working Memory

The way the human brain represents speech in memory is still unknown. An obvious characteristic of speech is its evolvement over time. During speech processing, neural oscillations are modulated by the temporal properties of the acoustic speech signal, but also acquired knowledge on the temporal structure of language influences speech perception-related brain activity. This suggests that speech could be represented in the temporal domain, a form of representation that the brain also uses to encode autobiographic memories. Empirical evidence for such a memory code is lacking. We investigated the nature of speech memory representations using direct cortical recordings in the left perisylvian cortex during delayed sentence reproduction in female and male patients undergoing awake tumor surgery. Our results reveal that the brain endogenously represents speech in the temporal domain. Temporal pattern similarity analyses revealed that the phase of frontotemporal low-frequency oscillations, primarily in the beta range, represents sentence identity in working memory. The positive relationship between beta power during working memory and task performance suggests that working memory representations benefit from increased phase separation.SIGNIFICANCE STATEMENT Memory is an endogenous source of information based on experience. While neural oscillations encode autobiographic memories in the temporal domain, little is known on their contribution to memory representations of human speech. Our electrocortical recordings in participants who maintain sentences in memory identify the phase of left frontotemporal beta oscillations as the most prominent information carrier of sentence identity. These observations provide evidence for a theoretical model on speech memory representations and explain why interfering with beta oscillations in the left inferior frontal cortex diminishes verbal working memory capacity. The lack of sentence identity coding at the syllabic rate suggests that sentences are represented in memory in a more abstract form compared with speech coding during speech perception and production.

Spatial-Temporal Dynamics of the Sensorimotor Cortex: Sustained and Transient Activity

How the sensorimotor cortex is organized with respect to controlling different features of movement is unclear. One unresolved question concerns the relation between the duration of an action and the duration of the associated neuronal activity change in the sensorimotor cortex. Using subdural electrocorticography electrodes, we investigated in five subjects, whether high frequency band (HFB; 75-135 Hz) power changes have a transient or sustained relation to speech duration, during pronunciation of the Dutch /i/ vowel with different durations. We showed that the neuronal activity patterns recorded from the sensorimotor cortex can be directly related to action duration in some locations, whereas in other locations, during the same action, neuronal activity is transient, with a peak in HFB activity at movement onset and/or offset. This data sheds light on the neural underpinnings of motor actions and we discuss the possible mechanisms underlying these different response types.

The peri-Sylvian cortical network underlying single word repetition revealed by electrocortical stimulation and direct neural recordings

Verbal repetition requires the coordination of auditory, memory, linguistic, and motor systems. To date, the basic dynamics of neural information processing in this deceptively simple behavior are largely unknown. Here, we examined the neural processes underlying verbal repetition using focal interruption (electrocortical stimulation) in 58 patients undergoing awake craniotomies, and neurophysiological recordings (electrocorticography) in 8 patients while they performed a single word repetition task. Electrocortical stimulation revealed that sub-components of the left peri-Sylvian network involved in single word repetition could be differentially interrupted, producing transient perceptual deficits, paraphasic errors, or speech arrest. Electrocorticography revealed the detailed spatio-temporal dynamics of cortical activation, involving a highly-ordered, but overlapping temporal progression of cortical high gamma (75-150Hz) activity throughout the peri-Sylvian cortex. We observed functionally distinct serial and parallel cortical processing corresponding to successive stages of general auditory processing (posterior superior temporal gyrus), speech-specific auditory processing (middle and posterior superior temporal gyrus), working memory (inferior frontal cortex), and motor articulation (sensorimotor cortex). Together, these methods reveal the dynamics of coordinated activity across peri-Sylvian cortex during verbal repetition.

Human motor cortex relies on sparse and action-specific activation during laughing, smiling and speech production

Smiling, laughing, and overt speech production are fundamental to human everyday communication. However, little is known about how the human brain achieves the highly accurate and differentiated control of such orofacial movement during natural conditions. Here, we utilized the high spatiotemporal resolution of subdural recordings to elucidate how human motor cortex is functionally engaged during control of real-life orofacial motor behaviour. For each investigated movement class-lip licking, speech production, laughing and smiling-our findings reveal a characteristic brain activity pattern within the mouth motor cortex with both spatial segregation and overlap between classes. Our findings thus show that motor cortex relies on sparse and action-specific activation during real-life orofacial behaviour, apparently organized in distinct but overlapping subareas that control different types of natural orofacial movements.

Keyword Spotting Using Human Electrocorticographic Recordings

Neural keyword spotting could form the basis of a speech brain-computer-interface for menu-navigation if it can be done with low latency and high specificity comparable to the “wake-word” functionality of modern voice-activated AI assistant technologies. This study investigated neural keyword spotting using motor representations of speech via invasively-recorded electrocorticographic signals as a proof-of-concept. Neural matched filters were created from monosyllabic consonant-vowel utterances: one keyword utterance, and 11 similar non-keyword utterances. These filters were used in an analog to the acoustic keyword spotting problem, applied for the first time to neural data. The filter templates were cross-correlated with the neural signal, capturing temporal dynamics of neural activation across cortical sites. Neural vocal activity detection (VAD) was used to identify utterance times and a discriminative classifier was used to determine if these utterances were the keyword or non-keyword speech. Model performance appeared to be highly related to electrode placement and spatial density. Vowel height (/a/ vs /i/) was poorly discriminated in recordings from sensorimotor cortex, but was highly discriminable using neural features from superior temporal gyrus during self-monitoring. The best performing neural keyword detection (5 keyword detections with two false-positives across 60 utterances) and neural VAD (100% sensitivity, ~1 false detection per 10 utterances) came from high-density (2 mm electrode diameter and 5 mm pitch) recordings from ventral sensorimotor cortex, suggesting the spatial fidelity and extent of high-density ECoG arrays may be sufficient for the purpose of speech brain-computer-interfaces.

Progress in the Field of Micro-Electrocorticography

Since the 1940s electrocorticography (ECoG) devices and, more recently, in the last decade, micro-electrocorticography (µECoG) cortical electrode arrays were used for a wide set of experimental and clinical applications, such as epilepsy localization and brain⁻computer interface (BCI) technologies. Miniaturized implantable µECoG devices have the advantage of providing greater-density neural signal acquisition and stimulation capabilities in a minimally invasive fashion. An increased spatial resolution of the µECoG array will be useful for greater specificity diagnosis and treatment of neuronal diseases and the advancement of basic neuroscience and BCI research. In this review, recent achievements of ECoG and µECoG are discussed. The electrode configurations and varying material choices used to design µECoG arrays are discussed, including advantages and disadvantages of µECoG technology compared to electroencephalography (EEG), ECoG, and intracortical electrode arrays. Electrode materials that are the primary focus include platinum, iridium oxide, poly(3,4-ethylenedioxythiophene) (PEDOT), indium tin oxide (ITO), and graphene. We discuss the biological immune response to µECoG devices compared to other electrode array types, the role of µECoG in clinical pathology, and brain⁻computer interface technology. The information presented in this review will be helpful to understand the current status, organize available knowledge, and guide future clinical and research applications of µECoG technologies.

The Potential for a Speech Brain-Computer Interface Using Chronic Electrocorticography

A brain-computer interface (BCI) is a technology that uses neural features to restore or augment the capabilities of its user. A BCI for speech would enable communication in real time via neural correlates of attempted or imagined speech. Such a technology would potentially restore communication and improve quality of life for locked-in patients and other patients with severe communication disorders. There have been many recent developments in neural decoders, neural feature extraction, and brain recording modalities facilitating BCI for the control of prosthetics and in automatic speech recognition (ASR). Indeed, ASR and related fields have developed significantly over the past years, and many lend many insights into the requirements, goals, and strategies for speech BCI. Neural speech decoding is a comparatively new field but has shown much promise with recent studies demonstrating semantic, auditory, and articulatory decoding using electrocorticography (ECoG) and other neural recording modalities. Because the neural representations for speech and language are widely distributed over cortical regions spanning the frontal, parietal, and temporal lobes, the mesoscopic scale of population activity captured by ECoG surface electrode arrays may have distinct advantages for speech BCI, in contrast to the advantages of microelectrode arrays for upper-limb BCI. Nevertheless, there remain many challenges for the translation of speech BCIs to clinical populations. This review discusses and outlines the current state-of-the-art for speech BCI and explores what a speech BCI using chronic ECoG might entail.

The Relative Contribution of High-Gamma Linguistic Processing Stages of Word Production, and Motor Imagery of Articulation in Class Separability of Covert Speech Tasks in EEG Data

Word production begins with high-Gamma automatic linguistic processing functions followed by speech motor planning and articulation. Phonetic properties are processed in both linguistic and motor stages of word production. Four phonetically dissimilar phonemic structures “BA”, “FO”, “LE”, and “RY” were chosen as covert speech tasks. Ten neurologically healthy volunteers with the age range of 21–33 participated in this experiment. Participants were asked to covertly speak a phonemic structure when they heard an auditory cue. EEG was recorded with 64 electrodes at 2048 samples/s. Initially, one-second trials were used, which contained linguistic and motor imagery activities. The four-class true positive rate was calculated. In the next stage, 312 ms trials were used to exclude covert articulation from analysis. By eliminating the covert articulation stage, the four-class grand average classification accuracy dropped from 96.4% to 94.5%. The most valuable features emerge after Auditory cue recognition (~100 ms post onset), and within the 70–128 Hz frequency range. The most significant identified brain regions were the Prefrontal Cortex (linked to stimulus driven executive control), Wernicke’s area (linked to Phonological code retrieval), the right IFG, and Broca’s area (linked to syllabification). Alpha and Beta band oscillations associated with motor imagery do not contain enough information to fully reflect the complexity of speech movements. Over 90% of the most class-dependent features were in the 30-128 Hz range, even during the covert articulation stage. As a result, compared to linguistic functions, the contribution of motor imagery of articulation in class separability of covert speech tasks from EEG data is negligible.

Differential Representation of Articulatory Gestures and Phonemes in Precentral and Inferior Frontal Gyri

Speech is a critical form of human communication and is central to our daily lives. Yet, despite decades of study, an understanding of the fundamental neural control of speech production remains incomplete. Current theories model speech production as a hierarchy from sentences and phrases down to words, syllables, speech sounds (phonemes), and the actions of vocal tract articulators used to produce speech sounds (articulatory gestures). Here, we investigate the cortical representation of articulatory gestures and phonemes in ventral precentral and inferior frontal gyri in men and women. Our results indicate that ventral precentral cortex represents gestures to a greater extent than phonemes, while inferior frontal cortex represents both gestures and phonemes. These findings suggest that speech production shares a common cortical representation with that of other types of movement, such as arm and hand movements. This has important implications both for our understanding of speech production and for the design of brain-machine interfaces to restore communication to people who cannot speak.SIGNIFICANCE STATEMENT Despite being studied for decades, the production of speech by the brain is not fully understood. In particular, the most elemental parts of speech, speech sounds (phonemes) and the movements of vocal tract articulators used to produce these sounds (articulatory gestures), have both been hypothesized to be encoded in motor cortex. Using direct cortical recordings, we found evidence that primary motor and premotor cortices represent gestures to a greater extent than phonemes. Inferior frontal cortex (part of Broca's area) appears to represent both gestures and phonemes. These findings suggest that speech production shares a similar cortical organizational structure with the movement of other body parts.