Human cortical encoding of pitch in tonal and non-tonal languages

TMNRED, A Chinese Language EEG Dataset for Fuzzy Semantic Target Identification in Natural Reading Environments

Semantic understanding is central to advanced cognitive functions, and the mechanisms by which the brain processes language information are still being explored. Existing EEG datasets often lack natural reading data specific to Chinese, limiting research on Chinese semantic decoding and natural language processing. This study aims to construct a Chinese natural reading EEG dataset, TMNRED, for semantic target identification in natural reading environments. TMNRED was collected from 30 participants reading sentences sourced from public internet resources and media reports. Each participant underwent 400-450 trials in a single day, resulting in a dataset with over 10 hours of continuous EEG data and more than 4000 trials. This dataset provides valuable physiological data for studying Chinese semantics and developing more accurate Chinese natural language processing models.

VocalMind: A Stereotactic EEG Dataset for Vocalized, Mimed, and Imagined Speech in Tonal Language

Speech BCIs based on implanted electrodes hold significant promise for enhancing spoken communication through high temporal resolution and invasive neural sensing. Despite the potential, acquiring such data is challenging due to its invasive nature, and publicly available datasets, particularly for tonal languages, are limited. In this study, we introduce VocalMind, a stereotactic electroencephalography (sEEG) dataset focused on Mandarin Chinese, a tonal language. This dataset includes sEEG-speech parallel recordings from three distinct speech modes, namely vocalized speech, mimed speech, and imagined speech, at both word and sentence levels, totaling over one hour of intracranial neural recordings related to speech production. This paper also presents a baseline model as the reference model for future studies, at the same time, ensuring the integrity of the dataset. The diversity of tasks and the substantial data volume provide a valuable resource for developing advanced algorithms for speech decoding, thereby advancing BCI research for spoken communication.

Cortical processing of discrete prosodic patterns in continuous speech

Prosody has a vital function in speech, structuring a speaker's intended message for the listener. The superior temporal gyrus (STG) is considered a critical hub for prosody, but the role of earlier auditory regions like Heschl's gyrus (HG), associated with pitch processing, remains unclear. Using intracerebral recordings in humans and non-human primate models, we investigated prosody processing in narrative speech, focusing on pitch accents-abstract phonological units that signal word prominence and communicative intent. In humans, HG encoded pitch accents as abstract representations beyond spectrotemporal features, distinct from segmental speech processing, and outperforms STG in disambiguating pitch accents. Multivariate models confirm HG's unique representation of pitch accent categories. In the non-human primate, pitch accents were not abstractly encoded, despite robust spectrotemporal processing, highlighting the role of experience in shaping abstract representations. These findings emphasize a key role for the HG in early prosodic abstraction and advance our understanding of human speech processing.

Parallel hierarchical encoding of linguistic representations in the human auditory cortex and recurrent automatic speech recognition systems

The human brain's ability to transform acoustic speech signals into rich linguistic representations has inspired advancements in automatic speech recognition (ASR) systems. While ASR systems now achieve human-level performance under controlled conditions, prior research on their parallels with the brain has been limited by the use of biologically implausible models, narrow feature sets, and comparisons that primarily emphasize predictability of brain activity without fully exploring shared underlying representations. Additionally, studies comparing the brain to text-based language models overlook the acoustic stages of speech processing, an essential part in transforming sound to meaning. Leveraging high-resolution intracranial recordings and a recurrent ASR model, this study bridges these gaps by uncovering a striking correspondence in the hierarchical encoding of linguistic features, from low-level acoustic signals to high-level semantic processing. Specifically, we demonstrate that neural activity in distinct regions of the auditory cortex aligns with representations in corresponding layers of the ASR model and, crucially, that both systems encode similar features at each stage of processing-from acoustic to phonetic, lexical, and semantic information. These findings suggest that both systems, despite their distinct architectures, converge on similar strategies for language processing, providing insight in the optimal computational principles underlying linguistic representation and the shared constraints shaping human and artificial speech processing.

Atypical vocal imitation of speech and song in autism spectrum disorder: Evidence from Mandarin speakers

LAY ABSTRACT

Atypical vocal imitation has been identified in English-speaking autistic individuals, whereas the characteristics of vocal imitation in tone-language-speaking autistic individuals remain unexplored. By comparing speech and song imitation, the present study reveals a unique pattern of atypical vocal imitation across speech and music domains among Mandarin-speaking autistic individuals. The findings suggest that tone language experience does not compensate for difficulties in vocal imitation in autistic individuals and extends our understanding of vocal imitation in autism across different languages.

Humans can use positive and negative spectrotemporal correlations to detect rising and falling pitch

To discern speech or appreciate music, the human auditory system detects how pitch increases or decreases over time. However, the algorithms used to detect changes in pitch, or pitch motion, are incompletely understood. Here, using psychophysics, computational modeling, functional neuroimaging, and analysis of recorded speech, we ask if humans can detect pitch motion using computations analogous to those used by the visual system. We adapted stimuli from studies of vision to create novel auditory correlated noise stimuli that elicited robust pitch motion percepts. Crucially, these stimuli are inharmonic and possess no persistent features across frequency or time, but do possess positive or negative local spectrotemporal correlations in intensity. In psychophysical experiments, we found clear evidence that humans can judge pitch direction based only on positive or negative spectrotemporal intensity correlations. The key behavioral result-robust sensitivity to the negative spectrotemporal correlations-is a direct analogue of illusory "reverse-phi" motion in vision, and thus constitutes a new auditory illusion. Our behavioral results and computational modeling led us to hypothesize that human auditory processing may employ pitch direction opponency. fMRI measurements in auditory cortex supported this hypothesis. To link our psychophysical findings to real-world pitch perception, we analyzed recordings of English and Mandarin speech and found that pitch direction was robustly signaled by both positive and negative spectrotemporal correlations, suggesting that sensitivity to both types of correlations confers ecological benefits. Overall, this work reveals how motion detection algorithms sensitive to local correlations are deployed by the central nervous system across disparate modalities (vision and audition) and dimensions (space and frequency).

Neural representations of phonological information in bilingual language production

Previous research has explored the neural mechanisms of bilinguals' language production, but most studies focused on neural mechanisms of cognitive control during language production. Therefore, it is unclear which brain regions represent lexical information (especially phonological information) during production and how they are affected by language context. To address those questions, we used representational similarity analysis to explore neural representations of phonological information in native (L1) and second languages (L2) in the single- and mixed-language contexts, respectively. Results showed that Chinese-English bilinguals behaviorally performed worse and exhibited more activations in brain regions associated with language processing and cognitive control in the mixed-language context relative to the single-language context. Further representational similarity analysis revealed that phonological representations of L1 were detected in the left pars opercularis, middle frontal gyrus, and anterior supramarginal gyrus, while phonological representations of L2 were detected in the bilateral occipitotemporal cortex regardless of the target language. More interestingly, robust phonological representations of L1 were observed in brain areas related to phonological processing during L2 production regardless of language context. These results provide direct neuroimaging evidence for the nonselective processing hypothesis and highlight the superiority of phonological representations in the dominant language during bilingual language production.

Dynamic degree centrality in stroke-induced Broca's aphasia varies based on first language: A functional MRI study

BACKGROUND AND PURPOSE

This study sought to explore dynamic degree centrality (DC) variability in particular regions of the brain in patients with poststroke Broca aphasia (BA) using a resting-state functional magnetic resonance imaging (rs-fMRI) approach, comparing differences between Uyghur and Chinese BA patients.

METHODS

This study investigated two factors, language and BA status, and divided patients into four groups: Uyghur aphasia patients (UA), Uyghur normal control subjects (UN), Chinese aphasia patients (CA), and Chinese normal subjects (CN) who underwent rs-fMRI analysis. Two-way analysis of variance (ANOVA) was used to calculate the comprehensive differences in dynamic DC among these four groups. Correlations between DC and language behavior were assessed with partial correlation analyses.

RESULTS

Two-way ANOVA revealed comparable results for the results of pairwise comparisons of dynamic DC variability among the four groups in the right middle frontal gyrus/orbital part (ORBmid.R), right superior frontal gyrus/dorsolateral, and right precuneus (PCUN.R), with results as follows: UA < UN, CA > CN, UA < CA, and UN > CN (p < .05, with the exception of the p-values for UA and UN in superior frontal gyrus/dorsolateral). In contrast, the opposite results were observed for the right calcarine fissure and surrounding cortex (CAL.R, p < .05).

CONCLUSION

The observed enhancement of dynamic DC variability in ORBmid.R and PCUN.R among Chinese BA patients and in CAL.R in Uyghur BA patients may be attributable to language network restructuring. Overall, these results suggest that BA patients who use different language families may exhibit differences in the network mechanisms that characterize observed impairments of language function.

A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages

Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.

Perception and adaptation of receptive prosody in autistic adolescents

A fundamental aspect of language processing is inferring others' minds from subtle variations in speech. The same word or sentence can often convey different meanings depending on its tempo, timing, and intonation-features often referred to as prosody. Although autistic children and adults are known to experience difficulty in making such inferences, the science remains unclear as to why. We hypothesize that detail-oriented perception in autism may interfere with the inference process if it lacks the adaptivity required to cope with the variability ubiquitous in human speech. Using a novel prosodic continuum that shifts the sentence meaning gradiently from a statement (e.g., "It's raining") to a question (e.g., "It's raining?"), we have investigated the perception and adaptation of receptive prosody in autistic adolescents and two groups of non-autistic controls. Autistic adolescents showed attenuated adaptivity in categorizing prosody, whereas they were equivalent to controls in terms of discrimination accuracy. Combined with recent findings in segmental (e.g., phoneme) recognition, the current results provide the basis for an emerging research framework for attenuated flexibility and reduced influence of contextual feedback as a possible source of deficits that hinder linguistic and social communication in autism.

Individual differences in working memory impact the trajectory of non-native speech category learning

What is the role of working memory over the course of non-native speech category learning? Prior work has predominantly focused on how working memory might influence learning assessed at a single timepoint. Here, we substantially extend this prior work by examining the role of working memory on speech learning performance over time (i.e., over several months) and leverage a multifaceted approach that provides key insights into how working memory influences learning accuracy, maintenance of knowledge over time, generalization ability, and decision processes. We found that the role of working memory in non-native speech learning depends on the timepoint of learning and whether individuals learned the categories at all. Among learners, across all stages of learning, working memory was associated with higher accuracy as well as faster and slightly more cautious decision making. Further, while learners and non-learners did not have substantially different working memory performance, learners had faster evidence accumulation and more cautious decision thresholds throughout all sessions. Working memory may enhance learning by facilitating rapid category acquisition in initial stages and enabling faster and slightly more careful decision-making strategies that may reduce the overall effort needed to learn. Our results have important implications for developing interventions to improve learning in naturalistic language contexts.

Genetic association analysis of human median voice pitch identifies a common locus for tonal and non-tonal languages

The genetic influence on human vocal pitch in tonal and non-tonal languages remains largely unknown. In tonal languages, such as Mandarin Chinese, pitch changes differentiate word meanings, whereas in non-tonal languages, such as Icelandic, pitch is used to convey intonation. We addressed this question by searching for genetic associations with interindividual variation in median pitch in a Chinese major depression case-control cohort and compared our results with a genome-wide association study from Iceland. The same genetic variant, rs11046212-T in an intron of the ABCC9 gene, was one of the most strongly associated loci with median pitch in both samples. Our meta-analysis revealed four genome-wide significant hits, including two novel associations. The discovery of genetic variants influencing vocal pitch across both tonal and non-tonal languages suggests the possibility of a common genetic contribution to the human vocal system shared in two distinct populations with languages that differ in tonality (Icelandic and Mandarin).

Encoding of melody in the human auditory cortex

Melody is a core component of music in which discrete pitches are serially arranged to convey emotion and meaning. Perception varies along several pitch-based dimensions: (i) the absolute pitch of notes, (ii) the difference in pitch between successive notes, and (iii) the statistical expectation of each note given prior context. How the brain represents these dimensions and whether their encoding is specialized for music remains unknown. We recorded high-density neurophysiological activity directly from the human auditory cortex while participants listened to Western musical phrases. Pitch, pitch-change, and expectation were selectively encoded at different cortical sites, indicating a spatial map for representing distinct melodic dimensions. The same participants listened to spoken English, and we compared responses to music and speech. Cortical sites selective for music encoded expectation, while sites that encoded pitch and pitch-change in music used the same neural code to represent equivalent properties of speech. Findings reveal how the perception of melody recruits both music-specific and general-purpose sound representations.

Complexity of STG signals and linguistic rhythm: a methodological study for EEG data

The superior temporal and the Heschl's gyri of the human brain play a fundamental role in speech processing. Neurons synchronize their activity to the amplitude envelope of the speech signal to extract acoustic and linguistic features, a process known as neural tracking/entrainment. Electroencephalography has been extensively used in language-related research due to its high temporal resolution and reduced cost, but it does not allow for a precise source localization. Motivated by the lack of a unified methodology for the interpretation of source reconstructed signals, we propose a method based on modularity and signal complexity. The procedure was tested on data from an experiment in which we investigated the impact of native language on tracking to linguistic rhythms in two groups: English natives and Spanish natives. In the experiment, we found no effect of native language but an effect of language rhythm. Here, we compare source projected signals in the auditory areas of both hemispheres for the different conditions using nonparametric permutation tests, modularity, and a dynamical complexity measure. We found increasing values of complexity for decreased regularity in the stimuli, giving us the possibility to conclude that languages with less complex rhythms are easier to track by the auditory cortex.

Dissecting neural computations in the human auditory pathway using deep neural networks for speech

The human auditory system extracts rich linguistic abstractions from speech signals. Traditional approaches to understanding this complex process have used linear feature-encoding models, with limited success. Artificial neural networks excel in speech recognition tasks and offer promising computational models of speech processing. We used speech representations in state-of-the-art deep neural network (DNN) models to investigate neural coding from the auditory nerve to the speech cortex. Representations in hierarchical layers of the DNN correlated well with the neural activity throughout the ascending auditory system. Unsupervised speech models performed at least as well as other purely supervised or fine-tuned models. Deeper DNN layers were better correlated with the neural activity in the higher-order auditory cortex, with computations aligned with phonemic and syllabic structures in speech. Accordingly, DNN models trained on either English or Mandarin predicted cortical responses in native speakers of each language. These results reveal convergence between DNN model representations and the biological auditory pathway, offering new approaches for modeling neural coding in the auditory cortex.

EEG-based assessment of temporal fine structure and envelope effect in mandarin syllable and tone perception

In recent years, speech perception research has benefited from low-frequency rhythm entrainment tracking of the speech envelope. However, speech perception is still controversial regarding the role of speech envelope and temporal fine structure, especially in Mandarin. This study aimed to discuss the dependence of Mandarin syllables and tones perception on the speech envelope and the temporal fine structure. We recorded the electroencephalogram (EEG) of the subjects under three acoustic conditions using the sound chimerism analysis, including (i) the original speech, (ii) the speech envelope and the sinusoidal modulation, and (iii) the fine structure of time and the modulation of the non-speech (white noise) sound envelope. We found that syllable perception mainly depended on the speech envelope, while tone perception depended on the temporal fine structure. The delta bands were prominent, and the parietal and prefrontal lobes were the main activated brain areas, regardless of whether syllable or tone perception was involved. Finally, we decoded the spatiotemporal features of Mandarin perception from the microstate sequence. The spatiotemporal feature sequence of the EEG caused by speech material was found to be specific, suggesting a new perspective for the subsequent auditory brain-computer interface. These results provided a new scheme for the coding strategy of new hearing aids for native Mandarin speakers.

HIGHLIGHTS

Neural control of lexical tone production in human laryngeal motor cortex

In tonal languages, which are spoken by nearly one-third of the world's population, speakers precisely control the tension of vocal folds in the larynx to modulate pitch in order to distinguish words with completely different meanings. The specific pitch trajectories for a given tonal language are called lexical tones. Here, we used high-density direct cortical recordings to determine the neural basis of lexical tone production in native Mandarin-speaking participants. We found that instead of a tone category-selective coding, local populations in the bilateral laryngeal motor cortex (LMC) encode articulatory kinematic information to generate the pitch dynamics of lexical tones. Using a computational model of tone production, we discovered two distinct patterns of population activity in LMC commanding pitch rising and lowering. Finally, we showed that direct electrocortical stimulation of different local populations in LMC evoked pitch rising and lowering during tone production, respectively. Together, these results reveal the neural basis of vocal pitch control of lexical tones in tonal languages.

Encoding of melody in the human auditory cortex

Melody is a core component of music in which discrete pitches are serially arranged to convey emotion and meaning. Perception of melody varies along several pitch-based dimensions: (1) the absolute pitch of notes, (2) the difference in pitch between successive notes, and (3) the higher-order statistical expectation of each note conditioned on its prior context. While humans readily perceive melody, how these dimensions are collectively represented in the brain and whether their encoding is specialized for music remains unknown. Here, we recorded high-density neurophysiological activity directly from the surface of human auditory cortex while Western participants listened to Western musical phrases. Pitch, pitch-change, and expectation were selectively encoded at different cortical sites, indicating a spatial code for representing distinct dimensions of melody. The same participants listened to spoken English, and we compared evoked responses to music and speech. Cortical sites selective for music were systematically driven by the encoding of expectation. In contrast, sites that encoded pitch and pitch-change used the same neural code to represent equivalent properties of speech. These findings reveal the multidimensional nature of melody encoding, consisting of both music-specific and domain-general sound representations in auditory cortex.

Teaser

The human brain contains both general-purpose and music-specific neural populations for processing distinct attributes of melody.

Mapping cortical activations underlying covert and overt language production using high-density diffuse optical tomography

Gold standard neuroimaging modalities such as functional magnetic resonance imaging (fMRI), positron emission tomography (PET), and more recently electrocorticography (ECoG) have provided profound insights regarding the neural mechanisms underlying the processing of language, but they are limited in applications involving naturalistic language production especially in developing brains, during face-to-face dialogues, or as a brain-computer interface. High-density diffuse optical tomography (HD-DOT) provides high-fidelity mapping of human brain function with comparable spatial resolution to that of fMRI but in a silent and open scanning environment similar to real-life social scenarios. Therefore, HD-DOT has potential to be used in naturalistic settings where other neuroimaging modalities are limited. While HD-DOT has been previously validated against fMRI for mapping the neural correlates underlying language comprehension and covert (i.e., "silent") language production, HD-DOT has not yet been established for mapping the cortical responses to overt (i.e., "out loud") language production. In this study, we assessed the brain regions supporting a simple hierarchy of language tasks: silent reading of single words, covert production of verbs, and overt production of verbs in normal hearing right-handed native English speakers (n = 33). First, we found that HD-DOT brain mapping is resilient to movement associated with overt speaking. Second, we observed that HD-DOT is sensitive to key activations and deactivations in brain function underlying the perception and naturalistic production of language. Specifically, statistically significant results were observed that show recruitment of regions in occipital, temporal, motor, and prefrontal cortices across all three tasks after performing stringent cluster-extent based thresholding. Our findings lay the foundation for future HD-DOT studies of imaging naturalistic language comprehension and production during real-life social interactions and for broader applications such as presurgical language assessment and brain-machine interfaces.

Frequency detection, frequency discrimination, and spectro-temporal pattern perception in older and younger typically hearing adults

Elderly adults often experience difficulties in speech understanding, possibly due to age-related deficits in frequency perception. It is unclear whether age-related deficits in frequency perception differ between the apical or basal regions of the cochlea. It is also unclear how aging might differently affect frequency discrimination or detection of a change in frequency within a stimulus. In the present study, pure-tone frequency thresholds were measured in 19 older (61-74 years) and 20 younger (22-28 years) typically hearing adults. Participants were asked to discriminate between reference and probe frequencies or to detect changes in frequency within a probe stimulus. Broadband spectro-temporal pattern perception was also measured using the spectro-temporal modulated ripple test (SMRT). Frequency thresholds were significantly poorer in the basal than in the apical region of the cochlea; the deficit in the basal region was 2 times larger for the older than for the younger group. Frequency thresholds were significantly poorer in the older group, especially in the basal region where frequency detection thresholds were 3.9 times poorer for the older than for the younger group. SMRT thresholds were 1.5 times better for the younger than for the older group. Significant age effects were observed for SMRT thresholds and for frequency thresholds only in the basal region. SMRT thresholds were significantly correlated with frequency thresholds only in the older group. The poorer frequency and spectro-temporal pattern perception may contribute to age-related deficits in speech perception, even when audiometric thresholds are nearly normal.

Negative-emotion-induced reduction in speech-in-noise recognition is associated with source-monitoring deficits and psychiatric symptoms in mandarin-speaking patients with schizophrenia

BACKGROUND

Patients with schizophrenia (SCH) have deficits in source monitoring (SM), speech-in-noise recognition (SR), and auditory prosody recognition. This study aimed to test the covariation between SM and SR alteration induced by negative prosodies and their association with psychiatric symptoms in SCH.

METHODS

Fifty-four SCH patients and 59 healthy controls (HCs) underwent a speech SM task, an SR task, and the assessment of positive and negative syndrome scale (PANSS). We used the multivariate analyses of partial least squares (PLS) regression to explore the associations among SM (external/internal/new attribution error [AE] and response bias [RB]), SR alteration/release induced by four negative-emotion (sad, angry, fear, and disgust) prosodies of target speech, and psychiatric symptoms.

RESULTS

In SCH, but not HCs, a profile (linear combination) of SM (especially the external-source RB) was positively associated with a profile of SR reductions (induced especially by the angry prosody). Moreover, two SR reduction profiles (especially in the anger and sadness conditions) were related to two profiles of psychiatric symptoms (negative symptoms, lack of insight, and emotional disturbances). The two PLS components explained 50.4% of the total variances of the release-symptom association.

CONCLUSION

Compared to HCs, SCH is more likely to perceive the external-source speech as internal/new source speech. The SM-related SR reduction induced by the angry prosody was mainly associated with negative symptoms. These findings help understand the psychopathology of SCH and may provide a potential direction to improve negative symptoms via minimizing emotional SR reduction in schizophrenia.

Decoding and synthesizing tonal language speech from brain activity

Recent studies have shown that the feasibility of speech brain-computer interfaces (BCIs) as a clinically valid treatment in helping nontonal language patients with communication disorders restore their speech ability. However, tonal language speech BCI is challenging because additional precise control of laryngeal movements to produce lexical tones is required. Thus, the model should emphasize the features from the tonal-related cortex. Here, we designed a modularized multistream neural network that directly synthesizes tonal language speech from intracranial recordings. The network decoded lexical tones and base syllables independently via parallel streams of neural network modules inspired by neuroscience findings. The speech was synthesized by combining tonal syllable labels with nondiscriminant speech neural activity. Compared to commonly used baseline models, our proposed models achieved higher performance with modest training data and computational costs. These findings raise a potential strategy for approaching tonal language speech restoration.

Intact speech perception after resection of dominant hemisphere primary auditory cortex for the treatment of medically refractory epilepsy: illustrative case

BACKGROUND

In classic speech network models, the primary auditory cortex is the source of auditory input to Wernicke's area in the posterior superior temporal gyrus (pSTG). Because resection of the primary auditory cortex in the dominant hemisphere removes inputs to the pSTG, there is a risk of speech impairment. However, recent research has shown the existence of other, nonprimary auditory cortex inputs to the pSTG, potentially reducing the risk of primary auditory cortex resection in the dominant hemisphere.

OBSERVATIONS

Here, the authors present a clinical case of a woman with severe medically refractory epilepsy with a lesional epileptic focus in the left (dominant) Heschl's gyrus. Analysis of neural responses to speech stimuli was consistent with primary auditory cortex localization to Heschl's gyrus. Although the primary auditory cortex was within the proposed resection margins, she underwent lesionectomy with total resection of Heschl's gyrus. Postoperatively, she had no speech deficits and her seizures were fully controlled.

LESSONS

While resection of the dominant hemisphere Heschl's gyrus/primary auditory cortex warrants caution, this case illustrates the ability to resect the primary auditory cortex without speech impairment and supports recent models of multiple parallel inputs to the pSTG.

Auditory attention decoding from EEG-based Mandarin speech envelope reconstruction

In the cocktail party circumstance, the human auditory system extracts the information from a specific speaker of interest and ignores others. Many studies have focused on auditory attention decoding (AAD), but the stimulation materials were mainly non-tonal languages. We used a tonal language (Mandarin) as the speech stimulus and constructed a Long Short-Term Memory (LSTM) architecture for speech envelope reconstruction based on electroencephalogram (EEG) data. The correlation coefficient between the reconstructed and candidate envelopes was calculated to determine the subject's auditory attention. The proposed LSTM architecture outperformed the linear models. The average decoding accuracy in cross-subject and inter-subject cases varies from 63.02 to 74.29%, with the highest accuracy rate of 89.1% in a decision window of 0.15 s. In addition, the beta-band rhythm was found to play an essential role in identifying the attention and the non-attention state. These results provide a new AAD architecture to help develop neuro-steered hearing devices, especially for tonal languages.

Cortical tracking of voice pitch in the presence of multiple speakers depends on selective attention

Voice pitch carries linguistic and non-linguistic information. Previous studies have described cortical tracking of voice pitch in clean speech, with responses reflecting both pitch strength and pitch value. However, pitch is also a powerful cue for auditory stream segregation, especially when competing streams have pitch differing in fundamental frequency, as is the case when multiple speakers talk simultaneously. We therefore investigated how cortical speech pitch tracking is affected in the presence of a second, task-irrelevant speaker. We analyzed human magnetoencephalography (MEG) responses to continuous narrative speech, presented either as a single talker in a quiet background or as a two-talker mixture of a male and a female speaker. In clean speech, voice pitch was associated with a right-dominant response, peaking at a latency of around 100 ms, consistent with previous electroencephalography and electrocorticography results. The response tracked both the presence of pitch and the relative value of the speaker's fundamental frequency. In the two-talker mixture, the pitch of the attended speaker was tracked bilaterally, regardless of whether or not there was simultaneously present pitch in the speech of the irrelevant speaker. Pitch tracking for the irrelevant speaker was reduced: only the right hemisphere still significantly tracked pitch of the unattended speaker, and only during intervals in which no pitch was present in the attended talker's speech. Taken together, these results suggest that pitch-based segregation of multiple speakers, at least as measured by macroscopic cortical tracking, is not entirely automatic but strongly dependent on selective attention.

An investigation across 45 languages and 12 language families reveals a universal language network

To understand the architecture of human language, it is critical to examine diverse languages; however, most cognitive neuroscience research has focused on only a handful of primarily Indo-European languages. Here we report an investigation of the fronto-temporo-parietal language network across 45 languages and establish the robustness to cross-linguistic variation of its topography and key functional properties, including left-lateralization, strong functional integration among its brain regions and functional selectivity for language processing.

Decoding Neural Correlation of Language-Specific Imagined Speech using EEG Signals

Speech impairments due to cerebral lesions and degenerative disorders can be devastating. For humans with severe speech deficits, imagined speech in the brain-computer interface has been a promising hope for reconstructing the neural signals of speech production. However, studies in the EEG-based imagined speech domain still have some limitations due to high variability in spatial and temporal information and low signal-to-noise ratio. In this paper, we investigated the neural signals for two groups of native speakers with two tasks with different languages, English and Chinese. Our assumption was that English, a non-tonal and phonogram-based language, would have spectral differences in neural computation compared to Chinese, a tonal and ideogram-based language. The results showed the significant difference in the relative power spectral density between English and Chinese in specific frequency band groups. Also, the spatial evaluation of Chinese native speakers in the theta band was distinctive during the imagination task. Hence, this paper would suggest the key spectral and spatial information of word imagination with specialized language while decoding the neural signals of speech. Clinical Relevance- Imagined speech-related studies lead to the development of assistive communication technology especially for patients with speech disorders such as aphasia due to brain damage. This study suggests significant spectral features by analyzing cross-language differences of EEG-based imagined speech using two widely used languages.

Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language

Silent speech decoding (SSD), based on articulatory neuromuscular activities, has become a prevalent task of brain–computer interfaces (BCIs) in recent years. Many works have been devoted to decoding surface electromyography (sEMG) from articulatory neuromuscular activities. However, restoring silent speech in tonal languages such as Mandarin Chinese is still difficult. This paper proposes an optimized sequence-to-sequence (Seq2Seq) approach to synthesize voice from the sEMG-based silent speech. We extract duration information to regulate the sEMG-based silent speech using the audio length. Then, we provide a deep-learning model with an encoder–decoder structure and a state-of-the-art vocoder to generate the audio waveform. Experiments based on six Mandarin Chinese speakers demonstrate that the proposed model can successfully decode silent speech in Mandarin Chinese and achieve a character error rate (CER) of 6.41% on average with human evaluation.

Alternations and Applications of the Structural and Functional Connectome in Gliomas: A Mini-Review

In the central nervous system, gliomas are the most common, but complex primary tumors. Genome-based molecular and clinical studies have revealed different classifications and subtypes of gliomas. Neuroradiological approaches have non-invasively provided a macroscopic view for surgical resection and therapeutic effects. The connectome is a structural map of a physical object, the brain, which raises issues of spatial scale and definition, and it is calculated through diffusion magnetic resonance imaging (MRI) and functional MRI. In this study, we reviewed the basic principles and attributes of the structural and functional connectome, followed by the alternations of connectomes and their influences on glioma. To extend the applications of connectome, we demonstrated that a series of multi-center projects still need to be conducted to systemically investigate the connectome and the structural-functional coupling of glioma. Additionally, the brain-computer interface based on accurate connectome could provide more precise structural and functional data, which are significant for surgery and postoperative recovery. Besides, integrating the data from different sources, including connectome and other omics information, and their processing with artificial intelligence, together with validated biological and clinical findings will be significant for the development of a personalized surgical strategy.

Functional Remodeling Associated With Language Recovery After Repetitive Transcranial Magnetic Stimulation in Chronic Aphasic Stroke

Background

Repetitive transcranial magnetic stimulation (rTMS) has shown promising efficacy in improving the language functions in poststroke aphasia. However, randomized controlled trials were lacking to investigate the rTMS-related neuroimaging changes underlying the therapeutic effects on language improvement in chronic aphasia.

Objective

In this study, we aimed to evaluate the effects of low-frequency rTMS (LF-rTMS) on chronic poststroke aphasia. We hypothesized that the deactivation of the right pars triangularis could restore the balance of interhemispheric inhibition and, hence, facilitated the functional remodeling of language networks in both the hemispheres. Furthermore, the rTMS-induced functional reorganization should underpin the language recovery after rTMS.

Methods

A total of 33 patients (22 males; age: 58.70 ± 13.77 years) with chronic stroke in the left hemisphere and nonfluent aphasia were recruited in this randomized double-blinded study. The ratio of randomization between the rTMS and sham groups is 17:16. All the patients received real 1-Hz rTMS or sham stimulation (placebo coil delivered < 5% of magnetic output with similar audible click-on discharge) at the right posterior pars triangularis for 10 consecutive weekdays (stroke onset to the first stimulation: 10.97 ± 10.35 months). Functional connectivity of language networks measured by resting-state fMRI was calculated and correlated to the scores of the Concise Chinese Aphasia Test by using the stepwise regression analysis.

Results

After LF-rTMS intervention, significant improvement in language functions in terms of comprehension and expression abilities was observed compared with the sham group. The rTMS group showed a significant decrease of coupling strength between right pars triangularis and pars opercularis with a strengthened connection between right pars orbitalis and angular gyrus. Furthermore, the LF-rTMS significantly enhanced the coupling strength associated with left Wernicke area. Results of regression analysis showed that the identified functional remodeling involving both the hemispheres could support and predict the language recovery after LF-rTMS treatment.

Conclusion

We reported the therapeutic effects of LF-rTMS and corresponding functional remodeling in chronic poststroke aphasia. Our results provided neuroimage evidence reflecting the rebalance of interhemispheric inhibition induced by LF-rTMS, which could facilitate future research in the refinement of rTMS protocol to optimize the neuromodulation efficacy and benefit the clinical management of patients with stroke.

Controversy over the temporal cortical terminations of the left arcuate fasciculus: a reappraisal

The arcuate fasciculus has been considered a major dorsal fronto-temporal white matter pathway linking frontal language production regions with auditory perception in the superior temporal gyrus, the so-called Wernicke's area. In line with this tradition, both historical and contemporary models of language function have assigned primacy to superior temporal projections of the arcuate fasciculus. However, classical anatomical descriptions and emerging behavioural data are at odds with this assumption. On one hand, fronto-temporal projections to Wernicke's area may not be unique to the arcuate fasciculus. On the other hand, dorsal stream language deficits have been reported also for damage to middle, inferior and basal temporal gyri which may be linked to arcuate disconnection. These findings point to a reappraisal of arcuate projections in the temporal lobe. Here, we review anatomical and functional evidence regarding the temporal cortical terminations of the left arcuate fasciculus by incorporating dissection and tractography findings with stimulation data using cortico-cortical evoked potentials and direct electrical stimulation mapping in awake patients. Firstly, we discuss the fibers of the arcuate fasciculus projecting to the superior temporal gyrus and the functional rostro-caudal gradient in this region where both phonological encoding and auditory-motor transformation may be performed. Caudal regions within the temporoparietal junction may be involved in articulation and associated with temporoparietal projections of the third branch of the superior longitudinal fasciculus, while more rostral regions may support encoding of acoustic phonetic features, supported by arcuate fibres. We then move to examine clinical data showing that multimodal phonological encoding is facilitated by projections of the arcuate fasciculus to superior, but also middle, inferior and basal temporal regions. Hence, we discuss how projections of the arcuate fasciculus may contribute to acoustic (middle-posterior superior and middle temporal gyri), visual (posterior inferior temporal/fusiform gyri comprising the visual word form area) and lexical (anterior-middle inferior temporal/fusiform gyri in the basal temporal language area) information in the temporal lobe to be processed, encoded and translated into a dorsal phonological route to the frontal lobe. Finally, we point out surgical implications for this model in terms of the prediction and avoidance of neurological deficit.

Neural Measures of Pitch Processing in EEG Responses to Running Speech

Linearized encoding models are increasingly employed to model cortical responses to running speech. Recent extensions to subcortical responses suggest clinical perspectives, potentially complementing auditory brainstem responses (ABRs) or frequency-following responses (FFRs) that are current clinical standards. However, while it is well-known that the auditory brainstem responds both to transient amplitude variations and the stimulus periodicity that gives rise to pitch, these features co-vary in running speech. Here, we discuss challenges in disentangling the features that drive the subcortical response to running speech. Cortical and subcortical electroencephalographic (EEG) responses to running speech from 19 normal-hearing listeners (12 female) were analyzed. Using forward regression models, we confirm that responses to the rectified broadband speech signal yield temporal response functions consistent with wave V of the ABR, as shown in previous work. Peak latency and amplitude of the speech-evoked brainstem response were correlated with standard click-evoked ABRs recorded at the vertex electrode (Cz). Similar responses could be obtained using the fundamental frequency (F0) of the speech signal as model predictor. However, simulations indicated that dissociating responses to temporal fine structure at the F0 from broadband amplitude variations is not possible given the high co-variance of the features and the poor signal-to-noise ratio (SNR) of subcortical EEG responses. In cortex, both simulations and data replicated previous findings indicating that envelope tracking on frontal electrodes can be dissociated from responses to slow variations in F0 (relative pitch). Yet, no association between subcortical F0-tracking and cortical responses to relative pitch could be detected. These results indicate that while subcortical speech responses are comparable to click-evoked ABRs, dissociating pitch-related processing in the auditory brainstem may be challenging with natural speech stimuli.

Acoustic-level and language-specific processing of native and non-native phonological sequence onsets in the low gamma and theta-frequency bands

Acoustic structures associated with native-language phonological sequences are enhanced within auditory pathways for perception, although the underlying mechanisms are not well understood. To elucidate processes that facilitate perception, time-frequency (T-F) analyses of EEGs obtained from native speakers of English and Polish were conducted. Participants listened to same and different nonword pairs within counterbalanced attend and passive conditions. Nonwords contained the onsets /pt/, /pət/, /st/, and /sət/ that occur in both the Polish and English languages with the exception of /pt/, which never occurs in the English language in word onset. Measures of spectral power and inter-trial phase locking (ITPL) in the low gamma (LG) and theta-frequency bands were analyzed from two bilateral, auditory source-level channels, created through source localization modeling. Results revealed significantly larger spectral power in LG for the English listeners to the unfamiliar /pt/ onsets from the right hemisphere at early cortical stages, during the passive condition. Further, ITPL values revealed distinctive responses in high and low-theta to acoustic characteristics of the onsets, which were modulated by language exposure. These findings, language-specific processing in LG and acoustic-level and language-specific processing in theta, support the view that multi scale temporal processing in the LG and theta-frequency bands facilitates speech perception.

Speech Computations of the Human Superior Temporal Gyrus

Human speech perception results from neural computations that transform external acoustic speech signals into internal representations of words. The superior temporal gyrus (STG) contains the nonprimary auditory cortex and is a critical locus for phonological processing. Here, we describe how speech sound representation in the STG relies on fundamentally nonlinear and dynamical processes, such as categorization, normalization, contextual restoration, and the extraction of temporal structure. A spatial mosaic of local cortical sites on the STG exhibits complex auditory encoding for distinct acoustic-phonetic and prosodic features. We propose that as a population ensemble, these distributed patterns of neural activity give rise to abstract, higher-order phonemic and syllabic representations that support speech perception. This review presents a multi-scale, recurrent model of phonological processing in the STG, highlighting the critical interface between auditory and language systems.

Learning nonnative speech sounds changes local encoding in the adult human cortex

Adults can learn to identify nonnative speech sounds with training, albeit with substantial variability in learning behavior. Increases in behavioral accuracy are associated with increased separability for sound representations in cortical speech areas. However, it remains unclear whether individual auditory neural populations all show the same types of changes with learning, or whether there are heterogeneous encoding patterns. Here, we used high-resolution direct neural recordings to examine local population response patterns, while native English listeners learned to recognize unfamiliar vocal pitch patterns in Mandarin Chinese tones. We found a distributed set of neural populations in bilateral superior temporal gyrus and ventrolateral frontal cortex, where the encoding of Mandarin tones changed throughout training as a function of trial-by-trial accuracy ("learning effect"), including both increases and decreases in the separability of tones. These populations were distinct from populations that showed changes as a function of exposure to the stimuli regardless of trial-by-trial accuracy. These learning effects were driven in part by more variable neural responses to repeated presentations of acoustically identical stimuli. Finally, learning effects could be predicted from speech-evoked activity even before training, suggesting that intrinsic properties of these populations make them amenable to behavior-related changes. Together, these results demonstrate that nonnative speech sound learning involves a wide array of changes in neural representations across a distributed set of brain regions.