Neural responses in human superior temporal cortex support coding of voice representations

Cortical processing of discrete prosodic patterns in continuous speech

Prosody has a vital function in speech, structuring a speaker's intended message for the listener. The superior temporal gyrus (STG) is considered a critical hub for prosody, but the role of earlier auditory regions like Heschl's gyrus (HG), associated with pitch processing, remains unclear. Using intracerebral recordings in humans and non-human primate models, we investigated prosody processing in narrative speech, focusing on pitch accents-abstract phonological units that signal word prominence and communicative intent. In humans, HG encoded pitch accents as abstract representations beyond spectrotemporal features, distinct from segmental speech processing, and outperforms STG in disambiguating pitch accents. Multivariate models confirm HG's unique representation of pitch accent categories. In the non-human primate, pitch accents were not abstractly encoded, despite robust spectrotemporal processing, highlighting the role of experience in shaping abstract representations. These findings emphasize a key role for the HG in early prosodic abstraction and advance our understanding of human speech processing.

Networks extracted from nonlinear fMRI connectivity exhibit unique spatial variation and enhanced sensitivity to differences between individuals with schizophrenia and controls

Schizophrenia is a chronic brain disorder associated with widespread alterations in functional brain connectivity. Although data-driven approaches such as independent component analysis are often used to study how schizophrenia impacts linearly connected networks, alterations within the underlying nonlinear functional connectivity structure remain largely unknown. Here we report the analysis of networks from explicitly nonlinear functional magnetic resonance imaging connectivity in a case-control dataset. We found systematic spatial variation, with higher nonlinear weight within core regions, suggesting that linear analyses underestimate functional connectivity within network centers. We also found that a unique nonlinear network incorporating default-mode, cingulo-opercular and central executive regions exhibits hypoconnectivity in schizophrenia, indicating that typically hidden connectivity patterns may reflect inefficient network integration in psychosis. Moreover, nonlinear networks including those previously implicated in auditory, linguistic and self-referential cognition exhibit heightened statistical sensitivity to schizophrenia diagnosis, collectively underscoring the potential of our methodology to resolve complex brain phenomena and transform clinical connectivity analysis.

Functional connectivity across the human subcortical auditory system using an autoregressive matrix-Gaussian copula graphical model approach with partial correlations

The auditory system comprises multiple subcortical brain structures that process and refine incoming acoustic signals along the primary auditory pathway. Due to technical limitations of imaging small structures deep inside the brain, most of our knowledge of the subcortical auditory system is based on research in animal models using invasive methodologies. Advances in ultrahigh-field functional magnetic resonance imaging (fMRI) acquisition have enabled novel noninvasive investigations of the human auditory subcortex, including fundamental features of auditory representation such as tonotopy and periodotopy. However, functional connectivity across subcortical networks is still underexplored in humans, with ongoing development of related methods. Traditionally, functional connectivity is estimated from fMRI data with full correlation matrices. However, partial correlations reveal the relationship between two regions after removing the effects of all other regions, reflecting more direct connectivity. Partial correlation analysis is particularly promising in the ascending auditory system, where sensory information is passed in an obligatory manner, from nucleus to nucleus up the primary auditory pathway, providing redundant but also increasingly abstract representations of auditory stimuli. While most existing methods for learning conditional dependency structures based on partial correlations assume independently and identically Gaussian distributed data, fMRI data exhibit significant deviations from Gaussianity as well as high-temporal autocorrelation. In this paper, we developed an autoregressive matrix-Gaussian copula graphical model (ARMGCGM) approach to estimate the partial correlations and thereby infer the functional connectivity patterns within the auditory system while appropriately accounting for autocorrelations between successive fMRI scans. Our results show strong positive partial correlations between successive structures in the primary auditory pathway on each side (left and right), including between auditory midbrain and thalamus, and between primary and associative auditory cortex. These results are highly stable when splitting the data in halves according to the acquisition schemes and computing partial correlations separately for each half of the data, as well as across cross-validation folds. In contrast, full correlation-based analysis identified a rich network of interconnectivity that was not specific to adjacent nodes along the pathway. Overall, our results demonstrate that unique functional connectivity patterns along the auditory pathway are recoverable using novel connectivity approaches and that our connectivity methods are reliable across multiple acquisitions.

Cortical-striatal brain network distinguishes deepfake from real speaker identity

Deepfakes are viral ingredients of digital environments, and they can trick human cognition into misperceiving the fake as real. Here, we test the neurocognitive sensitivity of 25 participants to accept or reject person identities as recreated in audio deepfakes. We generate high-quality voice identity clones from natural speakers by using advanced deepfake technologies. During an identity matching task, participants show intermediate performance with deepfake voices, indicating levels of deception and resistance to deepfake identity spoofing. On the brain level, univariate and multivariate analyses consistently reveal a central cortico-striatal network that decoded the vocal acoustic pattern and deepfake-level (auditory cortex), as well as natural speaker identities (nucleus accumbens), which are valued for their social relevance. This network is embedded in a broader neural identity and object recognition network. Humans can thus be partly tricked by deepfakes, but the neurocognitive mechanisms identified during deepfake processing open windows for strengthening human resilience to fake information.

A hierarchy of processing complexity and timescales for natural sounds in human auditory cortex

Efficient behavior is supported by humans' ability to rapidly recognize acoustically distinct sounds as members of a common category. Within auditory cortex, there are critical unanswered questions regarding the organization and dynamics of sound categorization. Here, we performed intracerebral recordings in the context of epilepsy surgery as 20 patient-participants listened to natural sounds. We built encoding models to predict neural responses using features of these sounds extracted from different layers within a sound-categorization deep neural network (DNN). This approach yielded highly accurate models of neural responses throughout auditory cortex. The complexity of a cortical site's representation (measured by the depth of the DNN layer that produced the best model) was closely related to its anatomical location, with shallow, middle, and deep layers of the DNN associated with core (primary auditory cortex), lateral belt, and parabelt regions, respectively. Smoothly varying gradients of representational complexity also existed within these regions, with complexity increasing along a posteromedial-to-anterolateral direction in core and lateral belt, and along posterior-to-anterior and dorsal-to-ventral dimensions in parabelt. When we estimated the time window over which each recording site integrates information, we found shorter integration windows in core relative to lateral belt and parabelt. Lastly, we found a relationship between the length of the integration window and the complexity of information processing within core (but not lateral belt or parabelt). These findings suggest hierarchies of timescales and processing complexity, and their interrelationship, represent a functional organizational principle of the auditory stream that underlies our perception of complex, abstract auditory information.

rTMS of the auditory association cortex improves speech intelligibility in patients with sensorineural hearing loss

OBJECTIVE

Sensorineural hearing-loss (SHL) is accompanied by changes in the entire ear-brain pathway and its connected regions. While hearing-aid (HA) partially compensates for SHL, speech perception abilities often continue to remain poor, resulting in consequences in everyday activities. Repetitive transcranial magnetic stimulation (rTMS) promotes cortical network plasticity and may enhance language comprehension in SHL patients.

METHODS

27 patients using HA and with SHL were randomly assigned to a treatment protocol consisting of five consecutive days of either real (Active group: 13 patients) or placebo rTMS (Sham group: 14 patients). The stimulation parameters were as follows: 2-second trains at 10 Hz, 4-second inter-train-interval, and 1800 pulses. Neuronavigated rTMS was applied over the left superior temporal sulcus. Audiological tests were administered before (T0), immediately after (T1), and one week following treatment completion (T2) to evaluate the speech reception threshold (SRT) and the Pure Tone Average (PTA).

RESULTS

In the context of a general improvement likely due to learning, the treatment with real rTMS induced significant reduction of the SRT and PTA at T1 and T2 versus placebo.

CONCLUSIONS

The long-lasting effects on SRT and PTA observed in the Active group indicates that rTMS administered over the auditory cortex could promote sustained neuromodulatory-induced changes in the brain, improving the perception of complex sentences and pure tones reception skills.

SIGNIFICANCE

Five days of rTMS treatment enhances overall speech intelligibility and PTA in SHL patients.

Unveiling the development of human voice perception: Neurobiological mechanisms and pathophysiology

The human voice is a critical stimulus for the auditory system that promotes social connection, informs the listener about identity and emotion, and acts as the carrier for spoken language. Research on voice processing in adults has informed our understanding of the unique status of the human voice in the mature auditory cortex and provided potential explanations for mechanisms that underly voice selectivity and identity processing. There is evidence that voice perception undergoes developmental change starting in infancy and extending through early adolescence. While even young infants recognize the voice of their mother, there is an apparent protracted course of development to reach adult-like selectivity for human voice over other sound categories and recognition of other talkers by voice. Gaps in the literature do not allow for an exact mapping of this trajectory or an adequate description of how voice processing and its neural underpinnings abilities evolve. This review provides a comprehensive account of developmental voice processing research published to date and discusses how this evidence fits with and contributes to current theoretical models proposed in the adult literature. We discuss how factors such as cognitive development, neural plasticity, perceptual narrowing, and language acquisition may contribute to the development of voice processing and its investigation in children. We also review evidence of voice processing abilities in premature birth, autism spectrum disorder, and phonagnosia to examine where and how deviations from the typical trajectory of development may manifest.

Networks extracted from nonlinear fMRI connectivity exhibit unique spatial variation and enhanced sensitivity to differences between individuals with schizophrenia and controls

Functional magnetic resonance imaging (fMRI) studies often estimate brain intrinsic connectivity networks (ICNs) from temporal relationships between hemodynamic signals using approaches such as independent component analysis (ICA). While ICNs are thought to represent functional sources that play important roles in various psychological phenomena, current approaches have been tailored to identify ICNs that mainly reflect linear statistical relationships. However, the elements comprising neural systems often exhibit remarkably complex nonlinear interactions that may be involved in cognitive operations and altered in psychiatric conditions such as schizophrenia. Consequently, there is a need to develop methods capable of effectively capturing ICNs from measures that are sensitive to nonlinear relationships. Here, we advance a novel approach to estimate ICNs from explicitly nonlinear whole-brain functional connectivity (ENL-wFC) by transforming resting-state fMRI (rsfMRI) data into the connectivity domain, allowing us to capture unique information from distance correlation patterns that would be missed by linear whole-brain functional connectivity (LIN-wFC) analysis. Our findings provide evidence that ICNs commonly extracted from linear (LIN) relationships are also reflected in explicitly nonlinear (ENL) connectivity patterns. ENL ICN estimates exhibit higher reliability and stability, highlighting our approach's ability to effectively quantify ICNs from rsfMRI data. Additionally, we observed a consistent spatial gradient pattern between LIN and ENL ICNs with higher ENL weight in core ICN regions, suggesting that ICN function may be subserved by nonlinear processes concentrated within network centers. We also found that a uniquely identified ENL ICN distinguished individuals with schizophrenia from healthy controls while a uniquely identified LIN ICN did not, emphasizing the valuable complementary information that can be gained by incorporating measures that are sensitive to nonlinearity in future analyses. Moreover, the ENL estimates of ICNs associated with auditory, linguistic, sensorimotor, and self-referential processes exhibit heightened sensitivity towards differentiating between individuals with schizophrenia and controls compared to LIN counterparts, demonstrating the translational value of our approach and of the ENL estimates of ICNs that are frequently reported as disrupted in schizophrenia. In summary, our findings underscore the tremendous potential of connectivity domain ICA and nonlinear information in resolving complex brain phenomena and revolutionizing the landscape of clinical FC analysis.

The effects of attention in auditory-visual integration revealed by time-varying networks

Attention and audiovisual integration are crucial subjects in the field of brain information processing. A large number of previous studies have sought to determine the relationship between them through specific experiments, but failed to reach a unified conclusion. The reported studies explored the relationship through the frameworks of early, late, and parallel integration, though network analysis has been employed sparingly. In this study, we employed time-varying network analysis, which offers a comprehensive and dynamic insight into cognitive processing, to explore the relationship between attention and auditory-visual integration. The combination of high spatial resolution functional magnetic resonance imaging (fMRI) and high temporal resolution electroencephalography (EEG) was used. Firstly, a generalized linear model (GLM) was employed to find the task-related fMRI activations, which was selected as regions of interesting (ROIs) for nodes of time-varying network. Then the electrical activity of the auditory-visual cortex was estimated via the normalized minimum norm estimation (MNE) source localization method. Finally, the time-varying network was constructed using the adaptive directed transfer function (ADTF) technology. Notably, Task-related fMRI activations were mainly observed in the bilateral temporoparietal junction (TPJ), superior temporal gyrus (STG), primary visual and auditory areas. And the time-varying network analysis revealed that V1/A1↔STG occurred before TPJ↔STG. Therefore, the results supported the theory that auditory-visual integration occurred before attention, aligning with the early integration framework.

Electrocorticography reveals the dynamics of famous voice responses in human fusiform gyrus

Voice and face processing occur through convergent neural systems that facilitate speaker recognition. Neuroimaging studies suggest that familiar voice processing engages early visual cortex, including the bilateral fusiform gyrus (FG) on the basal temporal lobe. However, what role the FG plays in voice processing and whether it is driven by bottom-up or top-down mechanisms is unresolved. In this study we directly examined neural responses to famous voices and faces in human FG with direct cortical surface recordings (electrocorticography) in epilepsy surgery patients. We tested the hypothesis that neural populations in human FG respond to famous voices and investigated the temporal properties of voice responses in FG. Recordings were acquired from five adult participants during a person identification task using visual and auditory stimuli from famous speakers (U.S. Presidents Barack Obama, George W. Bush, and Bill Clinton). Patients were presented with images of presidents or clips of their voices and asked to identify the portrait/speaker. Our results demonstrate that a subset of face-responsive sites in and near FG also exhibit voice responses that are both lower in magnitude and delayed (300-600 ms) compared with visual responses. The dynamics of voice processing revealed by direct cortical recordings suggests a top-down feedback-mediated response to famous voices in FG that may facilitate speaker identification.NEW & NOTEWORTHY Interactions between auditory and visual cortices play an important role in person identification, but the dynamics of these interactions remain poorly understood. We performed direct brain recordings of fusiform face cortex in human epilepsy patients performing a famous voice naming task, revealing the dynamics of famous voice processing in human fusiform face cortex. The findings support a model of top-down interactions from auditory to visual cortex to facilitate famous voice recognition.

The Temporal Voice Areas are not "just" Speech Areas

The Temporal Voice Areas (TVAs) respond more strongly to speech sounds than to non-speech vocal sounds, but does this make them Temporal "Speech" Areas? We provide a perspective on this issue by combining univariate, multivariate, and representational similarity analyses of fMRI activations to a balanced set of speech and non-speech vocal sounds. We find that while speech sounds activate the TVAs more than non-speech vocal sounds, which is likely related to their larger temporal modulations in syllabic rate, they do not appear to activate additional areas nor are they segregated from the non-speech vocal sounds when their higher activation is controlled. It seems safe, then, to continue calling these regions the Temporal Voice Areas.

The path of voices in our brain

Categorising voices is crucial for auditory-based social interactions. This Primer explores a PLOS Biiology study that capitalises on human intracranial recordings to describe the spatiotemporal pattern of neural activity leading to voice-selective responses in associative auditory cortex.