The connectivity signature of co-speech gesture integration: The superior temporal sulcus modulates connectivity between areas related to visual gesture and auditory speech processing

fNIRS Changes in the Middle Temporal and Occipital Cortices After a Cochlear Implant

OBJECTIVES

The relationship between the middle temporal (MTG) and occipital cortices in post-lingually deaf (PLD) individuals is unclear. This study aimed to investigate changes in the MTG and occipital cortices excitability and their effects on the occipital cortex in individuals with PLD after receiving a cochlear implant (CI).

METHODS

Twenty-six individuals with severe-to-profound binaural sensorineural PLD were assessed clinically. Nine individuals had received a unilateral cochlear implant over 6 months, while 17 had not. Brodmann area 19 (BA19, extra-striate occipital cortex) and MTG (auditory-related area of cortex) were selected as regions of interest. The excitability of the ROI was observed and compared in the surgery and no-surgery groups by functional near-infrared spectroscopy (fNIRS) in the resting state, and correlations between connectivity of the MTG and occipital cortex, and as well as the duration of time that had elapsed following CI surgery, were investigated.

RESULTS

fNIRS revealed enhanced global cortical connectivity in the BA19 and MTG on the operative side (p < 0.05) and the connectivity between BA19 and the MTG also increased (p < 0.05). The connectivity between the MTG and BA19 was positively correlated with the duration of cochlear implantation, as was the case for BA18.

CONCLUSION

There was evidence for remodeling of the cerebral cortex: increased excitability was observed in the MTG and BA19, and their connectivity was enhanced, indicating a synergistic effect. Moreover, the MTG may further stimulate the visual cortex by strengthening their connectivity after CI.

LEVEL OF EVIDENCE

3 Laryngoscope, 135:331-338, 2025.

Dynamic causal modeling analysis reveals the modulation of motor cortex and integration in superior temporal gyrus during multisensory speech perception

The processing of speech information from various sensory modalities is crucial for human communication. Both left posterior superior temporal gyrus (pSTG) and motor cortex importantly involve in the multisensory speech perception. However, the dynamic integration of primary sensory regions to pSTG and the motor cortex remain unclear. Here, we implemented a behavioral experiment of classical McGurk effect paradigm and acquired the task functional magnetic resonance imaging (fMRI) data during synchronized audiovisual syllabic perception from 63 normal adults. We conducted dynamic causal modeling (DCM) analysis to explore the cross-modal interactions among the left pSTG, left precentral gyrus (PrG), left middle superior temporal gyrus (mSTG), and left fusiform gyrus (FuG). Bayesian model selection favored a winning model that included modulations of connections to PrG (mSTG → PrG, FuG → PrG), from PrG (PrG → mSTG, PrG → FuG), and to pSTG (mSTG → pSTG, FuG → pSTG). Moreover, the coupling strength of the above connections correlated with behavioral McGurk susceptibility. In addition, significant differences were found in the coupling strength of these connections between strong and weak McGurk perceivers. Strong perceivers modulated less inhibitory visual influence, allowed less excitatory auditory information flowing into PrG, but integrated more audiovisual information in pSTG. Taken together, our findings show that the PrG and pSTG interact dynamically with primary cortices during audiovisual speech, and support the motor cortex plays a specifically functional role in modulating the gain and salience between auditory and visual modalities.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-023-09945-z.

Language and gesture neural correlates: A meta-analysis of functional magnetic resonance imaging studies

BACKGROUND

Humans often use co-speech gestures to promote effective communication. Attention has been paid to the cortical areas engaged in the processing of co-speech gestures.

AIMS

To investigate the neural network underpinned in the processing of co-speech gestures and to observe whether there is a relationship between areas involved in language and gesture processing.

METHODS & PROCEDURES

We planned to include studies with neurotypical and/or stroke participants who underwent a bimodal task (i.e., processing of co-speech gestures with relative speech) and a unimodal task (i.e., speech or gesture alone) during a functional magnetic resonance imaging (fMRI) session. After a database search, abstract and full-text screening were conducted. Qualitative and quantitative data were extracted, and a meta-analysis was performed with the software GingerALE 3.0.2, performing contrast analyses of uni- and bimodal tasks.

MAIN CONTRIBUTION

The database search produced 1024 records. After the screening process, 27 studies were included in the review. Data from 15 studies were quantitatively analysed through meta-analysis. Meta-analysis found three clusters with a significant activation of the left middle frontal gyrus and inferior frontal gyrus, and bilateral middle occipital gyrus and inferior temporal gyrus.

CONCLUSIONS

There is a close link at the neural level for the semantic processing of auditory and visual information during communication. These findings encourage the integration of the use of co-speech gestures during aphasia treatment as a strategy to foster the possibility to communicate effectively for people with aphasia.

WHAT THIS PAPER ADDS

What is already known on this subject Gestures are an integral part of human communication, and they may have a relationship at neural level with speech processing. What this paper adds to the existing knowledge During processing of bi- and unimodal communication, areas related to semantic processing and multimodal processing are activated, suggesting that there is a close link between co-speech gestures and spoken language at a neural level. What are the potential or actual clinical implications of this work? Knowledge of the functions related to gesture and speech processing neural networks will allow for the adoption of model-based neurorehabilitation programs to foster recovery from aphasia by strengthening the specific functions of these brain networks.

Age-Related Changes to Multisensory Integration and Audiovisual Speech Perception

Multisensory integration is essential for the quick and accurate perception of our environment, particularly in everyday tasks like speech perception. Research has highlighted the importance of investigating bottom-up and top-down contributions to multisensory integration and how these change as a function of ageing. Specifically, perceptual factors like the temporal binding window and cognitive factors like attention and inhibition appear to be fundamental in the integration of visual and auditory information-integration that may become less efficient as we age. These factors have been linked to brain areas like the superior temporal sulcus, with neural oscillations in the alpha-band frequency also being implicated in multisensory processing. Age-related changes in multisensory integration may have significant consequences for the well-being of our increasingly ageing population, affecting their ability to communicate with others and safely move through their environment; it is crucial that the evidence surrounding this subject continues to be carefully investigated. This review will discuss research into age-related changes in the perceptual and cognitive mechanisms of multisensory integration and the impact that these changes have on speech perception and fall risk. The role of oscillatory alpha activity is of particular interest, as it may be key in the modulation of multisensory integration.

Integrative interaction of emotional speech in audio-visual modality

Emotional clues are always expressed in many ways in our daily life, and the emotional information we receive is often represented by multiple modalities. Successful social interactions require a combination of multisensory cues to accurately determine the emotion of others. The integration mechanism of multimodal emotional information has been widely investigated. Different brain activity measurement methods were used to determine the location of brain regions involved in the audio-visual integration of emotional information, mainly in the bilateral superior temporal regions. However, the methods adopted in these studies are relatively simple, and the materials of the study rarely contain speech information. The integration mechanism of emotional speech in the human brain still needs further examinations. In this paper, a functional magnetic resonance imaging (fMRI) study was conducted using event-related design to explore the audio-visual integration mechanism of emotional speech in the human brain by using dynamic facial expressions and emotional speech to express emotions of different valences. Representational similarity analysis (RSA) based on regions of interest (ROIs), whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis were used to analyze and verify the role of relevant brain regions. Meanwhile, a weighted RSA method was used to evaluate the contributions of each candidate model in the best fitted model of ROIs. The results showed that only the left insula was detected by all methods, suggesting that the left insula played an important role in the audio-visual integration of emotional speech. Whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis together revealed that the bilateral middle temporal gyrus (MTG), right inferior parietal lobule and bilateral precuneus might be involved in the audio-visual integration of emotional speech from other aspects.

The Processing of Semantic Complexity and Cospeech Gestures in Schizophrenia: A Naturalistic, Multimodal fMRI Study

Schizophrenia is marked by aberrant processing of complex speech and gesture, which may contribute functionally to its impaired social communication. To date, extant neuroscientific studies of schizophrenia have largely investigated dysfunctional speech and gesture in isolation, and no prior research has examined how the two communicative channels may interact in more natural contexts. Here, we tested if patients with schizophrenia show aberrant neural processing of semantically complex story segments, and if speech-associated gestures (co-speech gestures) might modulate this effect. In a functional MRI study, we presented to 34 participants (16 patients and 18 matched-controls) an ecologically-valid retelling of a continuous story, performed via speech and spontaneous gestures. We split the entire story into ten-word segments, and measured the semantic complexity for each segment with idea density, a linguistic measure that is commonly used clinically to evaluate aberrant language dysfunction at the semantic level. Per segment, the presence of numbers of gestures varied (n = 0, 1, +2). Our results suggest that, in comparison to controls, patients showed reduced activation for more complex segments in the bilateral middle frontal and inferior parietal regions. Importantly, this neural aberrance was normalized in segments presented with gestures. Thus, for the first time with a naturalistic multimodal stimulation paradigm, we show that gestures reduced group differences when processing a natural story, probably by facilitating the processing of semantically complex segments of the story in schizophrenia.

Dynamic Causal Modeling on the Identification of Interacting Networks in the Brain: A Systematic Review

Dynamic causal modeling (DCM) has long been used to characterize effective connectivity within networks of distributed neuronal responses. Previous reviews have highlighted the understanding of the conceptual basis behind DCM and its variants from different aspects. However, no detailed summary or classification research on the task-related effective connectivity of various brain regions has been made formally available so far, and there is also a lack of application analysis of DCM for hemodynamic and electrophysiological measurements. This review aims to analyze the effective connectivity of different brain regions using DCM for different measurement data. We found that, in general, most studies focused on the networks between different cortical regions, and the research on the networks between other deep subcortical nuclei or between them and the cerebral cortex are receiving increasing attention, but far from the same scale. Our analysis also reveals a clear bias towards some task types. Based on these results, we identify and discuss several promising research directions that may help the community to attain a clear understanding of the brain network interactions under different tasks.

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process.

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Schizophrenia is characterized by marked communication dysfunctions encompassing potential impairments in the processing of social-abstract and non-social-concrete information, especially in everyday situations where multiple modalities are present in the form of speech and gesture. To date, the neurobiological basis of these deficits remains elusive. In a functional magnetic resonance imaging (fMRI) study, 17 patients with schizophrenia or schizoaffective disorder, and 18 matched controls watched videos of an actor speaking, gesturing (unimodal), and both speaking and gesturing (bimodal) about social or non-social events in a naturalistic way. Participants were asked to judge whether each video contains person-related (social) or object-related (non-social) information. When processing social-abstract content, patients showed reduced activation in the medial prefrontal cortex (mPFC) only in the gesture but not in the speech condition. For non-social-concrete content, remarkably, reduced neural activation for patients in the left postcentral gyrus and the right insula was observed only in the speech condition. Moreover, in the bimodal conditions, patients displayed improved task performance and comparable activation to controls in both social and non-social content. To conclude, patients with schizophrenia displayed modality-specific aberrant neural processing of social and non-social information, which is not present for the bimodal conditions. This finding provides novel insights into dysfunctional multimodal communication in schizophrenia, and may have potential therapeutic implications.

Changes in Electroencephalogram Coherence in Adolescents With Autism Spectrum Disorder After a Social Skills Intervention

Autism spectrum disorder (ASD) is a developmental condition that affects social communication and behavior. There is consensus that neurological differences are present in ASD. Further, theories emphasize the mixture of hypo- and hyper-connectivity as a neuropathologies in ASD [O'Reilly, Lewis, & Elsabbagh, 2017]; however, there is a paucity of studies specifically testing neurological underpinnings as predictors of success on social skills interventions. This study examined functional neural connectivity (electroencephalogram [EEG], coherence) of adolescents with ASD before and after the Program for the Education and Enrichment of Relational Skills (PEERS®) intervention, using a randomized controlled trial of two groups: an Experimental ASD (EXP) Group and a Waitlist Control ASD (WL) Group. The study had two purposes. First, the study aimed to determine whether changes in EEG coherence differed for adolescents that received PEERS® versus those that did not receive PEERS®. Results revealed a significant increase in connectivity in the occipital left to temporal left pair for the EXP group after intervention. Second, the study aimed to determine if changes in EEG coherence related to changes in behavior, friendships, and social skills measured by questionnaires. At post-intervention, results indicated: (a) positive change in frontal right to parietal right coherence was linked to an increase in social skills scores; and (b) positive changes in occipital right to temporal right coherence and occipital left to parietal left coherence were linked to an increase in the total number of get-togethers. Results of this study support utilizing neurobehavioral domains as indicators of treatment outcome. Lay Summary: This study examined how well various areas of the brain communicate in adolescents with ASD before and after a social skills intervention. Results revealed increased connectivity in the adolescents that received the intervention. Secondly, the study aimed to determine if changes in connectivity of brain areas related to changes in behavior, friendships, and social skills. Results indicated that changes in connectivity were also linked to increased social skills. Autism Res 2021, 14: 787-803. © 2021 International Society for Autism Research and Wiley Periodicals LLC.

Gesture, communication, and adult acquired hearing loss

Nonverbal communication, specifically hand and arm movements (commonly known as gesture), has long been recognized and explored as a significant element in human interaction as well as potential compensatory behavior for individuals with communication difficulties. The use of gesture as a compensatory communication method in expressive and receptive human communication disorders has been the subject of much investigation. Yet within the context of adult acquired hearing loss, gesture has received limited research attention and much remains unknown about patterns of nonverbal behaviors in conversations in which hearing loss is a factor. This paper presents key elements of the background of gesture studies and the theories of gesture function and production followed by a review of research focused on adults with hearing loss and the role of gesture and gaze in rehabilitation. The current examination of the visual resource of co-speech gesture in the context of everyday interactions involving adults with acquired hearing loss suggests the need for the development of an evidence base to effect enhancements and changes in the way in which rehabilitation services are conducted.

Dysbalanced Resting-State Functional Connectivity Within the Praxis Network Is Linked to Gesture Deficits in Schizophrenia

Patients with schizophrenia frequently present deficits in gesture production and interpretation, greatly affecting their communication skills. As these gesture deficits can be found early in the course of illness and as they can predict later outcomes, exploring their neural basis may lead to a better understanding of schizophrenia. While gesturing has been reported to rely on a left lateralized network of brain regions, termed praxis network, in healthy subjects and lesioned patients, studies in patients with schizophrenia are sparse. It is currently unclear whether within-network connectivity at rest is linked to gesture deficit. Here, we compared the functional connectivity between regions of the praxis network at rest between 46 patients and 44 healthy controls. All participants completed a validated test of hand gesture performance before resting-state functional magnetic resonance imaging (fMRI) was acquired. Patients performed gestures poorer than controls in all categories and domains. In patients, we also found significantly higher resting-state functional connectivity between left precentral gyrus and bilateral superior and inferior parietal lobule. Likewise, patients had higher connectivity from right precentral gyrus to left inferior and bilateral superior parietal lobule (SPL). In contrast, they exhibited lower connectivity between bilateral superior temporal gyrus (STG). Connectivity between right precentral gyrus and left SPL, as well as connectivity between bilateral STG, correlated with gesture performance in healthy controls. We failed to detect similar correlations in patients. We suggest that altered resting-state functional connectivity within the praxis network perturbs correct gesture planning in patients, reflecting the gesture deficit often seen in schizophrenia.

Timing of phase-amplitude coupling is essential for neuronal and functional maturation of audiovisual integration in adolescents

OBJECTIVE

The ability to integrate audiovisual information matures late in adolescents, but its neuronal mechanism is still unknown. Recent studies showed that phase-amplitude coupling (PAC) of neuronal oscillations, which is defined as the modulation of high-frequency amplitude by low-frequency phase, is associated with audiovisual integration in adults. Thus, we investigated how PAC develops in adolescents and whether it is related to the functional maturation of audiovisual integration. In particular, we focused on the timing of PAC (or the coupling phase), which is defined as the low-frequency phase with maximum high-frequency amplitude.

METHODS

Using magnetoencephalography (MEG) on 15 adults and 14 adolescents while they performed an audiovisual speech integration task, we examined PAC in association cortexes with a trial-by-trial analysis.

RESULTS

Whereas delta-beta coupling was consistently observed in both adults and adolescents, we found that the timing of delta-beta PAC was delayed by 20-40 milliseconds in adolescents compared with adults. In addition, a logistic regression analysis revealed that the task performance improves as the timing of delta-beta PAC in the right temporal pole (TP) got closer to the trough position (180 degrees).

CONCLUSION

These results suggest that the timing of PAC is essential for binding audiovisual information and underlies the developmental process in adolescents.

Dynamic Causal Modelling suggests impaired effective connectivity in patients with schizophrenia spectrum disorders during gesture-speech integration

Integrating visual and auditory information during gesture-speech integration (GSI) is important for successful social communication, which is often impaired in schizophrenia. Several studies suggested the posterior superior temporal sulcus (pSTS) to be a relevant multisensory integration site. However, intact STS activation patterns were often reported in patients. Thus, here we used Dynamic Causal Modelling (DCM) to analyze whether information processing in schizophrenia spectrum disorders (SSD) is impaired during GSI on network level. We investigated GSI in three different samples. First, we replicated a recently published connectivity model for GSI in a healthy subject group (n = 19). Second, we investigated differences between patients with SSD and a matched healthy control group (n = 17 each). Participants were presented videos of an actor performing intrinsically meaningful gestures accompanied by spoken sentences in German or Russian, or just telling a German sentence without gestures. Across all groups, fMRI analyses revealed similar activation patterns, and DCM analyses resulted in the same winning model for GSI. This finding directly replicates previous results. However, patients revealed significantly reduced connectivity in the verbal pathway (from left middle temporal gyrus (MTG) to left STS). The clinical significance of this connection is supported by its correlations with the severity of concretism and a subscale of negative symptoms (SANS). Our model confirms the importance of the pSTS as integration site during audio-visual integration. Patients showed generally intact connectivity during GSI, but revealed impaired information transfer via the verbal pathway. This might be the basis of interpersonal communication problems in patients with SSD.