Dissociated roles of the inferior frontal gyrus and superior temporal sulcus in audiovisual processing: top-down and bottom-up mismatch detection

Naturalistic Audiovisual Illusions Reveal the Cortical Sites Involved in the Multisensory Processing of Speech

Audiovisual speech illusions are a spectacular illustration of the effect of visual cues on the perception of speech. Because they allow dissociating perception from the physical characteristics of the sensory inputs, these illusions are useful to investigate the cerebral processing of audiovisual speech. However, the meaningless, monosyllabic utterances typically used to induce illusions are far removed from natural communication through speech. We developed naturalistic speech stimuli that embed mismatched auditory and visual cues within grammatically correct sentences to induce illusory perceptions in controlled fashion. Using intracranial EEG, we confirmed that the cortical processing of audiovisual speech recruits an ensemble of areas, from auditory and visual cortices to multisensory and associative regions. Importantly, we were able to resolve which cortical areas are driven more by the auditory or the visual contents of the speech stimulus or by the eventual perceptual report. Our results suggest that higher order sensory and associative areas, rather than early sensory cortices, are key loci for illusory perception. Naturalistic audiovisual speech illusions represent a powerful tool to dissect the specific roles of individual cortical areas in the processing of audiovisual speech.

Mediterranean diet and brain functional connectivity in a population without dementia

Introduction

Adjustable lifestyle factors, such as diet, are associated with cognitive functions, structural and functional brain measures, but the association between the functional connectivity (FC) and the Mediterranean Diet (Medicine) in population without dementia is yet to be explored.

Methods

The association between MeDi and brain FC in 105 individuals without dementia aged 63 (SD ± 8.72) years old who underwent brain MRI including resting-state (rs) functional MRI (fMRI) was examined. Dietary intake was evaluated through four 24-h recalls using the multiple-pass method and adherence to the MeDi was estimated using the MedDietScore, with higher values indicating greater adherence to MeDi. Multivariable linear regression models were used to investigate the associations between FC (both positive and negative associations) and MedDietScore.

Results

Rs-fMRI analysis revealed significant associations between FC and MedDietScore. The FC between the medial prefrontal cortex and a cluster located in left postcentral gyrus and in the left supramarginal gyrus was positively associated with MedDietScore. On the other hand, the FC between medial visual and right posterior division of both middle and superior temporal gyrus was negatively associated with MedDietScore. Of note, a temporal negative correlation was detected between above-mentioned FC networks. The FC between superior temporal gyrus and occipital regions was associated with participants' attention, executive functions, and memory scores. Furthermore, the associations for attention and executive functions were pronounced in participants with high adherence to MeDi compared to those with low adherence to MeDi.

Discussion

In conclusion, our study documented an association between higher adherence to MeDi and rs-FC in fronto-parietal and temporo-occipital regions, particularly in areas that are involved in cognitive processes altered across normal and pathological aging. From a clinical point of view, our findings support a favorable role of MeDi on FC which may have significant clinical implications in the rapidly aging population. Rs-fMRI is also proposed as a useful tool in the emerging field of nutritional neuroscience and a candidate non-invasive biomarker of brain aging.

Multiple dimensions of syntactic structure are resolved earliest in posterior temporal cortex

How we combine minimal linguistic units into larger structures remains an unresolved topic in neuroscience. Language processing involves the abstract construction of 'vertical' and 'horizontal' information simultaneously (e.g., phrase structure, morphological agreement), but previous paradigms have been constrained in isolating only one type of composition and have utilized poor spatiotemporal resolution. Using intracranial recordings, we report multiple experiments designed to separate phrase structure from morphosyntactic agreement. Epilepsy patients (n = 10) were presented with auditory two-word phrases grouped into pseudoword-verb ('trab run') and pronoun-verb either with or without Person agreement ('they run' vs. 'they runs'). Phrase composition and Person violations both resulted in significant increases in broadband high gamma activity approximately 300 ms after verb onset in posterior middle temporal gyrus (pMTG) and posterior superior temporal sulcus (pSTS), followed by inferior frontal cortex (IFC) at 500 ms. While sites sensitive to only morphosyntactic violations were distributed, those sensitive to both composition types were generally confined to pSTS/pMTG and IFC. These results indicate that posterior temporal cortex shows the earliest sensitivity for hierarchical linguistic structure across multiple dimensions, providing neural resources for distinct windows of composition. This region is comprised of sparsely interwoven heterogeneous constituents that afford cortical search spaces for dissociable syntactic relations.

An Exploratory Analysis of the Neural Correlates of Human-Robot Interactions With Functional Near Infrared Spectroscopy

Functional near infrared spectroscopy (fNIRS) has been gaining increasing interest as a practical mobile functional brain imaging technology for understanding the neural correlates of social cognition and emotional processing in the human prefrontal cortex (PFC). Considering the cognitive complexity of human-robot interactions, the aim of this study was to explore the neural correlates of emotional processing of congruent and incongruent pairs of human and robot audio-visual stimuli in the human PFC with fNIRS methodology. Hemodynamic responses from the PFC region of 29 subjects were recorded with fNIRS during an experimental paradigm which consisted of auditory and visual presentation of human and robot stimuli. Distinct neural responses to human and robot stimuli were detected at the dorsolateral prefrontal cortex (DLPFC) and orbitofrontal cortex (OFC) regions. Presentation of robot voice elicited significantly less hemodynamic response than presentation of human voice in a left OFC channel. Meanwhile, processing of human faces elicited significantly higher hemodynamic activity when compared to processing of robot faces in two left DLPFC channels and a left OFC channel. Significant correlation between the hemodynamic and behavioral responses for the face-voice mismatch effect was found in the left OFC. Our results highlight the potential of fNIRS for unraveling the neural processing of human and robot audio-visual stimuli, which might enable optimization of social robot designs and contribute to elucidation of the neural processing of human and robot stimuli in the PFC in naturalistic conditions.

A studyforrest extension, MEG recordings while watching the audio-visual movie "Forrest Gump"

Naturalistic stimuli, such as movies, are being increasingly used to map brain function because of their high ecological validity. The pioneering studyforrest and other naturalistic neuroimaging projects have provided free access to multiple movie-watching functional magnetic resonance imaging (fMRI) datasets to prompt the community for naturalistic experimental paradigms. However, sluggish blood-oxygenation-level-dependent fMRI signals are incapable of resolving neuronal activity with the temporal resolution at which it unfolds. Instead, magnetoencephalography (MEG) measures changes in the magnetic field produced by neuronal activity and is able to capture rich dynamics of the brain at the millisecond level while watching naturalistic movies. Herein, we present the first public prolonged MEG dataset collected from 11 participants while watching the 2 h long audio-visual movie "Forrest Gump". Minimally preprocessed data was also provided to facilitate the use of the dataset. As a studyforrest extension, we envision that this dataset, together with fMRI data from the studyforrest project, will serve as a foundation for exploring the neural dynamics of various cognitive functions in real-world contexts.

Minimal Phrase Composition Revealed by Intracranial Recordings

The ability to comprehend phrases is an essential integrative property of the brain. Here, we evaluate the neural processes that enable the transition from single-word processing to a minimal compositional scheme. Previous research has reported conflicting timing effects of composition, and disagreement persists with respect to inferior frontal and posterior temporal contributions. To address these issues, 19 patients (10 male, 9 female) implanted with penetrating depth or surface subdural intracranial electrodes, heard auditory recordings of adjective-noun, pseudoword-noun, and adjective-pseudoword phrases and judged whether the phrase matched a picture. Stimulus-dependent alterations in broadband gamma activity, low-frequency power, and phase-locking values across the language-dominant left hemisphere were derived. This revealed a mosaic located on the lower bank of the posterior superior temporal sulcus (pSTS), in which closely neighboring cortical sites displayed exclusive sensitivity to either lexicality or phrase structure, but not both. Distinct timings were found for effects of phrase composition (210-300 ms) and pseudoword processing (∼300-700 ms), and these were localized to neighboring electrodes in pSTS. The pars triangularis and temporal pole encoded anticipation of composition in broadband low frequencies, and both regions exhibited greater functional connectivity with pSTS during phrase composition. Our results suggest that the pSTS is a highly specialized region composed of sparsely interwoven heterogeneous constituents that encodes both lower and higher level linguistic features. This hub in pSTS for minimal phrase processing may form the neural basis for the human-specific computational capacity for forming hierarchically organized linguistic structures.SIGNIFICANCE STATEMENT Linguists have claimed that the integration of multiple words into a phrase demands a computational procedure distinct from single-word processing. Here, we provide intracranial recordings from a large patient cohort, with high spatiotemporal resolution, to track the cortical dynamics of phrase composition. Epileptic patients volunteered to participate in a task in which they listened to phrases (red boat), word-pseudoword or pseudoword-word pairs (e.g., red fulg). At the onset of the second word in phrases, greater broadband high gamma activity was found in posterior superior temporal sulcus in electrodes that exclusively indexed phrasal meaning and not lexical meaning. These results provide direct, high-resolution signatures of minimal phrase composition in humans, a potentially species-specific computational capacity.

Reduced frontal white matter microstructure in healthy older adults with low tactile recognition performance

The aging of the nervous system is a heterogeneous process. It remains a significant challenge to identify relevant markers of pathological and healthy brain aging. A central aspect of aging are decreased sensory acuities, especially because they correlate with the decline in higher cognitive functioning. Sensory and higher cognitive processing relies on information flow between distant brain areas. Aging leads to disintegration of the underlying white matter tracts. While this disintegration is assumed to contribute to higher cognitive decline, data linking structural integrity and sensory function are sparse. The investigation of their interrelation may provide valuable insight into the mechanisms of brain aging. We used a combined behavioral and neuroimaging approach and investigated to what extent changes in microstructural white matter integrity reflect performance declines in tactile pattern recognition with aging. Poor performance in older participants was related to decreased integrity in the anterior corpus callosum. Probabilistic tractography showed that this structure is connected to the prefrontal cortices. Our data point to decreased integrity in the anterior corpus callosum as a marker for advanced brain aging. The correlation between impaired tactile recognition and disintegration in frontal brain networks could provide an explanation why the decrease of sensory function predicts cognitive decline.

Electrophysiology of the Human Superior Temporal Sulcus during Speech Processing

The superior temporal sulcus (STS) is a crucial hub for speech perception and can be studied with high spatiotemporal resolution using electrodes targeting mesial temporal structures in epilepsy patients. Goals of the current study were to clarify functional distinctions between the upper (STSU) and the lower (STSL) bank, hemispheric asymmetries, and activity during self-initiated speech. Electrophysiologic properties were characterized using semantic categorization and dialog-based tasks. Gamma-band activity and alpha-band suppression were used as complementary measures of STS activation. Gamma responses to auditory stimuli were weaker in STSL compared with STSU and had longer onset latencies. Activity in anterior STS was larger during speaking than listening; the opposite pattern was observed more posteriorly. Opposite hemispheric asymmetries were found for alpha suppression in STSU and STSL. Alpha suppression in the STS emerged earlier than in core auditory cortex, suggesting feedback signaling within the auditory cortical hierarchy. STSL was the only region where gamma responses to words presented in the semantic categorization tasks were larger in subjects with superior task performance. More pronounced alpha suppression was associated with better task performance in Heschl's gyrus, superior temporal gyrus, and STS. Functional differences between STSU and STSL warrant their separate assessment in future studies.

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

During natural speech perception, humans must parse temporally continuous auditory and visual speech signals into sequences of words. However, most studies of speech perception present only single words or syllables. We used electrocorticography (subdural electrodes implanted on the brains of epileptic patients) to investigate the neural mechanisms for processing continuous audiovisual speech signals consisting of individual sentences. Using partial correlation analysis, we found that posterior superior temporal gyrus (pSTG) and medial occipital cortex tracked both the auditory and the visual speech envelopes. These same regions, as well as inferior temporal cortex, responded more strongly to a dynamic video of a talking face compared to auditory speech paired with a static face. Occipital cortex and pSTG carry temporal information about both auditory and visual speech dynamics. Visual speech tracking in pSTG may be a mechanism for enhancing perception of degraded auditory speech.

Neural responses to action contingency error in different cortical areas are attributable to forward prediction or sensory processing

The contingency of sensory feedback to one's actions is essential for the sense of agency, and experimental violation of this contingency is a standard paradigm in the neuroscience of self-awareness and schizophrenia. However, neural responses to this violation have arbitrarily been interpreted either as activation of the system generating forward prediction (agency-error account) or decreased suppression of processing of predictable input (prediction-error account). In this functional magnetic resonance imaging (fMRI) study, the regions responsive to auditory contingency errors were examined if they exhibited responses to an isolated auditory stimulus and to passive-contingency delay, which the prediction-error account expects. These responses were observed only in the auditory association cortex in the right superior temporal gyrus. Several multimodal and motor-association cortices did not exhibit these responses, suggesting their relevance to the agency-error account. Thus, we formulated the coexistence and dissociation of two accounts in neural contingency-error responses.

The functional and structural asymmetries of the superior temporal sulcus

The superior temporal sulcus (STS) is an anatomical structure that increasingly interests researchers. This structure appears to receive multisensory input and is involved in several perceptual and cognitive core functions, such as speech perception, audiovisual integration, (biological) motion processing and theory of mind capacities. In addition, the superior temporal sulcus is not only one of the longest sulci of the brain, but it also shows marked functional and structural asymmetries, some of which have only been found in humans. To explore the functional-structural relationships of these asymmetries in more detail, this study combines functional and structural magnetic resonance imaging. Using a speech perception task, an audiovisual integration task, and a theory of mind task, this study again demonstrated an involvement of the STS in these processes, with an expected strong leftward asymmetry for the speech perception task. Furthermore, this study confirmed the earlier described, human-specific asymmetries, namely that the left STS is longer than the right STS and that the right STS is deeper than the left STS. However, this study did not find any relationship between these structural asymmetries and the detected brain activations or their functional asymmetries. This can, on the other hand, give further support to the notion that the structural asymmetry of the STS is not directly related to the functional asymmetry of the speech perception and the language system as a whole, but that it may have other causes and functions.

Coupling between Theta Oscillations and Cognitive Control Network during Cross-Modal Visual and Auditory Attention: Supramodal vs Modality-Specific Mechanisms

Cortical theta band oscillations (4–8 Hz) in EEG signals have been shown to be important for a variety of different cognitive control operations in visual attention paradigms. However the synchronization source of these signals as defined by fMRI BOLD activity and the extent to which theta oscillations play a role in multimodal attention remains unknown. Here we investigated the extent to which cross-modal visual and auditory attention impacts theta oscillations. Using a simultaneous EEG-fMRI paradigm, healthy human participants performed an attentional vigilance task with six cross-modal conditions using naturalistic stimuli. To assess supramodal mechanisms, modulation of theta oscillation amplitude for attention to either visual or auditory stimuli was correlated with BOLD activity by conjunction analysis. Negative correlation was localized to cortical regions associated with the default mode network and positively with ventral premotor areas. Modality-associated attention to visual stimuli was marked by a positive correlation of theta and BOLD activity in fronto-parietal area that was not observed in the auditory condition. A positive correlation of theta and BOLD activity was observed in auditory cortex, while a negative correlation of theta and BOLD activity was observed in visual cortex during auditory attention. The data support a supramodal interaction of theta activity with of DMN function, and modality-associated processes within fronto-parietal networks related to top-down theta related cognitive control in cross-modal visual attention. On the other hand, in sensory cortices there are opposing effects of theta activity during cross-modal auditory attention.

Altered regional homogeneity in patients with unilateral acute open-globe injury: a resting-state functional MRI study

OBJECTIVE

To investigate the underlying regional homogeneity (ReHo) brain activity changes in patients with unilateral acute open-globe injury (OGI) and their relationship with their clinical features.

PATIENTS AND METHODS

In total, 18 patients with acute OGI (16 males and two females) and 18 healthy controls (HCs; 16 males and two females) closely matched in age, sex, and education status participated in the study. Each subject underwent a resting-state functional magnetic resonance imaging scan. The ReHo method was used to assess local features of spontaneous brain activity. Receiver-operating characteristic curve was used to distinguish OGIs from HCs. The nonparametric statistical analysis was used to explore the relationship between the observed mean ReHo values of the different brain areas and the behavioral performance.

RESULTS

Compared with HCs, acute OGI patients had significantly increased ReHo values in the right cerebellum posterior lobe/lingual gyrus, left superior temporal gyrus/inferior frontal gyrus, left inferior frontal gyrus, left posterior cingulate cortex/precuneus, and left precentral operculum. However, there was no relationship between the observed mean ReHo values of the different brain areas and the behavioral performance.

CONCLUSION

Acute OGI may cause dysfunction in many brain regions, which may reflect the underlying pathologic mechanisms of acute vision loss in OGI patients.