Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions

Abnormal connectivity and activation during audiovisual speech perception in schizophrenia

The unconscious integration of vocal and facial cues during speech perception facilitates face-to-face communication. Recent studies have provided substantial behavioural evidence concerning impairments in audiovisual (AV) speech perception in schizophrenia. However, the specific neurophysiological mechanism underlying these deficits remains unknown. Here, we investigated activities and connectivities centered on the auditory cortex during AV speech perception in schizophrenia. Using magnetoencephalography, we recorded and analysed event-related fields in response to auditory (A: voice), visual (V: face) and AV (voice-face) stimuli in 23 schizophrenia patients (13 males) and 22 healthy controls (13 males). The functional connectivity associated with the subadditive response to AV stimulus (i.e., [AV] < [A] + [V]) was also compared between the two groups. Within the healthy control group, [AV] activity was smaller than the sum of [A] and [V] at latencies of approximately 100 ms in the posterior ramus of the lateral sulcus in only the left hemisphere, demonstrating a subadditive N1m effect. Conversely, the schizophrenia group did not show such a subadditive response. Furthermore, weaker functional connectivity from the posterior ramus of the lateral sulcus of the left hemisphere to the fusiform gyrus of the right hemisphere was observed in schizophrenia. Notably, this weakened connectivity was associated with the severity of negative symptoms. These results demonstrate abnormalities in connectivity between speech- and face-related cortical areas in schizophrenia. This aberrant subadditive response and connectivity deficits for integrating speech and facial information may be the neural basis of social communication dysfunctions in schizophrenia.

White matter tracts differentially associated with auditory hallucinations in first-episode psychosis: A correlational tractography diffusion spectrum imaging study

Auditory hallucinations (AH) are a debilitating symptom in psychosis, impacting cognition and real world functioning. Recent thought conceptualizes AH as a consequence of long-range brain communication dysfunction, or circuitopathy, within the auditory sensory/perceptual, language, and cognitive control systems. Recently we showed in first-episode psychosis (FEP) that, despite overall intact white matter integrity in the cortical-cortical and cortical-subcortical language tracts and the callosal tracts connecting auditory cortices, the severity of AH correlated inversely with white matter integrity. However, that hypothesis-driven isolation of specific tracts likely missed important white matter concomitants of AH. In this report, we used a whole-brain data-driven dimensional approach using correlational tractography to associate AH severity with white matter integrity in a sample of 175 individuals. Diffusion Spectrum Imaging (DSI) was used to image diffusion distribution. Quantitative Anisotropy (QA) in three tracts was greater with increased AH severity (FDR < 0.001) and QA in three tracts was lower with increased AH severity (FDR < 0.01). White matter tracts showing associations between QA and AH were generally associated with frontal-parietal-temporal connectivity (tracts with known relevance for cognitive control and the language system), in the cingulum bundle, and in prefrontal inter-hemispheric connectivity. The results of this whole brain data-driven analysis suggest that subtle white matter alterations connecting frontal, parietal, and temporal lobes in the service of sensory-perceptual, language/semantic, and cognitive control processes impact the expression of auditory hallucination in FEP. Disentangling the distributed neural circuits involved in AH should help to develop novel interventions, such as non-invasive brain stimulation.

Neural correlates of audiovisual narrative speech perception in children and adults on the autism spectrum: A functional magnetic resonance imaging study

Autistic individuals show substantially reduced benefit from observing visual articulations during audiovisual speech perception, a multisensory integration deficit that is particularly relevant to social communication. This has mostly been studied using simple syllabic or word-level stimuli and it remains unclear how altered lower-level multisensory integration translates to the processing of more complex natural multisensory stimulus environments in autism. Here, functional neuroimaging was used to examine neural correlates of audiovisual gain (AV-gain) in 41 autistic individuals to those of 41 age-matched non-autistic controls when presented with a complex audiovisual narrative. Participants were presented with continuous narration of a story in auditory-alone, visual-alone, and both synchronous and asynchronous audiovisual speech conditions. We hypothesized that previously identified differences in audiovisual speech processing in autism would be characterized by activation differences in brain regions well known to be associated with audiovisual enhancement in neurotypicals. However, our results did not provide evidence for altered processing of auditory alone, visual alone, audiovisual conditions or AV- gain in regions associated with the respective task when comparing activation patterns between groups. Instead, we found that autistic individuals responded with higher activations in mostly frontal regions where the activation to the experimental conditions was below baseline (de-activations) in the control group. These frontal effects were observed in both unisensory and audiovisual conditions, suggesting that these altered activations were not specific to multisensory processing but reflective of more general mechanisms such as an altered disengagement of Default Mode Network processes during the observation of the language stimulus across conditions.

The role of multimodal cues in second language comprehension

In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues - meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal - while using multimodal cues to a lesser extent than L1 comprehenders overall.

Evidence for a Causal Dissociation of the McGurk Effect and Congruent Audiovisual Speech Perception via TMS

Congruent visual speech improves speech perception accuracy, particularly in noisy environments. Conversely, mismatched visual speech can alter what is heard, leading to an illusory percept known as the McGurk effect. This illusion has been widely used to study audiovisual speech integration, illustrating that auditory and visual cues are combined in the brain to generate a single coherent percept. While prior transcranial magnetic stimulation (TMS) and neuroimaging studies have identified the left posterior superior temporal sulcus (pSTS) as a causal region involved in the generation of the McGurk effect, it remains unclear whether this region is critical only for this illusion or also for the more general benefits of congruent visual speech (e.g., increased accuracy and faster reaction times). Indeed, recent correlative research suggests that the benefits of congruent visual speech and the McGurk effect reflect largely independent mechanisms. To better understand how these different features of audiovisual integration are causally generated by the left pSTS, we used single-pulse TMS to temporarily impair processing while subjects were presented with either incongruent (McGurk) or congruent audiovisual combinations. Consistent with past research, we observed that TMS to the left pSTS significantly reduced the strength of the McGurk effect. Importantly, however, left pSTS stimulation did not affect the positive benefits of congruent audiovisual speech (increased accuracy and faster reaction times), demonstrating a causal dissociation between the two processes. Our results are consistent with models proposing that the pSTS is but one of multiple critical areas supporting audiovisual speech interactions. Moreover, these data add to a growing body of evidence suggesting that the McGurk effect is an imperfect surrogate measure for more general and ecologically valid audiovisual speech behaviors.

The effect of multisensory semantic congruency on unisensory object recognition in schizophrenia

Multisensory, as opposed to unisensory processing of stimuli, has been found to enhance the performance (e.g., reaction time, accuracy, and discrimination) of healthy individuals across various tasks. However, this enhancement is not as pronounced in patients with schizophrenia (SZ), indicating impaired multisensory integration (MSI) in these individuals. To the best of our knowledge, no study has yet investigated the impact of MSI deficits in the context of working memory, a domain highly reliant on multisensory processing and substantially impaired in schizophrenia. To address this research gap, we employed two adopted versions of the continuous object recognition task to investigate the effect of single-trail multisensory encoding on subsequent object recognition in 21 schizophrenia patients and 21 healthy controls (HC). Participants were tasked with discriminating between initial and repeated presentations. For the initial presentations, half of the stimuli were audiovisual pairings, while the other half were presented unimodal. The task-relevant stimuli were then presented a second time in a unisensory manner (either auditory stimuli in the auditory task or visual stimuli in the visual task). To explore the impact of semantic context on multisensory encoding, half of the audiovisual pairings were selected to be semantically congruent, while the remaining pairs were not semantically related to each other. Consistent with prior studies, our findings demonstrated that the impact of single-trial multisensory presentation during encoding remains discernible during subsequent object recognition. This influence could be distinguished based on the semantic congruity between the auditory and visual stimuli presented during the encoding. This effect was more robust in the auditory task. In the auditory task, when congruent multisensory pairings were encoded, both participant groups demonstrated a multisensory facilitation effect. This effect resulted in improved accuracy and RT performance. Regarding incongruent audiovisual encoding, as expected, HC did not demonstrate an evident multisensory facilitation effect on memory performance. In contrast, SZs exhibited an atypically accelerated reaction time during the subsequent auditory object recognition. Based on the predictive coding model we propose that this observed deviations indicate a reduced semantic modulatory effect and anomalous predictive errors signaling, particularly in the context of conflicting cross-modal sensory inputs in SZ.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Anterior cingulate glutamate levels associate with functional activation and connectivity during sensory integration in schizophrenia: a multimodal 1H-MRS and fMRI study

BACKGROUND

Glutamatergic dysfunction has been implicated in sensory integration deficits in schizophrenia, yet how glutamatergic function contributes to behavioural impairments and neural activities of sensory integration remains unknown.

METHODS

Fifty schizophrenia patients and 43 healthy controls completed behavioural assessments for sensory integration and underwent magnetic resonance spectroscopy (MRS) for measuring the anterior cingulate cortex (ACC) glutamate levels. The correlation between glutamate levels and behavioural sensory integration deficits was examined in each group. A subsample of 20 pairs of patients and controls further completed an audiovisual sensory integration functional magnetic resonance imaging (fMRI) task. Blood Oxygenation Level Dependent (BOLD) activation and task-dependent functional connectivity (FC) were assessed based on fMRI data. Full factorial analyses were performed to examine the Group-by-Glutamate Level interaction effects on fMRI measurements (group differences in correlation between glutamate levels and fMRI measurements) and the correlation between glutamate levels and fMRI measurements within each group.

RESULTS

We found that schizophrenia patients exhibited impaired sensory integration which was positively correlated with ACC glutamate levels. Multimodal analyses showed significantly Group-by-Glutamate Level interaction effects on BOLD activation as well as task-dependent FC in a 'cortico-subcortical-cortical' network (including medial frontal gyrus, precuneus, ACC, middle cingulate gyrus, thalamus and caudate) with positive correlations in patients and negative in controls.

CONCLUSIONS

Our findings indicate that ACC glutamate influences neural activities in a large-scale network during sensory integration, but the effects have opposite directionality between schizophrenia patients and healthy people. This implicates the crucial role of glutamatergic system in sensory integration processing in schizophrenia.

Deficient Audiovisual Speech Perception in Schizophrenia: An ERP Study

In everyday verbal communication, auditory speech perception is often disturbed by background noise. Especially in disadvantageous hearing conditions, additional visual articulatory information (e.g., lip movement) can positively contribute to speech comprehension. Patients with schizophrenia (SZs) demonstrate an aberrant ability to integrate visual and auditory sensory input during speech perception. Current findings about underlying neural mechanisms of this deficit are inconsistent. Particularly and despite the importance of early sensory processing in speech perception, very few studies have addressed these processes in SZs. Thus, in the present study, we examined 20 adult subjects with SZ and 21 healthy controls (HCs) while presenting audiovisual spoken words (disyllabic nouns) either superimposed by white noise (-12 dB signal-to-noise ratio) or not. In addition to behavioral data, event-related brain potentials (ERPs) were recorded. Our results demonstrate reduced speech comprehension for SZs compared to HCs under noisy conditions. Moreover, we found altered N1 amplitudes in SZ during speech perception, while P2 amplitudes and the N1-P2 complex were similar to HCs, indicating that there may be disturbances in multimodal speech perception at an early stage of processing, which may be due to deficits in auditory speech perception. Moreover, a positive relationship between fronto-central N1 amplitudes and the positive subscale of the Positive and Negative Syndrome Scale (PANSS) has been observed.

Rhythmic TMS as a Feasible Tool to Uncover the Oscillatory Signatures of Audiovisual Integration

Multisensory integration is quintessential to adaptive behavior, with clinical populations showing significant impairments in this domain, most notably hallucinatory reports. Interestingly, altered cross-modal interactions have also been reported in healthy individuals when engaged in tasks such as the Sound-Induced Flash-Illusion (SIFI). The temporal dynamics of the SIFI have been recently tied to the speed of occipital alpha rhythms (IAF), with faster oscillations entailing reduced temporal windows within which the illusion is experienced. In this regard, entrainment-based protocols have not yet implemented rhythmic transcranial magnetic stimulation (rhTMS) to causally test for this relationship. It thus remains to be evaluated whether rhTMS-induced acoustic and somatosensory sensations may not specifically interfere with the illusion. Here, we addressed this issue by asking 27 volunteers to perform a SIFI paradigm under different Sham and active rhTMS protocols, delivered over the occipital pole at the IAF. Although TMS has been proven to act upon brain tissues excitability, results show that the SIFI occurred for both Sham and active rhTMS, with the illusory rate not being significantly different between baseline and stimulation conditions. This aligns with the discrete sampling hypothesis, for which alpha amplitude modulation, known to reflect changes in cortical excitability, should not account for changes in the illusory rate. Moreover, these findings highlight the viability of rhTMS-based interventions as a means to probe the neuroelectric signatures of illusory and hallucinatory audiovisual experiences, in healthy and neuropsychiatric populations.

Audiovisual multisensory integration in individuals with reading and language impairments: A systematic review and meta-analysis

Differences in sensory function have been documented for a number of neurodevelopmental conditions, including reading and language impairments. Prior studies have measured audiovisual multisensory integration (i.e., the ability to combine inputs from the auditory and visual modalities) in these populations. The present study sought to systematically review and quantitatively synthesize the extant literature on audiovisual multisensory integration in individuals with reading and language impairments. A comprehensive search strategy yielded 56 reports, of which 38 were used to extract 109 group difference and 68 correlational effect sizes. There was an overall difference between individuals with reading and language impairments and comparisons on audiovisual integration. There was a nonsignificant trend towards moderation according to sample type (i.e., reading versus language) and publication/small study bias for this model. Overall, there was a small but non-significant correlation between metrics of audiovisual integration and reading or language ability; this model was not moderated by sample or study characteristics, nor was there evidence of publication/small study bias. Limitations and future directions for primary and meta-analytic research are discussed.

Where on the face do we look during phonemic restoration: An eye-tracking study

Face to face communication typically involves audio and visual components to the speech signal. To examine the effect of task demands on gaze patterns in response to a speaking face, adults participated in two eye-tracking experiments with an audiovisual (articulatory information from the mouth was visible) and a pixelated condition (articulatory information was not visible). Further, task demands were manipulated by having listeners respond in a passive (no response) or an active (button press response) context. The active experiment required participants to discriminate between speech stimuli and was designed to mimic environmental situations which require one to use visual information to disambiguate the speaker's message, simulating different listening conditions in real-world settings. Stimuli included a clear exemplar of the syllable /ba/ and a second exemplar in which the formant initial consonant was reduced creating an /a/-like consonant. Consistent with our hypothesis, results revealed that the greatest fixations to the mouth were present in the audiovisual active experiment and visual articulatory information led to a phonemic restoration effect for the /a/ speech token. In the pixelated condition, participants fixated on the eyes, and discrimination of the deviant token within the active experiment was significantly greater than the audiovisual condition. These results suggest that when required to disambiguate changes in speech, adults may look to the mouth for additional cues to support processing when it is available.

Effect of face masks on speech perception in noise of individuals with hearing aids

Although several previous studies have confirmed that listeners find it difficult to perceive the speech of face-mask-wearing speakers, there has been little research into how masks affect hearing-impaired individuals using hearing aids. Therefore, the aim of this study was to compare the effects of masks on the speech perception in noise of hearing-impaired individuals and normal-hearing individuals. We also investigated the effect of masks on the gain conferred by hearing aids. The hearing-impaired group included 24 listeners (age: M = 69.5, SD = 8.6; M:F = 13:11) who had used hearing aids in everyday life for >1 month (M = 20.7, SD = 24.0) and the normal-hearing group included 26 listeners (age: M = 57.9, SD = 11.1; M:F = 13:13). Speech perception in noise was measured under no mask-auditory-only (no-mask-AO), no mask-auditory-visual (no-mask-AV), and mask-AV conditions at five signal-to-noise ratios (SNRs; -16, -12, -8, -4, 0 dB) using five lists of 25 monosyllabic Korean words. Video clips that included a female speaker's face and sound or the sound only were presented through a monitor and a loudspeaker located 1 m in front of the listener in a sound-attenuating booth. The degree of deterioration in speech perception caused by the mask (no-mask-AV minus mask-AV) was significantly greater for hearing-impaired vs. normal-hearing participants only at 0 dB SNR (Bonferroni's corrected p < 0.01). When the effects of a mask on speech perception, with and without hearing aids, were compared in the hearing-impaired group, the degree of deterioration in speech perception caused by the mask was significantly reduced by the hearing aids compared with that without hearing aids at 0 and -4 dB SNR (Bonferroni's corrected p < 0.01). The improvement conferred by hearing aids (unaided speech perception score minus aided speech perception score) was significantly greater at 0 and -4 dB SNR than at -16 dB SNR in the mask-AV group (Bonferroni's corrected p < 0.01). These results demonstrate that hearing aids still improve speech perception when the speaker is masked, and that hearing aids partly offset the effect of a mask at relatively low noise levels.

Changing the Tendency to Integrate the Senses

Integration of sensory signals that emanate from the same source, such as the visual of lip articulations and the sound of the voice of a speaking individual, can improve perception of the source signal (e.g., speech). Because momentary sensory inputs are typically corrupted with internal and external noise, there is almost always a discrepancy between the inputs, facing the perceptual system with the problem of determining whether the two signals were caused by the same source or different sources. Thus, whether or not multisensory stimuli are integrated and the degree to which they are bound is influenced by factors such as the prior expectation of a common source. We refer to this factor as the tendency to bind stimuli, or for short, binding tendency. In theory, the tendency to bind sensory stimuli can be learned by experience through the acquisition of the probabilities of the co-occurrence of the stimuli. It can also be influenced by cognitive knowledge of the environment. The binding tendency varies across individuals and can also vary within an individual over time. Here, we review the studies that have investigated the plasticity of binding tendency. We discuss the protocols that have been reported to produce changes in binding tendency, the candidate learning mechanisms involved in this process, the possible neural correlates of binding tendency, and outstanding questions pertaining to binding tendency and its plasticity. We conclude by proposing directions for future research and argue that understanding mechanisms and recipes for increasing binding tendency can have important clinical and translational applications for populations or individuals with a deficiency in multisensory integration.

Neural correlates of multisensory enhancement in audiovisual narrative speech perception: a fMRI investigation

This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system. To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI. We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala. Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing. We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing.

Positive schizotypy is associated with amplified mnemonic discrimination and attenuated generalization

Tendency to experience inaccurate beliefs alongside perceptual anomalies constitutes positive schizotypal traits in the general population and shows continuity with the positive symptoms of schizophrenia. It has been hypothesized that the positive symptomatology of schizophrenia, and by extension, the odd beliefs and unusual perceptual experiences in the general population, are associated with specific alterations in memory functions. An imbalance between memory generalization and episodic memory specificity has been proposed on several counts; however, the direction of the imbalance is currently unclear. Here, we evaluated the association between positive schizotypy, and memory alterations related to hippocampal computations in a general population sample enriched for positive schizotypy. We found that memory generalization is attenuated while memory specificity is elevated in participants with more pronounced positive schizotypal traits. Our findings show that people who are prone to irrational beliefs and unusual experiences also show measurable alterations in memory and likely have difficulty grasping the global picture and rather be overpowered by fragments of information.

Deficits in Sense of Body Ownership, Sensory Processing, and Temporal Perception in Schizophrenia Patients With/Without Auditory Verbal Hallucinations

It has been claimed that individuals with schizophrenia have difficulty in self-recognition and, consequently, are unable to identify the sources of their sensory perceptions or thoughts, resulting in delusions, hallucinations, and unusual experiences of body ownership. The deficits also contribute to the enhanced rubber hand illusion (RHI; a body perception illusion, induced by synchronous visual and tactile stimulation). Evidence based on RHI paradigms is emerging that auditory information can make an impact on the sense of body ownership, which relies on the process of multisensory inputs and integration. Hence, we assumed that auditory verbal hallucinations (AVHs), as an abnormal auditory perception, could be linked with body ownership, and the RHI paradigm could be conducted in patients with AVHs to explore the underlying mechanisms. In this study, we investigated the performance of patients with/without AVHs in the RHI. We administered the RHI paradigm to 80 patients with schizophrenia (47 with AVHs and 33 without AVHs) and 36 healthy controls. We conducted the experiment under two conditions (synchronous and asynchronous) and evaluated the RHI effects by both objective and subjective measures. Both patient groups experienced the RHI more quickly and strongly than HCs. The RHI effects of patients with AVHs were significantly smaller than those of patients without AVHs. Another important finding was that patients with AVHs did not show a reduction in RHI under asynchronous conditions. These results emphasize the disturbances of the sense of body ownership in schizophrenia patients with/without AVHs and the associations with AVHs. Furthermore, it is suggested that patients with AVHs may have multisensory processing dysfunctions and internal timing deficits.

Early evoked brain activity underlies auditory and audiovisual speech recognition deficits in schizophrenia

•

Reduced N1 amplitudes reflect speech processing deficits in schizophrenia (SZ).

•

Crossmodal N1 amplitude suppression in audiovisual speech is preserved in SZ.

•

N1 amplitudes correlate with speech recognition performance in controls but not in SZ.

Objectives

People with Schizophrenia (SZ) show deficits in auditory and audiovisual speech recognition. It is possible that these deficits are related to aberrant early sensory processing, combined with an impaired ability to utilize visual cues to improve speech recognition. In this electroencephalography study we tested this by having SZ and healthy controls (HC) identify different unisensory auditory and bisensory audiovisual syllables at different auditory noise levels.

Methods

SZ (N = 24) and HC (N = 21) identified one of three different syllables (/da/, /ga/, /ta/) at three different noise levels (no, low, high). Half the trials were unisensory auditory and the other half provided additional visual input of moving lips. Task-evoked mediofrontal N1 and P2 brain potentials triggered to the onset of the auditory syllables were derived and related to behavioral performance.

Results

In comparison to HC, SZ showed speech recognition deficits for unisensory and bisensory stimuli. These deficits were primarily found in the no noise condition. Paralleling these observations, reduced N1 amplitudes to unisensory and bisensory stimuli in SZ were found in the no noise condition. In HC the N1 amplitudes were positively related to the speech recognition performance, whereas no such relationships were found in SZ. Moreover, no group differences in multisensory speech recognition benefits and N1 suppression effects for bisensory stimuli were observed.

Conclusion

Our study suggests that reduced N1 amplitudes reflect early auditory and audiovisual speech processing deficits in SZ. The findings that the amplitude effects were confined to salient speech stimuli and the attenuated relationship with behavioral performance in patients compared to HC, indicates a diminished decoding of the auditory speech signals in SZs. Our study also revealed relatively intact multisensory benefits in SZs, which implies that the observed auditory and audiovisual speech recognition deficits were primarily related to aberrant processing of the auditory syllables.

Alterations of dynamic functional connectivity between visual and executive-control networks in schizophrenia

Schizophrenia is a chronic mental disorder characterized by continuous or relapsing episodes of psychosis. While previous studies have detected functional network connectivity alterations in patients with schizophrenia, and most have focused on static functional connectivity. However, brain activity is believed to change dynamically over time. Therefore, we computed dynamic functional network connectivity using the sliding window method in 38 patients with schizophrenia and 31 healthy controls. We found that patients with schizophrenia exhibited higher occurrences in the weakly and sparsely connected state (state 3) than healthy controls, positively correlated with negative symptoms. In addition, patients exhibited fewer occurrences in a strongly connected state (state 4) than healthy controls. Lastly, the dynamic functional network connectivity between the right executive-control network and the medial visual network was decreased in schizophrenia patients compared to healthy controls. Our results further prove that brain activity is dynamic, and that alterations of dynamic functional network connectivity features might be a fundamental neural mechanism in schizophrenia.

Psychological Inflexibility in People with Chronic Psychosis: The Mediating Role of Self-Stigma and Social Functioning

Psychosis is associated with self-stigmatization and loss of social functioning that increase the severity of the disorder. Psychological inflexibility (PI)—an individual’s tendency to suppress undesirable private events—plays a fundamental role in the emergence and worst prognosis of psychosis. The main objective of this study was to analyze whether self-stigma and social functioning mediate the association of PI with the severity of psychosis in adults with chronic schizophrenia. The study was carried out with a sample of 103 outpatients. The Acceptance and Action Questionnaire, the Internalized Stigma of Mental Illness Scale, and the Social Functioning Scale were used for clinical assessments. Data analyses were performed by using the PROCESS macro for SPSS. Results showed that the link between PI and the severity of psychosis is not direct, but is better explained by mediation of the self-stigma and social functioning of those assessed. PI also predicts worse social functioning without the need to take self-stigma into account. Moreover, self-stigma alone does not predict the severity of psychotic symptoms; this relationship has to be mediated by social functioning. These findings suggest that interventions designed to increase psychological flexibility, such as Acceptance and Commitment Therapy (ACT), may offer an alternative to attenuate the negative impact of self-stigma and to improve the social functioning.

A systematic review of the neural correlates of multisensory integration in schizophrenia

Multisensory integration (MSI), in which sensory signals from different modalities are unified, is necessary for our comprehensive perception of and effective adaptation to the objects and events around us. However, individuals with schizophrenia suffer from impairments in MSI, which could explain typical symptoms like hallucination and reality distortion. Because the neural correlates of aberrant MSI in schizophrenia help us understand the physiognomy of this psychiatric disorder, we performed a systematic review of the current research on this subject. The literature search concerned investigated MSI in diagnosed schizophrenia patients compared to healthy controls using brain imaging. Seventeen of 317 identified studies were finally included. To assess risk of bias, the Newcastle-Ottawa quality assessment was used, and the review was written according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA). The results indicated that multisensory processes in schizophrenia are associated with aberrant, mainly reduced, neural activity in several brain regions, as measured by event-related potentials, oscillations, activity and connectivity. The conclusion is that a fronto-temporal region, comprising the frontal inferior gyrus, middle temporal gyrus and superior temporal gyrus/sulcus, along with the fusiform gyrus and dorsal visual stream in the occipital-parietal lobe are possible key regions of deficient MSI in schizophrenia.

Auditory processing in schizophrenia: Behavioural evidence of abnormal spatial awareness

Spatial processing deficits are the reason for many daily life problems of schizophrenia (SCZ) patients. In this study, we aimed to examine the possibility of abnormal bias to one hemifield, in form of hemispatial neglect and extinction, in auditory modality in SCZ. Twenty-five SCZ patients and 25 healthy individuals were compared on speech tasks to study the auditory neglect and extinction, as well as an auditory localization task for studying neglect. In the speech tasks, participants reproduced some nonsense syllables, played from one or two speakers on the right and/or left sides. On the localization task, examinees discriminated the subjective location of the noise stimuli presented randomly from five speakers. On the speech task, patients had significantly lower hit rates for the right ear compared with controls (p = 0.01). While healthy controls showed right ear advantage, SCZs showed a left ear priority. In the localization task, although both groups had a left-side bias, this bias was much more prominent for the patients (all p < 0.05). SCZ could potentially alter the auditory spatial function, which may appear in the form of auditory neglect and extinction on the right side, depending on the characteristics of patient population.

Schizotypal personality traits and multisensory integration: An investigation using the McGurk effect

Multisensory integration, the process by which sensory information from different sensory modalities are bound together, is hypothesized to contribute to perceptual symptomatology in schizophrenia, in whom multisensory integration differences have been consistently found. Evidence is emerging that these differences extend across the schizophrenia spectrum, including individuals in the general population with higher levels of schizotypal traits. In the current study, we used the McGurk task as a measure of multisensory integration. We measured schizotypal traits using the Schizotypal Personality Questionnaire (SPQ), hypothesizing that higher levels of schizotypal traits, specifically Unusual Perceptual Experiences and Odd Speech subscales, would be associated with decreased multisensory integration of speech. Surprisingly, Unusual Perceptual Experiences were not associated with multisensory integration. However, Odd Speech was associated with multisensory integration, and this association extended more broadly across the Disorganized factor of the SPQ, including Odd or Eccentric Behaviour. Individuals with higher levels of Odd or Eccentric Behaviour scores also demonstrated poorer lip-reading abilities, which partially explained performance in the McGurk task. This suggests that aberrant perceptual processes affecting individuals across the schizophrenia spectrum may relate to disorganized symptomatology.

The Impact of Temporally Coherent Visual Cues on Speech Perception in Complex Auditory Environments

Speech perception often takes place in noisy environments, where multiple auditory signals compete with one another. The addition of visual cues such as talkers’ faces or lip movements to an auditory signal can help improve the intelligibility of speech in those suboptimal listening environments. This is referred to as audiovisual benefits. The current study aimed to delineate the signal-to-noise ratio (SNR) conditions under which visual presentations of the acoustic amplitude envelopes have their most significant impact on speech perception. Seventeen adults with normal hearing were recruited. Participants were presented with spoken sentences in babble noise either in auditory-only or auditory-visual conditions with various SNRs at −7, −5, −3, −1, and 1 dB. The visual stimulus applied in this study was a sphere that varied in size syncing with the amplitude envelope of the target speech signals. Participants were asked to transcribe the sentences they heard. Results showed that a significant improvement in accuracy in the auditory-visual condition versus the audio-only condition was obtained at the SNRs of −3 and −1 dB, but no improvement was observed in other SNRs. These results showed that dynamic temporal visual information can benefit speech perception in noise, and the optimal facilitative effects of visual amplitude envelope can be observed under an intermediate SNR range.

Influence of prior beliefs on perception in early psychosis: Effects of illness stage and hierarchical level of belief

Alterations in the balance between prior expectations and sensory evidence may account for faulty perceptions and inferences leading to psychosis. However, uncertainties remain about the nature of altered prior expectations and the degree to which they vary with the emergence of psychosis. We explored how expectations arising at two different levels—cognitive and perceptual—influenced processing of sensory information and whether relative influences of higher- and lower-level priors differed across people with prodromal symptoms and those with psychotic illness. In two complementary auditory perception experiments, 91 participants (30 with first-episode psychosis, 29 at clinical risk for psychosis, and 32 controls) were required to decipher a phoneme within ambiguous auditory input. Expectations were generated in two ways: an accompanying visual input of lip movements observed during auditory presentation or through written presentation of a phoneme provided prior to auditory presentation. We determined how these different types of information shaped auditory perceptual experience, how this was altered across the prodromal and established phases of psychosis, and how this relates to cingulate glutamate levels assessed by magnetic resonance spectroscopy. The psychosis group relied more on high-level cognitive priors compared to both healthy controls and those at clinical risk for psychosis and relied more on low-level perceptual priors than the clinical risk group. The risk group was marginally less reliant on low-level perceptual priors than controls. The results are consistent with previous theory that influences of prior expectations in perceptions in psychosis differ according to level of prior and illness phase.

What we perceive and believe in any given moment will allow us to form expectations about what we will experience in the next. In psychosis, it is believed that the influence of these so-called perceptual and cognitive “prior” expectations on perception are altered, thereby giving rise to the symptoms seen in psychosis. However, research thus far has found mixed evidence, some suggesting an increase in the influence of priors and some finding a decrease. Here we test the hypothesis that perceptual and cognitive priors are differentially affected in individuals at risk for psychosis and individuals with a first episode of psychosis, thereby partially explaining the mixed findings in the literature. We indeed found evidence in favor of this hypothesis, finding weaker perceptual priors in individuals at risk but stronger cognitive priors in individuals with first-episode psychosis.

The role of expectancies and emotional load in false auditory perceptions among patients with schizophrenia spectrum disorders

Cognitive models suggest that top-down and emotional processes increase false perceptions in schizophrenia spectrum disorders (SSD). However, little is still known about the interaction of these processes in false auditory perceptions. The present study aimed at investigating the specific as well as joint impacts of expectancies and emotional load on false auditory perceptions in SSD. Thirty-three patients with SSD and 33 matched healthy controls were assessed with a false perception task. Participants were asked to detect a target stimulus (a word) in a white noise background (the word was present in 60% of the cases and absent in 40%). Conditions varied in terms of the level of expectancy (1. no cue prior to the stimulus, 2. semantic priming, 3. semantic priming accompanied by a video of a man's mouth spelling out the word). The words used were neutral or emotionally negative. Symptom severity was assessed with the Positive and Negative Syndrome Scale. Higher expectancy significantly increased the likelihood of false auditory perceptions only among the patients with SSD (the group x expectancy condition interaction was significant), which was unrelated to general cognitive performance. Emotional load had no impact on false auditory perceptions in either group. Patients made more false auditory perceptions with high confidence than controls did. False auditory perceptions were significantly correlated with the severity of positive symptoms and disorganization, but not with other dimensions. Perception in SSD seems to be susceptible to top-down processes, increasing the likelihood of high-confidence false auditory perceptions.

Exposure to first-person shooter videogames is associated with multisensory temporal precision and migraine incidence

Adaptive interactions with the environment require optimal integration and segregation of sensory information. Yet, temporal misalignments in the presentation of visual and auditory stimuli may generate illusory phenomena such as the sound-induced flash illusion, in which a single flash paired with multiple auditory stimuli induces the perception of multiple illusory flashes. This phenomenon has been shown to be robust and resistant to feedback training. According to a Bayesian account, this is due to a statistically optimal combination of the signals operated by the nervous system. From this perspective, individual susceptibility to the illusion might be moulded through prolonged experience. For example, repeated exposure to the illusion and prolonged training sessions partially impact on the reported illusion. Therefore, extensive and immersive audio-visual experience, such as first-person shooter videogames, should sharpen individual capacity to correctly integrate multisensory information over time, leading to more veridical perception. We tested this hypothesis by comparing the temporal profile of the sound-induced illusion in a group of expert first-person shooter gamers and a non-players group. In line with the hypotheses, gamers experience significantly narrower windows of illusion (~87 ms) relative to non-players (~105 ms), leading to higher veridical reports in gamers (~68%) relative to non-players (~59%). Moreover, according to recent literature, we tested whether audio-visual intensive training in gamers could be related to the incidence of migraine, and found that its severity may be directly proportioned to the time spent on videogames. Overall, these results suggest that continued training within audio-visual environments such as first-person shooter videogames improves temporal discrimination and sensory integration. This finding may pave the way for future therapeutic strategies based on self-administered multisensory training. On the other hand, the impact of intensive training on visual-related stress disorders, such as migraine incidence, should be taken into account as a risk factor during therapeutic planning.

Schizotypal traits are not related to multisensory integration or audiovisual speech perception

Multisensory integration, the binding of sensory information from different sensory modalities, may contribute to perceptual symptomatology in schizophrenia, including hallucinations and aberrant speech perception. Differences in multisensory integration and temporal processing, an important component of multisensory integration, are consistently found in schizophrenia. Evidence is emerging that these differences extend across the schizophrenia spectrum, including individuals in the general population with higher schizotypal traits. In the current study, we investigated the relationship between schizotypal traits and perceptual functioning, using audiovisual speech-in-noise, McGurk, and ternary synchrony judgment tasks. We measured schizotypal traits using the Schizotypal Personality Questionnaire (SPQ), hypothesizing that higher scores on Unusual Perceptual Experiences and Odd Speech subscales would be associated with decreased multisensory integration, increased susceptibility to distracting auditory speech, and less precise temporal processing. Surprisingly, these measures were not associated with the predicted subscales, suggesting that these perceptual differences may not be present across the schizophrenia spectrum.

Visual-Tactile Spatial Multisensory Interaction in Adults With Autism and Schizophrenia

Background: Individuals with autism spectrum disorder (ASD) and schizophrenia (SZ) exhibit multisensory processing difficulties and social impairments, with growing evidence that the former contributes to the latter. However, this work has largely reported on separate cohorts, introducing method variance as a barrier to drawing broad conclusions across studies. Further, very few studies have addressed touch, resulting in sparse knowledge about how these two clinical groups may integrate somatic information with other senses. Methods: In this study, we compared adults with ASD (n = 29), SZ (n = 24), and typical developmental histories (TD, n = 37) on two tasks requiring visual-tactile spatial multisensory processing. In the first task (crossmodal congruency), participants judged the location of a tactile stimulus in the presence or absence of simultaneous visual input that was either spatially congruent or incongruent, with poorer performance for incongruence an index of spatial multisensory interaction. In the second task, participants reacted to touch in the presence or absence of dynamic visual stimuli that appeared to approach or recede from the body. Within a certain radius around the body, defined as peripersonal space (PPS), an approaching visual or auditory stimulus reliably speeds reaction times (RT) to touch; outside of this radius, in extrapersonal space (EPS), there is no multisensory effect. PPS can be defined both by its size (radius) and slope (sharpness of the PPS-EPS boundary). Clinical measures were administered to explore relations with visual-tactile processing. Results: Neither clinical group differed from controls on the crossmodal congruency task. The ASD group had significantly smaller and more sharply-defined PPSs compared to the other two groups. Small PPS size was related to social symptom severity across groups, but was largely driven by the TD group, without significant effects in either clinical group. Conclusions: These results suggest that: (1) spatially static visual-tactile facilitation is intact in adults with ASD and SZ, (2) spatially dynamic visual-tactile facilitation impacting perception of the body boundary is affected in ASD but not SZ, and (3) body boundary perception is related to social-emotional function, but not in a way that maps on to clinical status.

Multisensory Integration of Emotion in Schizophrenic Patients

Multisensory integration (MSI) of emotion has been increasingly recognized as an essential element of schizophrenic patients' impairments, leading to the breakdown of their interpersonal functioning. The present review provides an updated synopsis of schizophrenics' MSI abilities in emotion processing by examining relevant behavioral and neurological research. Existing behavioral studies have adopted well-established experimental paradigms to investigate how participants understand multisensory emotion stimuli, and interpret their reciprocal interactions. Yet it remains controversial with regard to congruence-induced facilitation effects, modality dominance effects, and generalized vs specific impairment hypotheses. Such inconsistencies are likely due to differences and variations in experimental manipulations, participants' clinical symptomatology, and cognitive abilities. Recent electrophysiological and neuroimaging research has revealed aberrant indices in event-related potential (ERP) and brain activation patterns, further suggesting impaired temporal processing and dysfunctional brain regions, connectivity and circuities at different stages of MSI in emotion processing. The limitations of existing studies and implications for future MSI work are discussed in light of research designs and techniques, study samples and stimuli, and clinical applications.

Design and Evaluation of a Real-Time Audio Source Separation Algorithm to Remix Music for Cochlear Implant Users

A cochlear implant (CI) is a surgically implanted electronic device that partially restores hearing to people suffering from profound hearing loss. Although CI users, in general, obtain a very good reception of continuous speech in the absence of background noise, they face severe limitations in the context of music perception and appreciation. The main reasons for these limitations are related to channel interactions created by the broad spread of electrical fields in the cochlea and to the low number of electrodes that stimulate it. Moreover, CIs have severe limitations when it comes to transmitting the temporal fine structure of acoustic signals, and hence, these devices elicit poor pitch and timber perception. For these reasons, several signal processing algorithms have been proposed to make music more accessible for CI users, trying to reduce the complexity of music signals or remixing them to enhance certain components, such as the lead singing voice. In this work, a deep neural network that performs real-time audio source separation to remix music for CI users is presented. The implementation is based on multi-layer perception (MLP) and has been evaluated using objective instrumental measurements to ensure clean source estimation. Furthermore, experiments in 10 normal hearing (NH) and 13 CI users to investigate how the vocals to instruments ratio (VIR) set by the tested listeners were affected in realistic environments with and without visual information. The objective instrumental results fulfill the benchmark reported in previous studies by introducing distortions that are shown to not be perceived by CI users. Moreover, the implemented model was optimized to perform real-time source separation. The experimental results show that CI users prefer vocals 8 dB enhanced with the respect to the instruments independent of acoustic sound scenarios and visual information. In contrast, NH listeners did not prefer a VIR different than zero dB.

Multisensory Integration as a Window into Orderly and Disrupted Cognition and Communication

During our everyday lives, we are confronted with a vast amount of information from several sensory modalities. This multisensory information needs to be appropriately integrated for us to effectively engage with and learn from our world. Research carried out over the last half century has provided new insights into the way such multisensory processing improves human performance and perception; the neurophysiological foundations of multisensory function; the time course for its development; how multisensory abilities differ in clinical populations; and, most recently, the links between multisensory processing and cognitive abilities. This review summarizes the extant literature on multisensory function in typical and atypical circumstances, discusses the implications of the work carried out to date for theory and research, and points toward next steps for advancing the field.

Multisensory Audiovisual Processing in Children With a Sensory Processing Disorder (II): Speech Integration Under Noisy Environmental Conditions

Background: There exists a cohort of children and adults who exhibit an inordinately high degree of discomfort when experiencing what would be considered moderate and manageable levels of sensory input. That is, they show over-responsivity in the face of entirely typical sound, light, touch, taste, or smell inputs, and this occurs to such an extent that it interferes with their daily functioning and reaches clinical levels of dysfunction. What marks these individuals apart is that this sensory processing disorder (SPD) is observed in the absence of other symptom clusters that would result in a diagnosis of Autism, ADHD, or other neurodevelopmental disorders more typically associated with sensory processing difficulties. One major theory forwarded to account for these SPDs posits a deficit in multisensory integration, such that the various sensory inputs are not appropriately integrated into the central nervous system, leading to an overwhelming sensory-perceptual environment, and in turn to the sensory-defensive phenotype observed in these individuals. Methods: We tested whether children (6-16 years) with an over-responsive SPD phenotype (N = 12) integrated multisensory speech differently from age-matched typically-developing controls (TD: N = 12). Participants identified monosyllabic words while background noise level and sensory modality (auditory-alone, visual-alone, audiovisual) were varied in pseudorandom order. Improved word identification when speech was both seen and heard compared to when it was simply heard served to index multisensory speech integration. Results: School-aged children with an SPD show a deficit in the ability to benefit from the combination of both seen and heard speech inputs under noisy environmental conditions, suggesting that these children do not benefit from multisensory integrative processing to the same extent as their typically developing peers. In contrast, auditory-alone performance did not differ between the groups, signifying that this multisensory deficit is not simply due to impaired processing of auditory speech. Conclusions: Children with an over-responsive SPD show a substantial reduction in their ability to benefit from complementary audiovisual speech, to enhance speech perception in a noisy environment. This has clear implications for performance in the classroom and other learning environments. Impaired multisensory integration may contribute to sensory over-reactivity that is the definitional of SPD.

Processing of Emotions in Speech in Forensic Patients With Schizophrenia: Impairments in Identification, Selective Attention, and Integration of Speech Channels

Individuals with schizophrenia show deficits in recognition of emotions which may increase the risk of violence. This study explored how forensic patients with schizophrenia process spoken emotion by: (a) identifying emotions expressed in prosodic and semantic content separately, (b) selectively attending to one speech channel while ignoring the other, and (c) integrating the prosodic and the semantic channels, compared to non-clinical controls. Twenty-one forensic patients with schizophrenia and 21 matched controls listened to sentences conveying four emotions (anger, happiness, sadness, and neutrality) presented in semantic or prosodic channels, in different combinations. They were asked to rate how much they agreed that the sentences conveyed a predefined emotion, focusing on one channel or on the sentence as a whole. Forensic patients with schizophrenia performed with intact identification and integration of spoken emotions, but their ratings indicated reduced discrimination, larger failures of selective attention, and under-ratings of negative emotions, compared to controls. This finding doesn't support previous reports of an inclination to interpret social situations in a negative way among individuals with schizophrenia. Finally, current results may guide rehabilitation approaches matched to the pattern of auditory emotional processing presented by forensic patients with schizophrenia, improving social interactions and quality of life.

Brief Sensory Training Narrows the Temporal Binding Window and Enhances Long-Term Multimodal Speech Perception

Our ability to integrate multiple sensory-based representations of our surrounding supplies us with a more holistic view of our world. There are many complex algorithms our nervous system uses to construct a coherent perception. An indicator to solve this 'binding problem' are the temporal characteristics with the specificity that environmental information has different propagation speeds (e.g., sound and electromagnetic waves) and sensory processing time and thus the temporal relationship of a stimulus pair derived from the same event must be flexibly adjusted by our brain. This tolerance can be conceptualized in the form of the cross-modal temporal binding window (TBW). Several studies showed the plasticity of the TBW and its importance concerning audio-visual illusions, synesthesia, as well as psychiatric disturbances. Using three audio-visual paradigms, we investigated the importance of length (short vs. long) as well as modality (uni- vs. multimodal) of a perceptual training aiming at reducing the TBW in a healthy population. We also investigated the influence of the TBW on speech intelligibility, where participants had to integrate auditory and visual speech information from a videotaped speaker. We showed that simple sensory trainings can change the TBW and are capable of optimizing speech perception at a very naturalistic level. While the training-length had no different effect on the malleability of the TBW, the multisensory trainings induced a significantly stronger narrowing of the TBW than their unisensory counterparts. Furthermore, a narrowing of the TBW was associated with a better performance in speech perception, meaning that participants showed a greater capacity for integrating informations from different sensory modalities in situations with one modality impaired. All effects persisted at least seven days. Our findings show the significance of multisensory temporal processing regarding ecologically valid measures and have important clinical implications for interventions that may be used to alleviate debilitating conditions (e.g., autism, schizophrenia), in which multisensory temporal function is shown to be impaired.

Auditory Deficits in Audiovisual Speech Perception in Adult Asperger's Syndrome: fMRI Study

Audiovisual (AV) integration deficits have been proposed to underlie difficulties in speech perception in Asperger’s syndrome (AS). It is not known, if the AV deficits are related to alterations in sensory processing at the level of unisensory processing or at levels of conjoint multisensory processing. Functional Magnetic-resonance images (MRI) was performed in 16 adult subjects with AS and 16 healthy controls (HC) matched for age, gender, and verbal IQ as they were exposed to disyllabic AV congruent and AV incongruent nouns. A simple semantic categorization task was used to ensure subjects’ attention to the stimuli. The left auditory cortex (BA41) showed stronger activation in HC than in subjects with AS with no interaction regarding AV congruency. This suggests that alterations in auditory processing in unimodal low-level areas underlie AV speech perception deficits in AS. Whether this is signaling a difficulty in the deployment of attention remains to be demonstrated.

A possible key role of vision in the development of schizophrenia

Based on a brief overview of the various aspects of schizophrenia reported by numerous studies, here we hypothesize that schizophrenia may originate (and in part be performed) from visual areas. In other words, it seems that a normal visual system or at least an evanescent visual perception may be an essential prerequisite for the development of schizophrenia as well as of various types of hallucinations. Our study focuses on auditory and visual hallucinations, as they are the most prominent features of schizophrenic hallucinations (and also the most studied types of hallucinations). Here, we evaluate the possible key role of the visual system in the development of schizophrenia.

"Seeing Rain": Integrating phenomenological and Bayesian predictive coding approaches to visual hallucinations and self-disturbances (Ichstörungen) in schizophrenia

We present a schizophrenia patient who reports "seeing rain" with attendant somatosensory features which separate him from his surroundings. Because visual/multimodal hallucinations are understudied in schizophrenia, we examine a case history to determine the role of these hallucinations in self-disturbances (Ichstörungen). Developed by the early Heidelberg School, self-disturbances comprise two components: 1. The self experiences its own automatic processing as alien to self in a split-off, "doubled-I." 2. In "I-paralysis," the disruption to automatic processing is now outside the self in omnipotent agents. Self-disturbances (as indicated by visual/multimodal hallucinations) involve impairment in the ability to predict moment-to-moment experiences in the ongoing perception-action cycle. The phenomenological approach to subjective experience of self-disturbances complements efforts to model psychosis using the computational framework of hierarchical predictive coding. We conclude that self-disturbances play an adaptive, compensatory role following the uncoupling of perception and action, and possibly, other low-level perceptual anomalies.

Pushing the Envelope: Developments in Neural Entrainment to Speech and the Biological Underpinnings of Prosody Perception

Prosodic cues in speech are indispensable for comprehending a speaker’s message, recognizing emphasis and emotion, parsing segmental units, and disambiguating syntactic structures. While it is commonly accepted that prosody provides a fundamental service to higher-level features of speech, the neural underpinnings of prosody processing are not clearly defined in the cognitive neuroscience literature. Many recent electrophysiological studies have examined speech comprehension by measuring neural entrainment to the speech amplitude envelope, using a variety of methods including phase-locking algorithms and stimulus reconstruction. Here we review recent evidence for neural tracking of the speech envelope and demonstrate the importance of prosodic contributions to the neural tracking of speech. Prosodic cues may offer a foundation for supporting neural synchronization to the speech envelope, which scaffolds linguistic processing. We argue that prosody has an inherent role in speech perception, and future research should fill the gap in our knowledge of how prosody contributes to speech envelope entrainment.

The experiences and perceptions of individuals diagnosed with schizophrenia regarding the challenges they experience to employment and coping strategies used in the open labor market in Cape Town, South Africa

BACKGROUND

There is limited research available regarding the coping skills of individuals living with schizophrenia and the strategies used in the open labor market, particularly from a South African context.

OBJECTIVE

The aim of the study was to explore and describe the barriers, facilitators and coping strategies that individuals with schizophrenia use when returning to the open labor market after participating in a vocational rehabilitation program to improve work skills.

PARTICIPANTS

Four individuals diagnosed with schizophrenia and two key informants (occupational therapists) participated in the study.

METHODS

The research study was positioned within the interpretive paradigm, specifically utilizing an exploratory and descriptive design. Semi-structured interviews were used in order to gather data from the participants.

RESULTS

Theme one reflects the barriers related to returning to work (i.e. society's acceptance of an individual's work potential). The second theme relates to the enabling factors related to returning to work (i.e. the usefulness of work preparation programs to enhance open labor market employability). Theme three relates to the coping skills that individuals with schizophrenia utilize (i.e. holistic support enhances participation in the worker role).

CONCLUSION

In conclusion, the findings indicate individuals with schizophrenia experience barriers that prevent them from being able to adapt to their work environments. However, with the support of the occupational therapist, family, and employer, together with improvements to the vocational rehabilitation program, these barriers may be alleviated, and there could be an improvement in the reintegration of individuals with schizophrenia into the open labor market. The findings suggest that the disclosure of an individual's medical diagnosis, in order to facilitate the return to work of an individual diagnosed with schizophrenia, could be viewed as a barrier and a facilitator.

Origin and evolution of human speech: Emergence from a trimodal auditory, visual and vocal network

In recent years, there have been important additions to the classical model of speech processing as originally depicted by the Broca-Wernicke model consisting of an anterior, productive region and a posterior, perceptive region, both connected via the arcuate fasciculus. The modern view implies a separation into a dorsal and a ventral pathway conveying different kinds of linguistic information, which parallels the organization of the visual system. Furthermore, this organization is highly conserved in evolution and can be seen as the neural scaffolding from which the speech networks originated. In this chapter we emphasize that the speech networks are embedded in a multimodal system encompassing audio-vocal and visuo-vocal connections, which can be referred to an ancestral audio-visuo-motor pathway present in nonhuman primates. Likewise, we propose a trimodal repertoire for speech processing and acquisition involving auditory, visual and motor representations of the basic elements of speech: phoneme, observation of mouth movements, and articulatory processes. Finally, we discuss this proposal in the context of a scenario for early speech acquisition in infants and in human evolution.

The circuit architecture of cortical multisensory processing: Distinct functions jointly operating within a common anatomical network

Our perceptual systems continuously process sensory inputs from different modalities and organize these streams of information such that our subjective representation of the outside world is a unified experience. By doing so, they also enable further cognitive processing and behavioral action. While cortical multisensory processing has been extensively investigated in terms of psychophysics and mesoscale neural correlates, an in depth understanding of the underlying circuit-level mechanisms is lacking. Previous studies on circuit-level mechanisms of multisensory processing have predominantly focused on cue integration, i.e. the mechanism by which sensory features from different modalities are combined to yield more reliable stimulus estimates than those obtained by using single sensory modalities. In this review, we expand the framework on the circuit-level mechanisms of cortical multisensory processing by highlighting that multisensory processing is a family of functions - rather than a single operation - which involves not only the integration but also the segregation of modalities. In addition, multisensory processing not only depends on stimulus features, but also on cognitive resources, such as attention and memory, as well as behavioral context, to determine the behavioral outcome. We focus on rodent models as a powerful instrument to study the circuit-level bases of multisensory processes, because they enable combining cell-type-specific recording and interventional techniques with complex behavioral paradigms. We conclude that distinct multisensory processes share overlapping anatomical substrates, are implemented by diverse neuronal micro-circuitries that operate in parallel, and are flexibly recruited based on factors such as stimulus features and behavioral constraints.

Examining the Reversibility of Long-Term Behavioral Disruptions in Progeny of Maternal SSRI Exposure

Serotonergic dysregulation is implicated in numerous psychiatric disorders. Serotonin plays widespread trophic roles during neurodevelopment; thus perturbations to this system during development may increase risk for neurodevelopmental disorders. Epidemiological studies have examined association between selective serotonin reuptake inhibitor (SSRI) treatment during pregnancy and increased autism spectrum disorder (ASD) risk in offspring. It is unclear from these studies whether ASD susceptibility is purely related to maternal psychiatric diagnosis, or if treatment poses additional risk. We sought to determine whether maternal SSRI treatment alone or in combination with genetically vulnerable background was sufficient to induce offspring behavior disruptions relevant to ASD. We exposed C57BL/6J or Celf6^+/- mouse dams to fluoxetine (FLX) during different periods of gestation and lactation and characterized offspring on tasks assessing social communicative interaction and repetitive behavior patterns including sensory sensitivities. We demonstrate robust reductions in pup ultrasonic vocalizations (USVs) and alterations in social hierarchy behaviors, as well as perseverative behaviors and tactile hypersensitivity. Celf6 mutant mice demonstrate social communicative deficits and perseverative behaviors, without further interaction with FLX. FLX re-exposure in adulthood ameliorates the tactile hypersensitivity yet exacerbates the dominance phenotype. This suggests acute deficiencies in serotonin levels likely underlie the abnormal responses to sensory stimuli, while the social alterations are instead due to altered development of social circuits. These findings indicate maternal FLX treatment, independent of maternal stress, can induce behavioral disruptions in mammalian offspring, thus contributing to our understanding of the developmental role of the serotonin system and the possible risks to offspring of SSRI treatment during pregnancy.

Schizophrenia alters intra-network functional connectivity in the caudate for detecting speech under informational speech masking conditions

BACKGROUND

Speech recognition under noisy "cocktail-party" environments involves multiple perceptual/cognitive processes, including target detection, selective attention, irrelevant signal inhibition, sensory/working memory, and speech production. Compared to health listeners, people with schizophrenia are more vulnerable to masking stimuli and perform worse in speech recognition under speech-on-speech masking conditions. Although the schizophrenia-related speech-recognition impairment under "cocktail-party" conditions is associated with deficits of various perceptual/cognitive processes, it is crucial to know whether the brain substrates critically underlying speech detection against informational speech masking are impaired in people with schizophrenia.

METHODS

Using functional magnetic resonance imaging (fMRI), this study investigated differences between people with schizophrenia (n = 19, mean age = 33 ± 10 years) and their matched healthy controls (n = 15, mean age = 30 ± 9 years) in intra-network functional connectivity (FC) specifically associated with target-speech detection under speech-on-speech-masking conditions.

RESULTS

The target-speech detection performance under the speech-on-speech-masking condition in participants with schizophrenia was significantly worse than that in matched healthy participants (healthy controls). Moreover, in healthy controls, but not participants with schizophrenia, the strength of intra-network FC within the bilateral caudate was positively correlated with the speech-detection performance under the speech-masking conditions. Compared to controls, patients showed altered spatial activity pattern and decreased intra-network FC in the caudate.

CONCLUSIONS

In people with schizophrenia, the declined speech-detection performance under speech-on-speech masking conditions is associated with reduced intra-caudate functional connectivity, which normally contributes to detecting target speech against speech masking via its functions of suppressing masking-speech signals.

Integration and Temporal Processing of Asynchronous Audiovisual Speech

Multisensory integration of visual mouth movements with auditory speech is known to offer substantial perceptual benefits, particularly under challenging (i.e., noisy) acoustic conditions. Previous work characterizing this process has found that ERPs to auditory speech are of shorter latency and smaller magnitude in the presence of visual speech. We sought to determine the dependency of these effects on the temporal relationship between the auditory and visual speech streams using EEG. We found that reductions in ERP latency and suppression of ERP amplitude are maximal when the visual signal precedes the auditory signal by a small interval and that increasing amounts of asynchrony reduce these effects in a continuous manner. Time–frequency analysis revealed that these effects are found primarily in the theta (4–8 Hz) and alpha (8–12 Hz) bands, with a central topography consistent with auditory generators. Theta effects also persisted in the lower portion of the band (3.5–5 Hz), and this late activity was more frontally distributed. Importantly, the magnitude of these late theta oscillations not only differed with the temporal characteristics of the stimuli but also served to predict participants' task performance. Our analysis thus reveals that suppression of single-trial brain responses by visual speech depends strongly on the temporal concordance of the auditory and visual inputs. It further illustrates that processes in the lower theta band, which we suggest as an index of incongruity processing, might serve to reflect the neural correlates of individual differences in multisensory temporal perception.

The underlying mechanism of deficits of speech comprehension and hallucinations in Chinese patients with schizophrenia

Sentence context and fundamental frequency (F0) contours are important factors to speech perception and comprehension. In Chinese-Mandarin, lexical tones can be distinguished by the F0 contours. Previous studies found healthy people could use the cue of context to recover the phonological representations of lexical tones from the altered tonal patterns to comprehend the sentences in quiet condition, but can not in noise environment. Lots of research showed that patients with schizophrenia have deficits of speech perception and comprehension. However, it is unclear how context and F0 contours influence speech perception and comprehension in patients with schizophrenia. This study detected the contribution of context and lexical tone to sentence comprehension in four types of sentences by manipulating the context and F0 contours in 32 patients with schizophrenia and 33 healthy controls. The results showed that (1) in patients with schizophrenia, the interaction between context and F0 contour was not significant, which was significant in healthy controls; (2) the scores of sentences with two types of sentences with flattened F0 contours were negatively correlated with hallucination trait scores; (3) the patients with schizophrenia showed significantly lower scores on the intelligibility of sentences in all conditions, which were negatively correlated with PANSS-P. The patients with schizophrenia couldn't use the cue of context to recover the phonological representations of lexical tones from the altered tonal patterns when they comprehend the sentences, inner noise may be the underlying mechanism for the deficits of speech perception and comprehension.

Ripe for solution: Delayed development of multisensory processing in autism and its remediation

Difficulty integrating inputs from different sensory sources is commonly reported in individuals with Autism Spectrum Disorder (ASD). Accumulating evidence consistently points to altered patterns of behavioral reactions and neural activity when individuals with ASD observe or act upon information arriving through multiple sensory systems. For example, impairments in the integration of seen and heard speech appear to be particularly acute, with obvious implications for interpersonal communication. Here, we explore the literature on multisensory processing in autism with a focus on developmental trajectories. While much remains to be understood, some consistent observations emerge. Broadly, sensory integration deficits are found in children with an ASD whereas these appear to be much ameliorated, or even fully recovered, in older teenagers and adults on the spectrum. This protracted delay in the development of multisensory processing raises the possibility of applying early intervention strategies focused on multisensory integration, to accelerate resolution of these functions. We also consider how dysfunctional cross-sensory oscillatory neural communication may be one key pathway to impaired multisensory processing in ASD.

Neural Correlates of Sensory Abnormalities Across Developmental Disabilities

Abnormalities in sensory processing are a common feature of many developmental disabilities (DDs). Sensory dysfunction can contribute to deficits in brain maturation, as well as many vital functions. Unfortunately, while some patients with DD benefit from the currently available treatments for sensory dysfunction, many do not. Deficiencies in clinical practice surrounding sensory dysfunction may be related to lack of understanding of the neural mechanisms that underlie sensory abnormalities. Evidence of overlap in sensory symptoms between diagnoses suggests that there may be common neural mechanisms that mediate many aspects of sensory dysfunction. Thus, the current manuscript aims to review the extant literature regarding the neural correlates of sensory dysfunction across DD in order to identify patterns of abnormality that span diagnostic categories. Such anomalies in brain structure, function, and connectivity may eventually serve as targets for treatment.

Listening to Sentences in Noise: Revealing Binaural Hearing Challenges in Patients with Schizophrenia

OBJECTIVE

The present, case-control, study investigates binaural hearing performance in schizophrenia patients towards sentences presented in quiet and noise.

METHODS

Participants were twenty-one healthy controls and sixteen schizophrenia patients with normal peripheral auditory functions. The binaural hearing was examined in four listening conditions by using the Malay version of hearing in noise test. The syntactically and semantically correct sentences were presented via headphones to the randomly selected subjects. In each condition, the adaptively obtained reception thresholds for speech (RTS) were used to determine RTS noise composite and spatial release from masking.

RESULTS

Schizophrenia patients demonstrated significantly higher mean RTS value relative to healthy controls (p=0.018). The large effect size found in three listening conditions, i.e., in quiet (d=1.07), noise right (d=0.88) and noise composite (d=0.90) indicates statistically significant difference between the groups. However, noise front and noise left conditions show medium (d=0.61) and small (d=0.50) effect size respectively. No statistical difference between groups was noted in regards to spatial release from masking on right (p=0.305) and left (p=0.970) ear.

CONCLUSION

The present findings suggest an abnormal unilateral auditory processing in central auditory pathway in schizophrenia patients. Future studies to explore the role of binaural and spatial auditory processing were recommended.