Making sense of multisensory integration

Forwarding the Science of Sensory Features in Autism and Related Conditions

This editorial accompanies the JADD Special Issue on Sensory Features in Autism and Related Conditions: Developmental Approaches, Mechanisms and Targeted Interventions. The editorial is a commentary on the state of the science in sensory features in autism and related conditions and provides a synopsis of the information contained in the special issue including provocative thoughts about moving the field forward in this area.

Operating in a Multisensory Context: Assessing the Interplay Between Multisensory Reaction Time Facilitation and Inter-sensory Task-switching Effects

Individuals respond faster to presentations of bisensory stimuli (e.g. audio-visual targets) than to presentations of either unisensory constituent in isolation (i.e. to the auditory-alone or visual-alone components of an audio-visual stimulus). This well-established multisensory speeding effect, termed the redundant signals effect (RSE), is not predicted by simple linear summation of the unisensory response time probability distributions. Rather, the speeding is typically faster than this prediction, leading researchers to ascribe the RSE to a so-called co-activation account. According to this account, multisensory neural processing occurs whereby the unisensory inputs are integrated to produce more effective sensory-motor activation. However, the typical paradigm used to test for RSE involves random sequencing of unisensory and bisensory inputs in a mixed design, raising the possibility of an alternate attention-switching account. This intermixed design requires participants to switch between sensory modalities on many task trials (e.g. from responding to a visual stimulus to an auditory stimulus). Here we show that much, if not all, of the RSE under this paradigm can be attributed to slowing of reaction times to unisensory stimuli resulting from modality switching, and is not in fact due to speeding of responses to AV stimuli. As such, the present data do not support a co-activation account, but rather suggest that switching and mixing costs akin to those observed during classic task-switching paradigms account for the observed RSE.

Ripe for solution: Delayed development of multisensory processing in autism and its remediation

Difficulty integrating inputs from different sensory sources is commonly reported in individuals with Autism Spectrum Disorder (ASD). Accumulating evidence consistently points to altered patterns of behavioral reactions and neural activity when individuals with ASD observe or act upon information arriving through multiple sensory systems. For example, impairments in the integration of seen and heard speech appear to be particularly acute, with obvious implications for interpersonal communication. Here, we explore the literature on multisensory processing in autism with a focus on developmental trajectories. While much remains to be understood, some consistent observations emerge. Broadly, sensory integration deficits are found in children with an ASD whereas these appear to be much ameliorated, or even fully recovered, in older teenagers and adults on the spectrum. This protracted delay in the development of multisensory processing raises the possibility of applying early intervention strategies focused on multisensory integration, to accelerate resolution of these functions. We also consider how dysfunctional cross-sensory oscillatory neural communication may be one key pathway to impaired multisensory processing in ASD.

A Computational Analysis of Neural Mechanisms Underlying the Maturation of Multisensory Speech Integration in Neurotypical Children and Those on the Autism Spectrum

Failure to appropriately develop multisensory integration (MSI) of audiovisual speech may affect a child's ability to attain optimal communication. Studies have shown protracted development of MSI into late-childhood and identified deficits in MSI in children with an autism spectrum disorder (ASD). Currently, the neural basis of acquisition of this ability is not well understood. Here, we developed a computational model informed by neurophysiology to analyze possible mechanisms underlying MSI maturation, and its delayed development in ASD. The model posits that strengthening of feedforward and cross-sensory connections, responsible for the alignment of auditory and visual speech sound representations in posterior superior temporal gyrus/sulcus, can explain behavioral data on the acquisition of MSI. This was simulated by a training phase during which the network was exposed to unisensory and multisensory stimuli, and projections were crafted by Hebbian rules of potentiation and depression. In its mature architecture, the network also reproduced the well-known multisensory McGurk speech effect. Deficits in audiovisual speech perception in ASD were well accounted for by fewer multisensory exposures, compatible with a lack of attention, but not by reduced synaptic connectivity or synaptic plasticity.

Sensitivity and Bias in the Resolution of Stream-Bounce Stimuli

The audiovisual stream-bounce effect refers to the resolution of ambiguous motion sequences as streaming or bouncing depending on the presence or absence of a sound. We used a novel experimental design and signal detection theory (SDT) to determine its sensory or decisional origins. To account for issues raised by Witt et al. on the interpretation of SDT results, we devised a pure signal detection (as opposed to signal discrimination) paradigm and measured participants' sensitivity and criterion when detecting a weak tone concurrent with objectively streaming or bouncing visual displays. We observed no change in sensitivity but a significant change in criterion with participants' criterion more liberal with bouncing targets than for streaming targets with. In a second experiment, we tasked participants with detecting a weak tone in noise while viewing an ambiguous motion sequence. They also indicated whether the targets appeared to stream or bounce. Participants' reported equivalent, mostly bouncing responses for hit and false alarm trials, and equivalent, mostly streaming responses for correct rejection and miss trials. Further, differences in participants' sensitivity and criterion measures for detecting tones in subjectively streaming compared to subjectively bouncing targets were inconsistent with sensory factors. These results support a decisional account of the sound-induced switch from mostly streaming to mostly bouncing responses in audiovisual stream-bounce displays.

Role of the anterior insular cortex in integrative causal signaling during multisensory auditory-visual attention

Coordinated attention to information from multiple senses is fundamental to our ability to respond to salient environmental events, yet little is known about brain network mechanisms that guide integration of information from multiple senses. Here we investigate dynamic causal mechanisms underlying multisensory auditory-visual attention, focusing on a network of right-hemisphere frontal-cingulate-parietal regions implicated in a wide range of tasks involving attention and cognitive control. Participants performed three 'oddball' attention tasks involving auditory, visual and multisensory auditory-visual stimuli during fMRI scanning. We found that the right anterior insula (rAI) demonstrated the most significant causal influences on all other frontal-cingulate-parietal regions, serving as a major causal control hub during multisensory attention. Crucially, we then tested two competing models of the role of the rAI in multisensory attention: an 'integrated' signaling model in which the rAI generates a common multisensory control signal associated with simultaneous attention to auditory and visual oddball stimuli versus a 'segregated' signaling model in which the rAI generates two segregated and independent signals in each sensory modality. We found strong support for the integrated, rather than the segregated, signaling model. Furthermore, the strength of the integrated control signal from the rAI was most pronounced on the dorsal anterior cingulate and posterior parietal cortices, two key nodes of saliency and central executive networks respectively. These results were preserved with the addition of a superior temporal sulcus region involved in multisensory processing. Our study provides new insights into the dynamic causal mechanisms by which the AI facilitates multisensory attention.

Projections to early visual areas v1 and v2 in the calcarine fissure from parietal association areas in the macaque

Non-extrastriate projections to area V1 in monkeys, now demonstrated by several anatomical studies, are potential substrates of physiologically documented multisensory effects in primary sensory areas. The full network of projections among association and primary areas, however, is likely to be complex and is still only partially understood. In the present report, we used the anterograde tracer biotinylated dextran amine to investigate projections to areas V1 and V2 from subdivisions of the parietal association cortex in macaque. Parietal cortex was chosen to allow comparisons between projections from this higher association area and from other previously reported areas. In addition, we were interested in further elucidating pathways to areas V1 and V2 from parietal areas, as potentially contributing to attention and active vision. Of eight cases, three brains had projections only to area V2, and the five others projected to both areas V1 and V2. Terminations in area V1 were sparse. These were located in supragranular layers I, II, upper III; occasionally in IVB; and in layer VI. Terminations in V2 were denser, and slightly more prevalent in the supragranular layers. For both areas, terminations were in the calcarine region, corresponding to the representation of the peripheral visual field. By reconstructions of single axons, we demonstrated that four of nine axons had collaterals, either to V1 and V2 (n = 1) or to area V1 and a ventral area likely to be TEO (n = 3). In area V1, axons extended divergently in layer VI as well as layer I. Overall, these and previous results suggest a nested connectivity architecture, consisting of multiple direct and indirect recurrent projections from association areas to area V1. Terminations in area V1 are not abundant, but could be potentiated by the network of indirect connections.