Multisensory simultaneity judgment and proximity to the body

Alpha-band sensory entrainment improves audiovisual temporal acuity

Visual and auditory stimuli are transmitted from the environment to sensory cortices with different timing, requiring the brain to encode when sensory inputs must be segregated or integrated into a single percept. The probability that different audiovisual (AV) stimuli are integrated into a single percept even when presented asynchronously is reflected in the construct of temporal binding window (TBW). There is a strong interest in testing whether it is possible to broaden or shrink TBW by using different neuromodulatory approaches that can speed up or slow down ongoing alpha oscillations, which have been repeatedly hypothesized to be an important determinant of the TBWs size. Here, we employed a web-based sensory entrainment protocol combined with a simultaneity judgment task using simple flash-beep stimuli. The aim was to test whether AV temporal acuity could be modulated trial by trial by synchronizing ongoing neural oscillations in the prestimulus period to a rhythmic sensory stream presented in the upper (∼12 Hz) or lower (∼8.5 Hz) alpha range. As a control, we implemented a nonrhythmic condition where only the first and the last entrainers were employed. Results show that upper alpha entrainment shrinks AV TBW and improves AV temporal acuity when compared with lower alpha and control conditions. Our findings represent a proof of concept of the efficacy of sensory entrainment to improve AV temporal acuity in a trial-by-trial manner, and they strengthen the idea that alpha oscillations may reflect the temporal unit of AV temporal binding.

The Remapping of Peripersonal Space in a Real but Not in a Virtual Environment

One of the most surprising features of our brain is the fact that it is extremely plastic. Among the various plastic processes supported by our brain, there is the neural representation of the space surrounding our body, the peripersonal space (PPS). The effects of real-world tool use on the PPS are well known in cognitive neuroscience, but little is still known whether similar mechanisms also govern virtual tool use. To this purpose, the present study investigated the plasticity of the PPS before and after a real (Experiment 1) or virtual motor training with a tool (Experiment 2). The results show the expansion of the PPS only following real-world tool use but not virtual use, highlighting how the two types of training potentially rely on different processes. This study enriches the current state of the art on the plasticity of PPS in real and virtual environments. We discuss our data with respect to the relevance for the development of effective immersive environment for trainings, learning and rehabilitation.

Acute stress affects peripersonal space representation in cortisol stress responders

Peripersonal space is the representation of the space near the body. It is implemented by a dedicated multisensory-motor network, whose purpose is to predict and plan interactions with the environment, and which can vary depending on environmental circumstances. Here, we investigated the effect on the PPS representation of an experimentally induced stress response and compared it to a control, non-stressful, manipulation. We assessed PPS representation in healthy humans, before and after a stressful manipulation, by quantifying visuotactile interactions as a function of the distance from the body, while monitoring salivary cortisol concentration. While PPS representation was not significantly different between the control and experimental group, a relation between cortisol response and changes in PPS emerged within the experimental group. Participants who showed a cortisol stress response presented enhanced visuotactile integration for stimuli close to the body and reduced for far stimuli. Conversely, individuals with a less pronounced cortisol response showed a reduced difference in visuotactile integration between the near and the far space. In our interpretation, physiological stress resulted in a freezing-like response, where multisensory-motor resources are allocated only to the area immediately surrounding the body.

The magnitude of the sound-induced flash illusion does not increase monotonically as a function of visual stimulus eccentricity

The sound-induced flash illusion (SIFI) occurs when a rapidly presented visual stimulus is accompanied by two auditory stimuli, creating the illusory percept of two visual stimuli. While much research has focused on how the temporal proximity of the audiovisual stimuli impacts susceptibility to the illusion, comparatively less research has focused on the impact of spatial manipulations. Here, we aimed to assess whether manipulating the eccentricity of visual flash stimuli altered the properties of the temporal binding window associated with the SIFI. Twenty participants were required to report whether they perceived one or two flashes that were concurrently presented with one or two beeps. Visual stimuli were presented at one of four different retinal eccentricities (2.5, 5, 7.5, or 10 degrees below fixation) and audiovisual stimuli were separated by one of eight stimulus-onset asynchronies. In keeping with previous findings, increasing stimulus-onset asynchrony between the auditory and visual stimuli led to a marked decrease in susceptibility to the illusion allowing us to estimate the width and amplitude of the temporal binding window. However, varying the eccentricity of the visual stimulus had no effect on either the width or the peak amplitude of the temporal binding window, with a similar pattern of results observed for both the “fission” and “fusion” variants of the illusion. Thus, spatial manipulations of the audiovisual stimuli used to elicit the SIFI appear to have a weaker effect on the integration of sensory signals than temporal manipulations, a finding which has implications for neuroanatomical models of multisensory integration.

Lower multisensory temporal acuity in individuals with high schizotypal traits: a web-based study

Natural events are often multisensory, requiring the brain to combine information from the same spatial location and timing, across different senses. The importance of temporal coincidence has led to the introduction of the temporal binding window (TBW) construct, defined as the time range within which multisensory inputs are highly likely to be perceptually bound into a single entity. Anomalies in TBWs have been linked to confused perceptual experiences and inaccurate filtering of sensory inputs coming from different environmental sources. Indeed, larger TBWs have been associated with disorders such as schizophrenia and autism and are also correlated to a higher level of subclinical traits of these conditions in the general population. Here, we tested the feasibility of using a web-based version of a classic audio-visual simultaneity judgment (SJ) task with simple flash-beep stimuli in order to measure multisensory temporal acuity and its relationship with schizotypal traits as measured in the general population. Results show that: (i) the response distribution obtained in the web-based SJ task was strongly similar to those reported by studies carried out in controlled laboratory settings, and (ii) lower multisensory temporal acuity was associated with higher schizotypal traits in the “cognitive-perceptual” domains. Our findings reveal the possibility of adequately using a web-based audio-visual SJ task outside a controlled laboratory setting, available to a more diverse and representative pool of participants. These results provide additional evidence for a close relationship between lower multisensory acuity and the expression of schizotypal traits in the general population.

Time Interaction With Two Spatial Dimensions: From Left/Right to Near/Far

In this study, we explored the time and space relationship according to two different spatial codings, namely, the left/right extension and the reachability of stimulus along a near/far dimension. Four experiments were carried out in which healthy participants performed the time and spatial bisection tasks in near/far space, before and after short or long tool-use training. Stimuli were prebisected horizontal lines of different temporal durations in which the midpoint was manipulated according to the Muller-Lyer illusion. The perceptual illusory effects emerged in spatial but not temporal judgments. We revealed that temporal and spatial representations dynamically change according to the action potentialities of an individual: temporal duration was perceived as shorter and the perceived line’s midpoint was shifted to the left in far than in near space. Crucially, this dissociation disappeared following a long but not short tool-use training. Finally, we observed age-related differences in spatial attention which may be crucial in building the memory temporal standard to categorize durations.

Multisensory Integration Is Modulated by Hypnotizability

This study investigated multisensory integration in 29 medium-to-high (mid-highs) and 24 low-to-medium (mid-lows) hypnotizable individuals, classified according to the Stanford Hypnotic Susceptibility Scale, Form A. Participants completed a simultaneity judgment (SJ) task, where an auditory and a visual stimulus were presented in close proximity to their body in a range of 11 stimulus onset asynchronies. Results show that mid-highs were prone to judge audiovisual stimuli as simultaneous over a wider range of time intervals between sensory stimuli, as expressed by a broader temporal binding window, when the visual stimulus precedes the auditory one. No significant difference was observed for response times. Findings indicate a role of hypnotizability in multisensory integration likely due to the highs' cerebellar peculiarities and/or sensory modality preference.

Multisensory integration in the mouse cortical connectome using a network diffusion model

Having a structural network representation of connectivity in the brain is instrumental in analyzing communication dynamics and neural information processing. In this work, we make steps towards understanding multisensory information flow and integration using a network diffusion approach. In particular, we model the flow of evoked activity, initiated by stimuli at primary sensory regions, using the asynchronous linear threshold (ALT) diffusion model. The ALT model captures how evoked activity that originates at a given region of the cortex “ripples through” other brain regions (referred to as an activation cascade). We find that a small number of brain regions–the claustrum and the parietal temporal cortex being at the top of the list–are involved in almost all cortical sensory streams. This suggests that the cortex relies on an hourglass architecture to first integrate and compress multisensory information from multiple sensory regions, before utilizing that lower dimensionality representation in higher level association regions and more complex cognitive tasks.

Having a structural network representation of connectivity in the brain is instrumental in analyzing communication dynamics and neural information processing. In this work, we make steps towards understanding multisensory information flow and integration using a network diffusion approach. In particular, we model the flow of evoked activity, initiated by stimuli at primary sensory regions, using the asynchronous linear threshold (ALT) diffusion model. The ALT model captures how evoked activity that originates at a given region of the cortex “ripples through” other brain regions (referred to as an activation cascade). We apply the ALT model to the mouse connectome provided by the Allen Institute for Brain Science. A first result, using functional datasets based on voltage-sensitive dye (VSD) imaging, is that the ALT model, despite its simplicity, predicts the temporal ordering of each sensory activation cascade quite accurately. We further apply this model to study multisensory integration and find that a small number of brain regionsthe claustrum and the parietal temporal cortex being at the top of the listare involved in almost all cortical sensory streams. This suggests that the cortex relies on an hourglass architecture to first integrate and compress multisensory information from multiple sensory regions, before utilizing that lower dimensionality representation in higher level association regions and more complex cognitive tasks.

Non-spatial skills differ in the front and rear peri-personal space

In measuring behavioural and pupillary responses to auditory oddball stimuli delivered in the front and rear peri-personal space, we find that pupils dilate in response to rare stimuli, both target and distracters. Dilation in response to targets is stronger than the response to distracters, implying a task relevance effect on pupil responses. Crucially, pupil dilation in response to targets is also selectively modulated by the location of sound sources: stronger in the front than in the rear peri-personal space, in spite of matching behavioural performance. This supports the concept that even non-spatial skills, such as the ability to alert in response to behaviourally relevant events, are differentially engaged across subregions of the peri-personal space.

Audiovisual temporal integration: Cognitive processing, neural mechanisms, developmental trajectory and potential interventions

To integrate auditory and visual signals into a unified percept, the paired stimuli must co-occur within a limited time window known as the Temporal Binding Window (TBW). The width of the TBW, a proxy of audiovisual temporal integration ability, has been found to be correlated with higher-order cognitive and social functions. A comprehensive review of studies investigating audiovisual TBW reveals several findings: (1) a wide range of top-down processes and bottom-up features can modulate the width of the TBW, facilitating adaptation to the changing and multisensory external environment; (2) a large-scale brain network works in coordination to ensure successful detection of audiovisual (a)synchrony; (3) developmentally, audiovisual TBW follows a U-shaped pattern across the lifespan, with a protracted developmental course into late adolescence and rebounding in size again in late life; (4) an enlarged TBW is characteristic of a number of neurodevelopmental disorders; and (5) the TBW is highly flexible via perceptual and musical training. Interventions targeting the TBW may be able to improve multisensory function and ameliorate social communicative symptoms in clinical populations.

Peripersonal space is diversely sensitive to a temporary vs permanent state of anxiety

Peripersonal Space (PPS) is the multisensory space immediately surrounding our body. Visual and tactile stimuli here are promptly processed, since their interaction gradually strengthens as the distance between visual stimulus and the body decreases. Recently, a modified version of the Temporal Order Judgment (TOJ) task was proposed to assess PPS based on the spatial congruence between somatosensory and visual stimuli. Here, we used this paradigm to explore how a temporary vs a permanent state of anxiety can alter PPS. Indeed, previous research showed that PPS boundaries are not fixed, but they can be enlarged by contingent factors (i.e. emotional features). Participants performed the TOJ paradigm twice, just before and after completing an anxiety-inducing task (experimental breathing condition) or a neutral one (control breathing condition), while their trait and state anxiety levels were repeatedly measured. We found that the pattern of visuo-tactile integration in PPS changes in the very opposite way following the two breathing tasks for participants with high levels of temporary anxiety, by strengthening and weakening its power after the experimental and control conditions, respectively. On the contrary, both the breathing tasks are capable of reducing the cross-modal interplay as compared to baseline for high trait-anxious participants, who show an overall stronger visuo-tactile integration inside the PPS than low trait anxious individuals. These results are discussed in the light of the double dissociation between orienting and alerting attentional network over-functioning, reported in state anxiety participants, and impoverished prefrontal attentional control shown by trait anxiety individuals.

Increased Neural Strength and Reliability to Audiovisual Stimuli at the Boundary of Peripersonal Space

The actionable space surrounding the body, referred to as peripersonal space (PPS), has been the subject of significant interest of late within the broader framework of embodied cognition. Neurophysiological and neuroimaging studies have shown the representation of PPS to be built from visuotactile and audiotactile neurons within a frontoparietal network and whose activity is modulated by the presence of stimuli in proximity to the body. In contrast to single-unit and fMRI studies, an area of inquiry that has received little attention is the EEG characterization associated with PPS processing. Furthermore, although PPS is encoded by multisensory neurons, to date there has been no EEG study systematically examining neural responses to unisensory and multisensory stimuli, as these are presented outside, near, and within the boundary of PPS. Similarly, it remains poorly understood whether multisensory integration is generally more likely at certain spatial locations (e.g., near the body) or whether the cross-modal tactile facilitation that occurs within PPS is simply due to a reduction in the distance between sensory stimuli when close to the body and in line with the spatial principle of multisensory integration. In the current study, to examine the neural dynamics of multisensory processing within and beyond the PPS boundary, we present auditory, visual, and audiovisual stimuli at various distances relative to participants' reaching limit—an approximation of PPS—while recording continuous high-density EEG. We question whether multisensory (vs. unisensory) processing varies as a function of stimulus–observer distance. Results demonstrate a significant increase of global field power (i.e., overall strength of response across the entire electrode montage) for stimuli presented at the PPS boundary—an increase that is largest under multisensory (i.e., audiovisual) conditions. Source localization of the major contributors to this global field power difference suggests neural generators in the intraparietal sulcus and insular cortex, hubs for visuotactile and audiotactile PPS processing. Furthermore, when neural dynamics are examined in more detail, changes in the reliability of evoked potentials in centroparietal electrodes are predictive on a subject-by-subject basis of the later changes in estimated current strength at the intraparietal sulcus linked to stimulus proximity to the PPS boundary. Together, these results provide a previously unrealized view into the neural dynamics and temporal code associated with the encoding of nontactile multisensory around the PPS boundary.

Toward understanding the neurophysiological basis of peripersonal space: An EEG study on healthy individuals

The subcortical mechanisms subtending the sensorimotor processes related to the peripersonal space (PPS) have been well characterized, whereas less evidence is available concerning the cortical mechanisms. We investigated the theta, alpha and beta event-related spectral perturbations (ERSP) while holding the forearm in different positions into the PPS of the face. Fifty healthy individuals were subjected to EEG recording while being provided with median nerve electric stimulation at the wrist of the right hand held at different hand-to-face distances. Theta and beta rhythms were significantly perturbed depending on the hand-to-face distance, whereas alpha oscillations reflected a more general, non-specific oscillatory response to the motor task. The perturbation of theta and beta frequency bands may reflect the processes of top-down modulation overseeing the conscious spatiotemporal encoding of sensory-motor information within the PPS. In other words, such perturbation reflects the continuous update of the conscious internal representations of the PPS to build up a purposeful and reflexive motor response.

Spatial receptive field shift by preceding cross-modal stimulation in the cat superior colliculus

KEY POINTS

It has been known for some time that sensory information of one type can bias the spatial perception of another modality. However, there is a lack of evidence of this occurring in individual neurons. In the present study, we found that the spatial receptive field of superior colliculus multisensory neurons could be dynamically shifted by a preceding stimulus in a different modality. The extent to which the receptive field shifted was dependent on both temporal and spatial gaps between the preceding and following stimuli, as well as the salience of the preceding stimulus. This result provides a neural mechanism that could underlie the process of cross-modal spatial calibration.

ABSTRACT

Psychophysical studies have shown that the different senses can be spatially entrained by each other. This can be observed in certain phenomena, such as ventriloquism, in which a visual stimulus can attract the perceived location of a spatially discordant sound. However, the neural mechanism underlying this cross-modal spatial recalibration has remained unclear, as has whether it takes place dynamically. We explored these issues in multisensory neurons of the cat superior colliculus (SC), a midbrain structure that involves both cross-modal and sensorimotor integration. Sequential cross-modal stimulation showed that the preceding stimulus can shift the receptive field (RF) of the lagging response. This cross-modal spatial calibration took place in both auditory and visual RFs, although auditory RFs shifted slightly more. By contrast, if a preceding stimulus was from the same modality, it failed to induce a similarly substantial RF shift. The extent of the RF shift was dependent on both temporal and spatial gaps between the preceding and following stimuli, as well as the salience of the preceding stimulus. A narrow time gap and high stimulus salience were able to induce larger RF shifts. In addition, when both visual and auditory stimuli were presented simultaneously, a substantial RF shift toward the location-fixed stimulus was also induced. These results, taken together, reveal an online cross-modal process and reflect the details of the organization of SC inter-sensory spatial calibration.

Frontier of Self and Impact Prediction

The construction of a coherent representation of our body and the mapping of the space immediately surrounding it are of the highest ecological importance. This space has at least three specificities: it is a space where actions are planned in order to interact with our environment; it is a space that contributes to the experience of self and self-boundaries, through tactile processing and multisensory interactions; last, it is a space that contributes to the experience of body integrity against external events. In the last decades, numerous studies have been interested in peripersonal space (PPS), defined as the space directly surrounding us and which we can interact with (for reviews, see Cléry et al., 2015b; de Vignemont and Iannetti, 2015; di Pellegrino and Làdavas, 2015). These studies have contributed to the understanding of how this space is constructed, encoded and modulated. The majority of these studies focused on subparts of PPS (the hand, the face or the trunk) and very few of them investigated the interaction between PPS subparts. In the present review, we summarize the latest advances in this research and we discuss the new perspectives that are set forth for futures investigations on this topic. We describe the most recent methods used to estimate PPS boundaries by the means of dynamic stimuli. We then highlight how impact prediction and approaching stimuli modulate this space by social, emotional and action-related components involving principally a parieto-frontal network. In a next step, we review evidence that there is not a unique representation of PPS but at least three sub-sections (hand, face and trunk PPS). Last, we discuss how these subspaces interact, and we question whether and how bodily self-consciousness (BSC) is functionally and behaviorally linked to PPS.

Different audio spatial metric representation around the body

Vision seems to have a pivotal role in developing spatial cognition. A recent approach, based on sensory calibration, has highlighted the role of vision in calibrating hearing in spatial tasks. It was shown that blind individuals have specific impairments during audio spatial bisection tasks. Vision is available only in the frontal space, leading to a "natural" blindness in the back. If vision is important for audio space calibration, then the auditory frontal space should be better represented than the back auditory space. In this study, we investigated this point by comparing frontal and back audio spatial metric representations. We measured precision in the spatial bisection task, for which vision seems to be fundamental to calibrate audition, in twenty-three sighted subjects. Two control tasks, a minimum audible angle and a temporal bisection were employed in order to evaluate auditory precision in the different regions considered. While no differences were observed between frontal and back space in the minimum audible angle (MAA) and temporal bisection task, a significant difference was found in the spatial bisection task, where subjects performed better in the frontal space. Our results are in agreement with the idea that vision is important in developing auditory spatial metric representation in sighted individuals.

Atypical audiovisual temporal function in autism and schizophrenia: similar phenotype, different cause

Binding across sensory modalities yields substantial perceptual benefits, including enhanced speech intelligibility. The coincidence of sensory inputs across time is a fundamental cue for this integration process. Recent work has suggested that individuals with diagnoses of schizophrenia (SZ) and autism spectrum disorder (ASD) will characterize auditory and visual events as synchronous over larger temporal disparities than their neurotypical counterparts. Namely, these clinical populations possess an enlarged temporal binding window (TBW). Although patients with SZ and ASD share aspects of their symptomatology, phenotypic similarities may result from distinct etiologies. To examine similarities and variances in audiovisual temporal function in these two populations, individuals diagnosed with ASD (n = 46; controls n = 40) and SZ (n = 16, controls = 16) completed an audiovisual simultaneity judgment task. In addition to standard psychometric analyses, synchrony judgments were assessed using Bayesian causal inference modeling. This approach permits distinguishing between distinct causes of an enlarged TBW: an a priori bias to bind sensory information and poor fidelity in the sensory representation. Findings indicate that both ASD and SZ populations show deficits in multisensory temporal acuity. Importantly, results suggest that while the wider TBWs in ASD most prominently results from atypical priors, the wider TBWs in SZ results from a trend toward changes in prior and weaknesses in the sensory representations. Results are discussed in light of current ASD and SZ theories and highlight that different perceptual training paradigms focused on improving multisensory integration may be most effective in these two clinical populations and emphasize that similar phenotypes may emanate from distinct mechanistic causes.

Audiovisual integration in depth: multisensory binding and gain as a function of distance

The integration of information across sensory modalities is dependent on the spatiotemporal characteristics of the stimuli that are paired. Despite large variation in the distance over which events occur in our environment, relatively little is known regarding how stimulus-observer distance affects multisensory integration. Prior work has suggested that exteroceptive stimuli are integrated over larger temporal intervals in near relative to far space, and that larger multisensory facilitations are evident in far relative to near space. Here, we sought to examine the interrelationship between these previously established distance-related features of multisensory processing. Participants performed an audiovisual simultaneity judgment and redundant target task in near and far space, while audiovisual stimuli were presented at a range of temporal delays (i.e., stimulus onset asynchronies). In line with the previous findings, temporal acuity was poorer in near relative to far space. Furthermore, reaction time to asynchronously presented audiovisual targets suggested a temporal window for fast detection-a range of stimuli asynchronies that was also larger in near as compared to far space. However, the range of reaction times over which multisensory response enhancement was observed was limited to a restricted range of relatively small (i.e., 150 ms) asynchronies, and did not differ significantly between near and far space. Furthermore, for synchronous presentations, these distance-related (i.e., near vs. far) modulations in temporal acuity and multisensory gain correlated negatively at an individual subject level. Thus, the findings support the conclusion that multisensory temporal binding and gain are asymmetrically modulated as a function of distance from the observer, and specifies that this relationship is specific for temporally synchronous audiovisual stimulus presentations.

Long-term music training modulates the recalibration of audiovisual simultaneity

To overcome differences in physical transmission time and neural processing, the brain adaptively recalibrates the point of simultaneity between auditory and visual signals by adapting to audiovisual asynchronies. Here, we examine whether the prolonged recalibration process of passively sensed visual and auditory signals is affected by naturally occurring multisensory training known to enhance audiovisual perceptual accuracy. Hence, we asked a group of drummers, of non-drummer musicians and of non-musicians to judge the audiovisual simultaneity of musical and non-musical audiovisual events, before and after adaptation with two fixed audiovisual asynchronies. We found that the recalibration for the musicians and drummers was in the opposite direction (sound leading vision) to that of non-musicians (vision leading sound), and change together with both increased music training and increased perceptual accuracy (i.e. ability to detect asynchrony). Our findings demonstrate that long-term musical training reshapes the way humans adaptively recalibrate simultaneity between auditory and visual signals.

Consciousness, Representation, Action: The Importance of Being Goal-Directed

Recent years have witnessed fierce debates on the dependence of consciousness on interactions between a subject and the environment. Reviewing neuroscientific, computational, and clinical evidence, I will address three questions. First, does conscious experience necessarily depend on acute interactions between a subject and the environment? Second, does it depend on specific perception-action loops in the longer run? Third, which types of action does consciousness cohere with, if not with all of them? I argue that conscious contents do not necessarily depend on acute or long-term brain-environment interactions. Instead, consciousness is proposed to be specifically associated with, and subserve, deliberate, goal-directed behavior (GDB). Brain systems implied in conscious representation are highly connected to, but distinct from, neural substrates mediating GDB and declarative memory.

Disrupted integration of exteroceptive and interoceptive signaling in autism spectrum disorder

In addition to deficits in social communication, individuals diagnosed with Autism Spectrum Disorder (ASD) frequently exhibit changes in sensory and multisensory function. Recent evidence has focused on changes in audiovisual temporal processing, and has sought to relate these sensory-based changes to weaknesses in social communication. These changes in audiovisual temporal function manifest as differences in the temporal epoch or "window" within which paired auditory and visual stimuli are integrated or bound, with those with ASD exhibiting expanded audiovisual temporal binding windows (TBWs). However, it is unknown whether this impairment is unique to audiovisual pairings, perhaps because of their relevance for speech processing, or whether it generalizes across pairings in different sensory modalities. In addition to the exteroceptive senses, there has been growing interest in ASD research in interoception (e.g., the monitoring of respiration, heartbeat, hunger, etc.), as these internally directed sensory processes appear to be altered as well in autism. In the current study, we sought to examine both exteroception and interoception in individuals with ASD and a group of typically developing (TD) matched controls, with an emphasis on temporal perception of audiovisual (exteroceptive) and cardiovisual (interoceptive to exteroceptive) cues. Results replicate prior findings showing expanded audiovisual TBWs in ASD in comparison to TD. In addition, strikingly, cardiovisual TBWs were fourfold larger in ASD than in TD, suggesting a putative complete lack of cardiovisual temporal acuity in ASD individuals. Results are discussed in light of recent evidence indicating a reduced tendency to rely on sensory priors in ASD. Autism Res 2018, 11: 194-205. © 2017 International Society for Autism Research, Wiley Periodicals, Inc.

LAY SUMMARY

Studies have shown that individuals with autism have difficulty in separating auditory and visual events in time. People with autism also weight sensory evidence originating from the external world and from their body differently. We measured simultaneity judgments regarding visual and auditory events and between visual and heartbeat events. Results suggest that while individuals with autism show unusual temporal function across the senses in a general manner, this deficit is greater when pairings bridged between the external world and the internal body.

Voluntary and Involuntary Movements Widen the Window of Subjective Simultaneity

Forming a coherent percept of an event requires different sensory inputs originating from the event to be bound. Perceiving synchrony aids in binding of these inputs. In two experiments, we investigated how voluntary movements influence the perception of simultaneity, by measuring simultaneity judgments (SJs) for an audiovisual (AV) stimulus pair triggered by a voluntary button press. In Experiment 1, we manipulated contiguity between the action and its consequences by introducing delays between the button press and the AV stimulus pair. We found a widened window of subjective simultaneity (WSS) when the action-feedback relationship was time contiguous. Introducing a delay narrowed the WSS, suggesting that the wider WSS around the time of an action might facilitate perception of simultaneity. In Experiment 2, we introduced an involuntary condition using an externally controlled button to assess the influence of action-related predictive processes on SJs. We found a widened WSS around the action time, regardless of movement type, supporting the influence of causal relations in the perception of synchrony. Interestingly, the slopes of the psychometric functions in the voluntary condition were significantly steeper than the slopes in the involuntary condition, suggesting a role of action-related predictive mechanisms in making SJs more precise.

On the contribution of overt tactile expectations to visuo-tactile interactions within the peripersonal space

Since the discovery of neural regions in the monkey brain that respond preferentially to multisensory stimuli presented in proximal space, researchers have been studying this specialised spatial representation in humans. It has been demonstrated that approaching auditory or visual stimuli modulate tactile processing, while they are within the peripersonal space (PPS). The aim of the current study is to investigate the additional effects of tactile expectation on the PPS-related multisensory interactions. Based on the output of a computational simulation, we expected that as tactile expectation increases rapidly during the course of the motion of the visual stimulus, the outcome RT curves would mask the multisensory contribution of PPS. When the tactile expectation remains constant during the motion, the PPS-related spatially selective multisensory processes become apparent. The behavioural results on human experiments followed the pattern predicted by the simulation. That is, rapidly changing levels of tactile expectation, caused by dynamic visual stimuli, masks the outcome of the multisensory processes within peripersonal space. This indicates that both PPS-related multisensory interactions and tactile expectations play an important role in anticipating and responding to interactions with the body.

Multisensory temporal function and EEG complexity in patients with epilepsy and psychogenic nonepileptic events

Cognitive and perceptual comorbidities frequently accompany epilepsy and psychogenic nonepileptic events (PNEE). However, and despite the fact that perceptual function is built upon a multisensory foundation, little knowledge exists concerning multisensory function in these populations. Here, we characterized facets of multisensory processing abilities in patients with epilepsy and PNEE, and probed the relationship between individual resting-state EEG complexity and these psychophysical measures in each patient. We prospectively studied a cohort of patients with epilepsy (N=18) and PNEE (N=20) patients who were admitted to Vanderbilt's Epilepsy Monitoring Unit (EMU) and weaned off of anticonvulsant drugs. Unaffected age-matched persons staying with the patients in the EMU (N=15) were also recruited as controls. All participants performed two tests of multisensory function: an audio-visual simultaneity judgment and an audio-visual redundant target task. Further, in the cohort of patients with epilepsy and PNEE we quantified resting state EEG gamma power and complexity. Compared with both patients with epilepsy and control subjects, patients with PNEE exhibited significantly poorer acuity in audiovisual temporal function as evidenced in significantly larger temporal binding windows (i.e., they perceived larger stimulus asynchronies as being presented simultaneously). These differences appeared to be specific for temporal function, as there was no difference among the three groups in a non-temporally based measure of multisensory function - the redundant target task. Further, patients with PNEE exhibited more complex resting state EEG patterns as compared to their patients with epilepsy, and EEG complexity correlated with multisensory temporal performance on a subject-by-subject manner. Taken together, findings seem to indicate that patients with PNEE bind information from audition and vision over larger temporal intervals when compared with control subjects as well as patients with epilepsy. This difference in multisensory function appears to be specific to the temporal domain, and may be a contributing factor to the behavioral and perceptual alterations seen in this population.

Event Related Potentials Index Rapid Recalibration to Audiovisual Temporal Asynchrony

Asynchronous arrival of multisensory information at the periphery is a ubiquitous property of signals in the natural environment due to differences in the propagation time of light and sound. Rapid adaptation to these asynchronies is crucial for the appropriate integration of these multisensory signals, which in turn is a fundamental neurobiological process in creating a coherent perceptual representation of our dynamic world. Indeed, multisensory temporal recalibration has been shown to occur at the single trial level, yet the mechanistic basis of this rapid adaptation is unknown. Here, we investigated the neural basis of rapid recalibration to audiovisual temporal asynchrony in human participants using a combination of psychophysics and electroencephalography (EEG). Consistent with previous reports, participant's perception of audiovisual temporal synchrony on a given trial (t) was influenced by the temporal structure of stimuli on the previous trial (t-1). When examined physiologically, event related potentials (ERPs) were found to be modulated by the temporal structure of the previous trial, manifesting as late differences (>125 ms post second-stimulus onset) in central and parietal positivity on trials with large stimulus onset asynchronies (SOAs). These findings indicate that single trial adaptation to audiovisual temporal asynchrony is reflected in modulations of late evoked components that have previously been linked to stimulus evaluation and decision-making.

The Impact of Feedback on the Different Time Courses of Multisensory Temporal Recalibration

The capacity to rapidly adjust perceptual representations confers a fundamental advantage when confronted with a constantly changing world. Unexplored is how feedback regarding sensory judgments (top-down factors) interacts with sensory statistics (bottom-up factors) to drive long- and short-term recalibration of multisensory perceptual representations. Here, we examined the time course of both cumulative and rapid temporal perceptual recalibration for individuals completing an audiovisual simultaneity judgment task in which they were provided with varying degrees of feedback. We find that in the presence of feedback (as opposed to simple sensory exposure) temporal recalibration is more robust. Additionally, differential time courses are seen for cumulative and rapid recalibration dependent upon the nature of the feedback provided. Whereas cumulative recalibration effects relied more heavily on feedback that informs (i.e., negative feedback) rather than confirms (i.e., positive feedback) the judgment, rapid recalibration shows the opposite tendency. Furthermore, differential effects on rapid and cumulative recalibration were seen when the reliability of feedback was altered. Collectively, our findings illustrate that feedback signals promote and sustain audiovisual recalibration over the course of cumulative learning and enhance rapid trial-to-trial learning. Furthermore, given the differential effects seen for cumulative and rapid recalibration, these processes may function via distinct mechanisms.

Atypical rapid audio-visual temporal recalibration in autism spectrum disorders

Changes in sensory and multisensory function are increasingly recognized as a common phenotypic characteristic of Autism Spectrum Disorders (ASD). Furthermore, much recent evidence suggests that sensory disturbances likely play an important role in contributing to social communication weaknesses-one of the core diagnostic features of ASD. An established sensory disturbance observed in ASD is reduced audiovisual temporal acuity. In the current study, we substantially extend these explorations of multisensory temporal function within the framework that an inability to rapidly recalibrate to changes in audiovisual temporal relations may play an important and under-recognized role in ASD. In the paradigm, we present ASD and typically developing (TD) children and adolescents with asynchronous audiovisual stimuli of varying levels of complexity and ask them to perform a simultaneity judgment (SJ). In the critical analysis, we test audiovisual temporal processing on trial t as a condition of trial t - 1. The results demonstrate that individuals with ASD fail to rapidly recalibrate to audiovisual asynchronies in an equivalent manner to their TD counterparts for simple and non-linguistic stimuli (i.e., flashes and beeps, hand-held tools), but exhibit comparable rapid recalibration for speech stimuli. These results are discussed in terms of prior work showing a speech-specific deficit in audiovisual temporal function in ASD, and in light of current theories of autism focusing on sensory noise and stability of perceptual representations. Autism Res 2017, 10: 121-129. © 2016 International Society for Autism Research, Wiley Periodicals, Inc.

The spatial self in schizophrenia and autism spectrum disorder

Schizophrenia (SZ) and autism spectrum disorder (ASD) have been both described as disorders of the self. However, the manner in which the sense of self is impacted in these disorders is strikingly different. In the current review, we propose that SZ and ASD lay at opposite extremes of a particular component of the representation of self; namely, self-location and the construct of peripersonal space. We evaluate emerging literature suggesting that while SZ individuals possess an extremely weak or variable bodily boundary between self and other, ASD patients possess a sharper self-other boundary. Furthermore, based on recent behavioral and neural network modeling findings, we propose that multisensory training focused on either sharpening (for SZ) or making shallower (for ASD) the self-other boundary may hold promise as an interventional tool in the treatment of these disorders.

Audiovisual Simultaneity Judgment and Rapid Recalibration throughout the Lifespan

Multisensory interactions are well established to convey an array of perceptual and behavioral benefits. One of the key features of multisensory interactions is the temporal structure of the stimuli combined. In an effort to better characterize how temporal factors influence multisensory interactions across the lifespan, we examined audiovisual simultaneity judgment and the degree of rapid recalibration to paired audiovisual stimuli (Flash-Beep and Speech) in a sample of 220 participants ranging from 7 to 86 years of age. Results demonstrate a surprisingly protracted developmental time-course for both audiovisual simultaneity judgment and rapid recalibration, with neither reaching maturity until well into adolescence. Interestingly, correlational analyses revealed that audiovisual simultaneity judgments (i.e., the size of the audiovisual temporal window of simultaneity) and rapid recalibration significantly co-varied as a function of age. Together, our results represent the most complete description of age-related changes in audiovisual simultaneity judgments to date, as well as being the first to describe changes in the degree of rapid recalibration as a function of age. We propose that the developmental time-course of rapid recalibration scaffolds the maturation of more durable audiovisual temporal representations.

Multisensory perceptual learning is dependent upon task difficulty

There has been a growing interest in developing behavioral tasks to enhance temporal acuity as recent findings have demonstrated changes in temporal processing in a number of clinical conditions. Prior research has demonstrated that perceptual training can enhance temporal acuity both within and across different sensory modalities. Although certain forms of unisensory perceptual learning have been shown to be dependent upon task difficulty, this relationship has not been explored for multisensory learning. The present study sought to determine the effects of task difficulty on multisensory perceptual learning. Prior to and following a single training session, participants completed a simultaneity judgment (SJ) task, which required them to judge whether a visual stimulus (flash) and auditory stimulus (beep) presented in synchrony or at various stimulus onset asynchronies (SOAs) occurred synchronously or asynchronously. During the training session, participants completed the same SJ task but received feedback regarding the accuracy of their responses. Participants were randomly assigned to one of three levels of difficulty during training: easy, moderate, and hard, which were distinguished based on the SOAs used during training. We report that only the most difficult (i.e., hard) training protocol enhanced temporal acuity. We conclude that perceptual training protocols for enhancing multisensory temporal acuity may be optimized by employing audiovisual stimuli for which it is difficult to discriminate temporal synchrony from asynchrony.