Keeping in touch with the visual system: spatial alignment and multisensory integration of visual-somatosensory inputs

ERPs evidence of multisensory integration deficits in spinal cord injury

Spinal cord injury (SCI) is associated with deficits in multisensory integration-the ability to synthesize cross-modal information. This study explores the neural mechanisms underlying these deficits using EEG and a detection task incorporating unisensory and multisensory stimuli: audio-visual, visuo-tactile, and audio-tactile. Behaviorally, participants with SCI showed reduced multisensory integration across all modalities, consistent with prior findings. Neurally, ERPs were analyzed in three conditions: audio-tactile (N100, P200), visuo-tactile (P170), and audio-visual (P100, N200). Higher ERP amplitudes for multisensory versus unisensory stimuli were only observed in the control group, whereas the SCI group showed similar amplitudes across both. In the SCI group, multisensory ERPs were significantly lower for audio-tactile P200, visuo-tactile P170, and audio-visual P100, indicating a deficit in multisensory processing. Auditory ERPs were preserved in SCI participants, while visual and tactile responses were reduced, suggesting an auditory dominance post-SCI. Cluster-based analysis on residual effects showed that the control group exhibited greater multisensory gain compared to SCI participants, with significant centro-parietal clusters observed for audio-tactile (50-100 ms, 120-180 ms, 300-500 ms), visuo-tactile (80-120 ms, 120-180 ms), and audio-visual (280-480 ms) residual effects. Overall, these results highlight that SCI has detrimental effects not only on the motor system, but also on the ability to process multisensory information. This study advances our understanding of multisensory integration mechanisms following sensorimotor deficits and highlights the need for targeted interventions to address multisensory impairments in this population.

Adults on the autism spectrum differ from neurotypical peers when self-generating but not passively-experiencing somatosensation: a high-density electrophysiological (EEG) mapping and virtual reality study

Little is known about how different features of tactile inputs affect somatosensory perception in autism. In this study we combined high-density electroencephalography (EEG) and virtual reality (VR) to assess how the volition and pattern consistency of somatosensory stimulation influenced the electrophysiological responses in neurotypical (n = 30) and autistic (n = 30) adults. Specifically, we compared N1 and P300 amplitudes when vibrotactile stimulation were actively triggered by self-motion (Active) versus passively triggered by target-motion (Passive). We also measured the mismatch negativity (MMN) to assess how deviations in the pattern of stimulus duration affected the electrophysiological responses. We observed comparable responses regardless of pattern deviation in the MMN time window between groups, but different patterns of amplitude in this time frame based on whether the stimulation was Active or Passive. In the autism group we observed smaller N1 amplitudes in response to Passive, but not Active, vibrations as compared to the control group. Conversely, there were overall larger magnitude P300 amplitudes in the autism group, but comparable levels of Passive-to-Active attenuation between groups. Overall, the autism cohort demonstrated variation from the neurotypical cohort with respect to the volition of the stimuli, but there were comparable results between groups in response to pattern deviation. These findings suggest that there are subtle differences in how adults with and without autism handle self-generated and externally-generated somatosensory sensations.

The Feel of Speech: Multisystem and Polymodal Somatosensation in Speech Production

PURPOSE

The oral structures such as the tongue and lips have remarkable somatosensory capacities, but understanding the roles of somatosensation in speech production requires a more comprehensive knowledge of somatosensation in the speech production system in its entirety, including the respiratory, laryngeal, and supralaryngeal subsystems. This review was conducted to summarize the system-wide somatosensory information available for speech production.

METHOD

The search was conducted with PubMed/Medline and Google Scholar for articles published until November 2023. Numerous search terms were used in conducting the review, which covered the topics of psychophysics, basic and clinical behavioral research, neuroanatomy, and neuroscience.

RESULTS AND CONCLUSIONS

The current understanding of speech somatosensation rests primarily on the two pillars of psychophysics and neuroscience. The confluence of polymodal afferent streams supports the development, maintenance, and refinement of speech production. Receptors are both canonical and noncanonical, with the latter occurring especially in the muscles innervated by the facial nerve. Somatosensory representation in the cortex is disproportionately large and provides for sensory interactions. Speech somatosensory function is robust over the lifespan, with possible declines in advanced aging. The understanding of somatosensation in speech disorders is largely disconnected from research and theory on speech production. A speech somatoscape is proposed as the generalized, system-wide sensation of speech production, with implications for speech development, speech motor control, and speech disorders.

A Spatiotemporal and Multisensory Approach to Designing Wearable Clinical ICU Alarms

In health care, auditory alarms are an important aspect of an informatics system that monitors patients and alerts clinicians attending to multiple concurrent tasks. However, the volume, design, and pervasiveness of existing Intensive Care Unit (ICU) alarms can make it difficult to quickly distinguish their meaning and importance. In this study, we evaluated the effectiveness of two design approaches not yet explored in a smartwatch-based alarm system designed for ICU use: (1) using audiovisual spatial colocalization and (2) adding haptic (i.e., touch) information. We compared the performance of 30 study participants using ICU smartwatch alarms containing auditory icons in two implementations of the audio modality: colocalized with the visual cue on the smartwatch's low-quality speaker versus delivered from a higher quality speaker located two feet away from participants (like a stationary alarm bay situated near patients in the ICU). Additionally, we compared participant performance using alarms with two sensory modalities (visual and audio) against alarms with three sensory modalities (adding haptic cues). Participants were 10.1% (0.24s) faster at responding to alarms when auditory information was delivered from the smartwatch instead of the higher quality external speaker. Meanwhile, adding haptic information to alarms improved response times to alarms by 12.2% (0.23s) and response times on their primary task by 10.3% (0.08s). Participants rated learnability and ease of use higher for alarms with haptic information. These small but statistically significant improvements demonstrate that audiovisual colocalization and multisensory alarm design can improve user response times.

Hierarchical unimodal processing within the primary somatosensory cortex during a bimodal detection task

Do sensory cortices process more than one sensory modality? To answer these questions, scientists have generated a wide variety of studies at distinct space-time scales in different animal models, and often shown contradictory conclusions. Some conclude that this process occurs in early sensory cortices, but others that this occurs in areas central to sensory cortices. Here, we sought to determine whether sensory neurons process and encode physical stimulus properties of different modalities (tactile and acoustic). For this, we designed a bimodal detection task where the senses of touch and hearing compete from trial to trial. Two Rhesus monkeys performed this novel task, while neural activity was recorded in areas 3b and 1 of the primary somatosensory cortex (S1). We analyzed neurons' coding properties and variability, organizing them by their receptive field's position relative to the stimulation zone. Our results indicate that neurons of areas 3b and 1 are unimodal, encoding only the tactile modality in both the firing rate and variability. Moreover, we found that neurons in area 3b carried more information about the periodic stimulus structure than those in area 1, possessed lower response and coding latencies, and had a lower intrinsic time scale. In sum, these differences reveal a hidden processing-based hierarchy. Finally, using a powerful nonlinear dimensionality reduction algorithm, we show that the activity from areas 3b and 1 can be separated, establishing a clear division in the functionality of these two subareas of S1.

Spatial proximity to others induces plastic changes in the neural representation of the peripersonal space

Peripersonal space (PPS) is a highly plastic "invisible bubble" surrounding the body whose boundaries are mapped through multisensory integration. Yet, it is unclear how the spatial proximity to others alters PPS boundaries. Across five experiments (N = 80), by recording behavioral and electrophysiological responses to visuo-tactile stimuli, we demonstrate that the proximity to others induces plastic changes in the neural PPS representation. The spatial proximity to someone else's hand shrinks the portion of space within which multisensory responses occur, thus reducing the PPS boundaries. This suggests that PPS representation, built from bodily and multisensory signals, plastically adapts to the presence of conspecifics to define the self-other boundaries, so that what is usually coded as "my space" is recoded as "your space". When the space is shared with conspecifics, it seems adaptive to move the other-space away from the self-space to discriminate whether external events pertain to the self-body or to other-bodies.

Multisensory-driven facilitation within the peripersonal space is modulated by the expectations about stimulus location on the body

Compelling evidence from human and non-human studies suggests that responses to multisensory events are fastened when stimuli occur within the space surrounding the bodily self (i.e., peripersonal space; PPS). However, some human studies did not find such effect. We propose that these dissonant voices might actually uncover a specific mechanism, modulating PPS boundaries according to sensory regularities. We exploited a visuo-tactile paradigm, wherein participants provided speeded responses to tactile stimuli and rated their perceived intensity while ignoring simultaneous visual stimuli, appearing near the stimulated hand (VTNear) or far from it (VTFar; near the non-stimulated hand). Tactile stimuli could be delivered only to one hand (unilateral task) or to both hands randomly (bilateral task). Results revealed that a space-dependent multisensory enhancement (i.e., faster responses and higher perceived intensity in VTNear than VTFar) was present when highly predictable tactile stimulation induced PPS to be circumscribed around the stimulated hand (unilateral task). Conversely, when stimulus location was unpredictable (bilateral task), participants showed a comparable multisensory enhancement in both bimodal conditions, suggesting a PPS widening to include both hands. We propose that the detection of environmental regularities actively shapes PPS boundaries, thus optimizing the detection and reaction to incoming sensory stimuli.

Resolution of impaired multisensory processing in autism and the cost of switching sensory modality

Children with autism spectrum disorders (ASD) exhibit alterations in multisensory processing, which may contribute to the prevalence of social and communicative deficits in this population. Resolution of multisensory deficits has been observed in teenagers with ASD for complex, social speech stimuli; however, whether this resolution extends to more basic multisensory processing deficits remains unclear. Here, in a cohort of 364 participants we show using simple, non-social audiovisual stimuli that deficits in multisensory processing observed in high-functioning children and teenagers with ASD are not evident in adults with the disorder. Computational modelling indicated that multisensory processing transitions from a default state of competition to one of facilitation, and that this transition is delayed in ASD. Further analysis revealed group differences in how sensory channels are weighted, and how this is impacted by preceding cross-sensory inputs. Our findings indicate that there is a complex and dynamic interplay among the sensory systems that differs considerably in individuals with ASD.

Crosse et al. study a cohort of 364 participants with autism spectrum disorders (ASD) and matched controls, and show that deficits in multisensory processing observed in high-functioning children and teenagers with ASD are not evident in adults with the disorder. Using computational modelling they go on to demonstrate that there is a delayed transition of multisensory processing from a default state of competition to one of facilitation in ASD, as well as differences in sensory weighting and the ability to switch between sensory modalities, which sheds light on the interplay among sensory systems that differ in ASD individuals.

Retinal innervation tunes circuits that drive nonphotic entrainment to food

Daily changes in light and food availability are major time cues that influence circadian timing¹. However, little is known about the circuits that integrate these time cues to drive a coherent circadian output^1-3. Here we investigate whether retinal inputs modulate entrainment to nonphotic cues such as time-restricted feeding. Photic information is relayed to the suprachiasmatic nucleus (SCN)-the central circadian pacemaker-and the intergeniculate leaflet (IGL) through intrinsically photosensitive retinal ganglion cells (ipRGCs)⁴. We show that adult mice that lack ipRGCs from the early postnatal stages have impaired entrainment to time-restricted feeding, whereas ablation of ipRGCs at later stages had no effect. Innervation of ipRGCs at early postnatal stages influences IGL neurons that express neuropeptide Y (NPY) (hereafter, IGL^NPY neurons), guiding the assembly of a functional IGL^NPY-SCN circuit. Moreover, silencing IGL^NPY neurons in adult mice mimicked the deficits that were induced by ablation of ipRGCs in the early postnatal stages, and acute inhibition of IGL^NPY terminals in the SCN decreased food-anticipatory activity. Thus, innervation of ipRGCs in the early postnatal period tunes the IGL^NPY-SCN circuit to allow entrainment to time-restricted feeding.

Operating in a Multisensory Context: Assessing the Interplay Between Multisensory Reaction Time Facilitation and Inter-sensory Task-switching Effects

Individuals respond faster to presentations of bisensory stimuli (e.g. audio-visual targets) than to presentations of either unisensory constituent in isolation (i.e. to the auditory-alone or visual-alone components of an audio-visual stimulus). This well-established multisensory speeding effect, termed the redundant signals effect (RSE), is not predicted by simple linear summation of the unisensory response time probability distributions. Rather, the speeding is typically faster than this prediction, leading researchers to ascribe the RSE to a so-called co-activation account. According to this account, multisensory neural processing occurs whereby the unisensory inputs are integrated to produce more effective sensory-motor activation. However, the typical paradigm used to test for RSE involves random sequencing of unisensory and bisensory inputs in a mixed design, raising the possibility of an alternate attention-switching account. This intermixed design requires participants to switch between sensory modalities on many task trials (e.g. from responding to a visual stimulus to an auditory stimulus). Here we show that much, if not all, of the RSE under this paradigm can be attributed to slowing of reaction times to unisensory stimuli resulting from modality switching, and is not in fact due to speeding of responses to AV stimuli. As such, the present data do not support a co-activation account, but rather suggest that switching and mixing costs akin to those observed during classic task-switching paradigms account for the observed RSE.

Multisensory Audiovisual Processing in Children With a Sensory Processing Disorder (II): Speech Integration Under Noisy Environmental Conditions

Background: There exists a cohort of children and adults who exhibit an inordinately high degree of discomfort when experiencing what would be considered moderate and manageable levels of sensory input. That is, they show over-responsivity in the face of entirely typical sound, light, touch, taste, or smell inputs, and this occurs to such an extent that it interferes with their daily functioning and reaches clinical levels of dysfunction. What marks these individuals apart is that this sensory processing disorder (SPD) is observed in the absence of other symptom clusters that would result in a diagnosis of Autism, ADHD, or other neurodevelopmental disorders more typically associated with sensory processing difficulties. One major theory forwarded to account for these SPDs posits a deficit in multisensory integration, such that the various sensory inputs are not appropriately integrated into the central nervous system, leading to an overwhelming sensory-perceptual environment, and in turn to the sensory-defensive phenotype observed in these individuals. Methods: We tested whether children (6-16 years) with an over-responsive SPD phenotype (N = 12) integrated multisensory speech differently from age-matched typically-developing controls (TD: N = 12). Participants identified monosyllabic words while background noise level and sensory modality (auditory-alone, visual-alone, audiovisual) were varied in pseudorandom order. Improved word identification when speech was both seen and heard compared to when it was simply heard served to index multisensory speech integration. Results: School-aged children with an SPD show a deficit in the ability to benefit from the combination of both seen and heard speech inputs under noisy environmental conditions, suggesting that these children do not benefit from multisensory integrative processing to the same extent as their typically developing peers. In contrast, auditory-alone performance did not differ between the groups, signifying that this multisensory deficit is not simply due to impaired processing of auditory speech. Conclusions: Children with an over-responsive SPD show a substantial reduction in their ability to benefit from complementary audiovisual speech, to enhance speech perception in a noisy environment. This has clear implications for performance in the classroom and other learning environments. Impaired multisensory integration may contribute to sensory over-reactivity that is the definitional of SPD.

Tool use modulates early stages of visuo-tactile integration in far space: Evidence from event-related potentials

The neural representation of multisensory space near the body is modulated by the active use of long tools in non-human primates. Here, we investigated whether the electrophysiological correlates of visuo-tactile integration in near and far space were modulated by active tool use in healthy humans. Participants responded to a tactile target delivered to one hand while an irrelevant visual stimulus was presented ipsilaterally in near or far space. This crossmodal task was performed after the use of either short or long tools. Crucially, the P100 components elicited by visuo-tactile stimuli was enhanced on far as compared to near space trials after the use of long tools, while no such difference was present after short tool use. Thus, we found increased neural responses in brain areas encoding tactile stimuli to the body when visual stimuli were presented close to the tip of the tool after long tool use. This increased visuo-tactile integration on far space trials following the use of long tools might indicate a transient remapping of multisensory space. We speculate that performing voluntary actions with long tools strengthens the representation of sensory information arising within portions of space (i.e. the hand and the tip of the tool) that are most functionally relevant to one's behavioural goals.

Stimulus intensity modulates multisensory temporal processing

One of the more challenging feats that multisensory systems must perform is to determine which sensory signals originate from the same external event, and thus should be integrated or "bound" into a singular perceptual object or event, and which signals should be segregated. Two important stimulus properties impacting this process are the timing and effectiveness of the paired stimuli. It has been well established that the more temporally aligned two stimuli are, the greater the degree to which they influence one another's processing. In addition, the less effective the individual unisensory stimuli are in eliciting a response, the greater the benefit when they are combined. However, the interaction between stimulus timing and stimulus effectiveness in driving multisensory-mediated behaviors has never been explored - which was the purpose of the current study. Participants were presented with either high- or low-intensity audiovisual stimuli in which stimulus onset asynchronies (SOAs) were parametrically varied, and were asked to report on the perceived synchrony/asynchrony of the paired stimuli. Our results revealed an interaction between the temporal relationship (SOA) and intensity of the stimuli. Specifically, individuals were more tolerant of larger temporal offsets (i.e., more likely to call them synchronous) when the paired stimuli were less effective. This interaction was also seen in response time (RT) distributions. Behavioral gains in RTs were seen with synchronous relative to asynchronous presentations, but this effect was more pronounced with high-intensity stimuli. These data suggest that stimulus effectiveness plays an underappreciated role in the perception of the timing of multisensory events, and reinforces the interdependency of the principles of multisensory integration in determining behavior and shaping perception.

Visual-somatotopic interactions in spatial perception

Ventriloquism is a well-studied multisensory illusion of audiovisual spatial perception in which the perceived location of an auditory stimulus is shifted in the direction of a synchronous, but spatially discrepant visual stimulus. This effect is because of vision's superior acuity in the spatial dimension, but has also been shown to be influenced by the perception of unity of the two signals. We sought to investigate whether a similar phenomenon may occur between vision and somatosensation along the surface of the body as vision is known to possess superior spatial acuity to somatosensation. We report the first demonstration of the visuotactile ventriloquist illusion: individuals were instructed to localize visual stimuli (small white disks) or tactile stimuli (brief localized vibrations) that were presented concurrently or individually along the surface of the forearm, where bimodal presentations included spatially congruent and incongruent stimuli. Participants showed strong visual-tactile interactions. The tactile localization was strongly biased in the direction of the visual stimulus and the magnitude of this bias decreased as the spatial disparity between the two stimuli increased. The Bayesian causal inference model that has previously been shown to account for auditory-visual spatial localization and the ventriloquism effect also accounted well for the present data. Therefore, crossmodal interactions involving spatial representation along the surface of the body follow the same rules as crossmodal interactions involving representations of external space (auditory-visual).