Cortex governs multisensory integration in the midbrain

The effect of developmental alcohol exposure on multisensory integration is larger in deeper cortical layers

Fetal Alcohol Spectrum Disorders (FASD) are one of the most common causes of mental disability in the world. Despite efforts to increase public awareness of the risks of drinking during pregnancy, epidemiological studies indicate a prevalence of 1-6% in all births. There is growing evidence that deficits in sensory processing may contribute to social problems observed in FASD. Multisensory (MS) integration occurs when a combination of inputs from two sensory modalities leads to enhancement or suppression of neuronal firing. MS enhancement is usually linked to processes that facilitate cognition and reaction time, whereas MS suppression has been linked to filtering unwanted sensory information. The rostral portion of the posterior parietal cortex (PPr) of the ferret is an area that shows robust visual-tactile integration and displays both MS enhancement and suppression. Recently, our lab demonstrated that ferrets exposed to alcohol during the "third trimester equivalent" of human gestation show less MS enhancement and more MS suppression in PPr than controls. Here we complement these findings by comparing in vivo electrophysiological recordings from channels located in shallow and deep cortical layers. We observed that while the effects of alcohol (less MS enhancement and more MS suppression) were found in all layers, the magnitude of these effects were more pronounced in putative layers V-VI. These findings extend our knowledge on the sensory deficits of FASD.

The Development of Multisensory Integration at the Neuronal Level

Multisensory integration is a fundamental function of the brain. In the typical adult, multisensory neurons' response to paired multisensory (e.g., audiovisual) cues is significantly more robust than the corresponding best unisensory response in many brain regions. Synthesizing sensory signals from multiple modalities can speed up sensory processing and improve the salience of outside events or objects. Despite its significance, multisensory integration is testified to be not a neonatal feature of the brain. Neurons' ability to effectively combine multisensory information does not occur rapidly but develops gradually during early postnatal life (for cats, 4-12 weeks required). Multisensory experience is critical for this developing process. If animals were restricted from sensing normal visual scenes or sounds (deprived of the relevant multisensory experience), the development of the corresponding integrative ability could be blocked until the appropriate multisensory experience is obtained. This section summarizes the extant literature on the development of multisensory integration (mainly using cat superior colliculus as a model), sensory-deprivation-induced cross-modal plasticity, and how sensory experience (sensory exposure and perceptual learning) leads to the plastic change and modification of neural circuits in cortical and subcortical areas.

Predictability alters multisensory responses by modulating unisensory inputs

The multisensory (deep) layers of the superior colliculus (SC) play an important role in detecting, localizing, and guiding orientation responses to salient events in the environment. Essential to this role is the ability of SC neurons to enhance their responses to events detected by more than one sensory modality and to become desensitized (‘attenuated’ or ‘habituated’) or sensitized (‘potentiated’) to events that are predictable via modulatory dynamics. To identify the nature of these modulatory dynamics, we examined how the repetition of different sensory stimuli affected the unisensory and multisensory responses of neurons in the cat SC. Neurons were presented with 2HZ stimulus trains of three identical visual, auditory, or combined visual–auditory stimuli, followed by a fourth stimulus that was either the same or different (‘switch’). Modulatory dynamics proved to be sensory-specific: they did not transfer when the stimulus switched to another modality. However, they did transfer when switching from the visual–auditory stimulus train to either of its modality-specific component stimuli and vice versa. These observations suggest that predictions, in the form of modulatory dynamics induced by stimulus repetition, are independently sourced from and applied to the modality-specific inputs to the multisensory neuron. This falsifies several plausible mechanisms for these modulatory dynamics: they neither produce general changes in the neuron’s transform, nor are they dependent on the neuron’s output.

A novel, interactive game to improve understanding of respiratory control pathways in first-year medical students

The physiology of respiration is a challenging subject for many medical students. To assist students, we have developed an active learning game that physically places students within a model outlining the respiratory control pathway. Participants were provided with a vodcast describing the physiology of respiratory control and instructed to view this before the activity. Once in the classroom, groups of students sat at tables marked to represent components of the respiratory control pathway (e.g., apneustic center, diaphragm etc.). Tables were connected with green and red ropes indicating excitatory or inhibitory effects, respectively. Students were presented with various scenarios (e.g., diabetic ketoacidosis) and asked to predict and illustrate the scenario's effect on subsequent steps in the respiratory pathway by waving the appropriate connecting rope. The next table would continue the pattern to simulate the collective physiological adaptation of the respiratory pathway. Thirty first-year medical students participated in this study. Following the activity, 25 out of the 30 participants completed an optional survey. The survey aimed to assess the benefits of adding this activity to our first-year medical curriculum to build a foundational understanding of the physiology of respiration. Responses were overwhelmingly favorable, and participants reported that playing the game significantly improved their perceived understanding of the physiology of respiratory control. All but one of the participants recommended using the activity in future classes. Because the small size of the study group may limit generalizability, future larger scale studies are planned.

Approaches to Understanding Multisensory Dysfunction in Autism Spectrum Disorder

Abnormal sensory responses are a DSM-5 symptom of autism spectrum disorder (ASD), and research findings demonstrate altered sensory processing in ASD. Beyond difficulties with processing information within single sensory domains, including both hypersensitivity and hyposensitivity, difficulties in multisensory processing are becoming a core issue of focus in ASD. These difficulties may be targeted by treatment approaches such as "sensory integration," which is frequently applied in autism treatment but not yet based on clear evidence. Recently, psychophysical data have emerged to demonstrate multisensory deficits in some children with ASD. Unlike deficits in social communication, which are best understood in humans, sensory and multisensory changes offer a tractable marker of circuit dysfunction that is more easily translated into animal model systems to probe the underlying neurobiological mechanisms. Paralleling experimental paradigms that were previously applied in humans and larger mammals, we and others have demonstrated that multisensory function can also be examined behaviorally in rodents. Here, we review the sensory and multisensory difficulties commonly found in ASD, examining laboratory findings that relate these findings across species. Next, we discuss the known neurobiology of multisensory integration, drawing largely on experimental work in larger mammals, and extensions of these paradigms into rodents. Finally, we describe emerging investigations into multisensory processing in genetic mouse models related to autism risk. By detailing findings from humans to mice, we highlight the advantage of multisensory paradigms that can be easily translated across species, as well as the potential for rodent experimental systems to reveal opportunities for novel treatments. LAY SUMMARY: Sensory and multisensory deficits are commonly found in ASD and may result in cascading effects that impact social communication. By using similar experiments to those in humans, we discuss how studies in animal models may allow an understanding of the brain mechanisms that underlie difficulties in multisensory integration, with the ultimate goal of developing new treatments. Autism Res 2020, 13: 1430-1449. © 2020 International Society for Autism Research, Wiley Periodicals, Inc.

Using superior colliculus principles of multisensory integration to reverse hemianopia

The diversity of our senses conveys many advantages; it enables them to compensate for one another when needed, and the information they provide about a common event can be integrated to facilitate its processing and, ultimately, adaptive responses. These cooperative interactions are produced by multisensory neurons. A well-studied model in this context is the multisensory neuron in the output layers of the superior colliculus (SC). These neurons integrate and amplify their cross-modal (e.g., visual-auditory) inputs, thereby enhancing the physiological salience of the initiating event and the probability that it will elicit SC-mediated detection, localization, and orientation behavior. Repeated experience with the same visual-auditory stimulus can also increase the neuron's sensitivity to these individual inputs. This observation raised the possibility that such plasticity could be engaged to restore visual responsiveness when compromised. For example, unilateral lesions of visual cortex compromise the visual responsiveness of neurons in the multisensory output layers of the ipsilesional SC and produces profound contralesional blindness (hemianopia). The possibility that multisensory plasticity could restore the visual responses of these neurons, and reverse blindness, was tested in the cat model of hemianopia. Hemianopic subjects were repeatedly presented with spatiotemporally congruent visual-auditory stimulus pairs in the blinded hemifield on a daily or weekly basis. After several weeks of this multisensory exposure paradigm, visual responsiveness was restored in SC neurons and behavioral responses were elicited by visual stimuli in the previously blind hemifield. The constraints on the effectiveness of this procedure proved to be the same as those constraining SC multisensory plasticity: whereas repetitions of a congruent visual-auditory stimulus was highly effective, neither exposure to its individual component stimuli, nor to these stimuli in non-congruent configurations was effective. The restored visual responsiveness proved to be robust, highly competitive with that in the intact hemifield, and sufficient to support visual discrimination.

What Can Computational Models Learn From Human Selective Attention? A Review From an Audiovisual Unimodal and Crossmodal Perspective

Selective attention plays an essential role in information acquisition and utilization from the environment. In the past 50 years, research on selective attention has been a central topic in cognitive science. Compared with unimodal studies, crossmodal studies are more complex but necessary to solve real-world challenges in both human experiments and computational modeling. Although an increasing number of findings on crossmodal selective attention have shed light on humans' behavioral patterns and neural underpinnings, a much better understanding is still necessary to yield the same benefit for intelligent computational agents. This article reviews studies of selective attention in unimodal visual and auditory and crossmodal audiovisual setups from the multidisciplinary perspectives of psychology and cognitive neuroscience, and evaluates different ways to simulate analogous mechanisms in computational models and robotics. We discuss the gaps between these fields in this interdisciplinary review and provide insights about how to use psychological findings and theories in artificial intelligence from different perspectives.

Cross-Talk of Low-Level Sensory and High-Level Cognitive Processing: Development, Mechanisms, and Relevance for Cross-Modal Abilities of the Brain

The emergence of cross-modal learning capabilities requires the interaction of neural areas accounting for sensory and cognitive processing. Convergence of multiple sensory inputs is observed in low-level sensory cortices including primary somatosensory (S1), visual (V1), and auditory cortex (A1), as well as in high-level areas such as prefrontal cortex (PFC). Evidence shows that local neural activity and functional connectivity between sensory cortices participate in cross-modal processing. However, little is known about the functional interplay between neural areas underlying sensory and cognitive processing required for cross-modal learning capabilities across life. Here we review our current knowledge on the interdependence of low- and high-level cortices for the emergence of cross-modal processing in rodents. First, we summarize the mechanisms underlying the integration of multiple senses and how cross-modal processing in primary sensory cortices might be modified by top-down modulation of the PFC. Second, we examine the critical factors and developmental mechanisms that account for the interaction between neuronal networks involved in sensory and cognitive processing. Finally, we discuss the applicability and relevance of cross-modal processing for brain-inspired intelligent robotics. An in-depth understanding of the factors and mechanisms controlling cross-modal processing might inspire the refinement of robotic systems by better mimicking neural computations.

Effects of Multimodal Association on Ambiguous Perception in Binocular Rivalry

When two eyes view dissimilar images, an observer typically reports ambiguous perception called binocular rivalry where the subjective perception fluctuates between the two inputs. This perceptual instability is often comprised of exclusive dominance of each image and a transition state called piecemeal state where the two images are intermingled in patchwork manner. Herein, we investigated the effects of multimodal association of sensory congruent pair, arbitrary pair, and reverse pair on piecemeal state in order to see how each level of association affects the ambiguous perception during binocular rivalry. To induce the multisensory associations, we designed a matching task with audiovisual feedback where subjects were required to respond according to given pairing rules. We found that explicit audiovisual associations can substantially affect the piecemeal state during binocular rivalry and that this congruency effect that reduces the amount of visual ambiguity originates primarily from explicit audiovisual association training rather than common sensory features. Furthermore, when one information is associated with multiple information, recent and preexisting associations work collectively to influence the perceptual ambiguity during rivalry. Our findings show that learned multimodal association directly affects the temporal dynamics of ambiguous perception during binocular rivalry by modulating not only the exclusive dominance but also the piecemeal state in a systematic manner.

Identification of calretinin-expressing retinal ganglion cells projecting to the mouse superior colliculus

In mice, retinal ganglion cells (RGCs), which consist of around 30 subtypes, exclusively transmit retinal information to the relevant brain systems through parallel visual pathways. The superior colliculus (SC) receives the vast majority of this information from several RGC subtypes. The objective of the current study is to identify the types of calretinin (CR)-expressing RGCs that project to the SC in mice. To label RGCs, we performed CR immunoreactivity in the mouse retina after injections of fluorescent dye, dextran into mouse SC. Subsequently, the neurons double-labeled for dextran and CR were iontophoretically injected with the lipophilic dye, DiI, to characterize the detailed morphological properties of these cells. The analysis of various morphological parameters, including dendritic arborization, dendritic field size and stratification, indicated that, of the ten different types of CR-expressing RGCs in the retina, the double-labeled cells consisted of at least eight types of RGCs that projected to the SC. These cells tended to have small-medium field sizes. However, except for dendritic field size, the cells did not exhibit consistent characteristics for the other morphometric parameters examined. The combination of a tracer and single-cell injections after immunohistochemistry for a particular molecule provided valuable data that confirmed the presence of distinct subtypes of RGCs within multiple-labeled RGCs that projected to specific brain regions.

Visual-Somatosensory Integration and Quantitative Gait Performance in Aging

Background: The ability to integrate information across sensory modalities is an integral aspect of mobility. Yet, the association between visual-somatosensory (VS) integration and gait performance has not been well-established in aging.

Methods: A total of 333 healthy older adults (mean age 76.53 ± 6.22; 53% female) participated in a visual-somatosensory simple reaction time task and underwent quantitative gait assessment using an instrumented walkway. Magnitude of VS integration was assessed using probability models, and then categorized into four integration classifications (superior, good, poor, or deficient). Associations of VS integration with three independent gait factors (Pace, Rhythm, and Variability derived by factor analysis method) were tested at cross-section using linear regression analyses. Given overlaps in neural circuitry necessary for both multisensory integration and goal-directed locomotion, we hypothesized that VS integration would be significantly associated with pace but not rhythm which is a more automatic process controlled mainly through brainstem and spinal networks.

Results: In keeping with our hypothesis, magnitude of VS integration was a strong predictor of pace (β = 0.12, p < 0.05) but not rhythm (β = −0.01, p = 0.83) in fully-adjusted models. While there was a trend for the association of magnitude of VS integration with variability (β = −0.11, p = 0.051), post-hoc testing of individual gait variables that loaded highest on the variability factor revealed that stride length variability (β = −0.13, p = 0.03) and not swing time variability (β = −0.08, p = 0.15) was significantly associated with magnitude of VS integration. Of the cohort, 29% had superior, 26% had good, 29% had poor, and 16% had deficient VS integration effects.

Conclusions: Worse VS integration in aging is associated with worse spatial but not temporal aspects of gait performance.

The Affective Core of the Self: A Neuro-Archetypical Perspective on the Foundations of Human (and Animal) Subjectivity

Psychologists usually considered the "Self" as an object of experience appearing when the individual perceives its existence within the conscious field. In accordance with such a view, the self-representing capacity of the human mind has been related to corticolimbic learning processes taking place within individual development. On the other hand, Carl Gustav Jung considered the Self as the core of our personality, in its conscious and unconscious aspects, as well as in its actual and potential forms. According to Jung, the Self originates from an inborn dynamic structure integrating the essential drives of our "brain-mind," and leading both to instinctual behavioral actions and to archetypal psychological experiences. Interestingly, recent neuroethological studies indicate that our subjective identity rests on ancient neuropsychic processes that humans share with other animals as part of their inborn constitutional repertoire. Indeed, brain activity within subcortical midline structures (SCMSs) is intrinsically related to the emergence of prototypical affective states, that not only influence our behavior in a flexible way, but alter our conscious field, giving rise to specific feelings or moods, which constitute the first form of self-orientation in the world. Moreover, such affective dynamics play a central role in the organization of individual personality and in the evolution of all other (more sophisticated) psychological functions. Therefore, on the base of the convergence between contemporary cutting-edge scientific research and some psychological intuitions of Jung, we intend here to explore the first neuroevolutional layer of human mind, that we call the affective core of the Self.

Seeing virtual while acting real: Visual display and strategy effects on the time and precision of eye-hand coordination

Effects of different visual displays on the time and precision of bare-handed or tool-mediated eye-hand coordination were investigated in a pick-and-place-task with complete novices. All of them scored well above average in spatial perspective taking ability and performed the task with their dominant hand. Two groups of novices, four men and four women in each group, had to place a small object in a precise order on the centre of five targets on a Real-world Action Field (RAF), as swiftly as possible and as precisely as possible, using a tool or not (control). Each individual session consisted of four visual display conditions. The order of conditions was counterbalanced between individuals and sessions. Subjects looked at what their hands were doing 1) directly in front of them (“natural” top-down view) 2) in top-down 2D fisheye view 3) in top-down undistorted 2D view or 4) in 3D stereoscopic top-down view (head-mounted OCULUS DK 2). It was made sure that object movements in all image conditions matched the real-world movements in time and space. One group was looking at the 2D images with the monitor positioned sideways (sub-optimal); the other group was looking at the monitor placed straight ahead of them (near-optimal). All image viewing conditions had significantly detrimental effects on time (seconds) and precision (pixels) of task execution when compared with “natural” direct viewing. More importantly, we find significant trade-offs between time and precision between and within groups, and significant interactions between viewing conditions and manipulation conditions. The results shed new light on controversial findings relative to visual display effects on eye-hand coordination, and lead to conclude that differences in camera systems and adaptive strategies of novices are likely to explain these.

An autism-associated serotonin transporter variant disrupts multisensory processing

Altered sensory processing is observed in many children with autism spectrum disorder (ASD), with growing evidence that these impairments extend to the integration of information across the different senses (that is, multisensory function). The serotonin system has an important role in sensory development and function, and alterations of serotonergic signaling have been suggested to have a role in ASD. A gain-of-function coding variant in the serotonin transporter (SERT) associates with sensory aversion in humans, and when expressed in mice produces traits associated with ASD, including disruptions in social and communicative function and repetitive behaviors. The current study set out to test whether these mice also exhibit changes in multisensory function when compared with wild-type (WT) animals on the same genetic background. Mice were trained to respond to auditory and visual stimuli independently before being tested under visual, auditory and paired audiovisual (multisensory) conditions. WT mice exhibited significant gains in response accuracy under audiovisual conditions. In contrast, although the SERT mutant animals learned the auditory and visual tasks comparably to WT littermates, they failed to show behavioral gains under multisensory conditions. We believe these results provide the first behavioral evidence of multisensory deficits in a genetic mouse model related to ASD and implicate the serotonin system in multisensory processing and in the multisensory changes seen in ASD.

Getting nowhere fast: trade-off between speed and precision in training to execute image-guided hand-tool movements

Background

The speed and precision with which objects are moved by hand or hand-tool interaction under image guidance depend on a specific type of visual and spatial sensorimotor learning. Novices have to learn to optimally control what their hands are doing in a real-world environment while looking at an image representation of the scene on a video monitor. Previous research has shown slower task execution times and lower performance scores under image-guidance compared with situations of direct action viewing. The cognitive processes for overcoming this drawback by training are not yet understood.

Methods

We investigated the effects of training on the time and precision of direct view versus image guided object positioning on targets of a Real-world Action Field (RAF). Two men and two women had to learn to perform the task as swiftly and as precisely as possible with their dominant hand, using a tool or not and wearing a glove or not. Individuals were trained in sessions of mixed trial blocks with no feed-back.

Results

As predicted, image-guidance produced significantly slower times and lesser precision in all trainees and sessions compared with direct viewing. With training, all trainees get faster in all conditions, but only one of them gets reliably more precise in the image-guided conditions. Speed-accuracy trade-offs in the individual performance data show that the highest precision scores and steepest learning curve, for time and precision, were produced by the slowest starter. Fast starters produced consistently poorer precision scores in all sessions. The fastest starter showed no sign of stable precision learning, even after extended training.

Conclusions

Performance evolution towards optimal precision is compromised when novices start by going as fast as they can. The findings have direct implications for individual skill monitoring in training programmes for image-guided technology applications with human operators.

Listening to another sense: somatosensory integration in the auditory system

Conventionally, sensory systems are viewed as separate entities, each with its own physiological process serving a different purpose. However, many functions require integrative inputs from multiple sensory systems and sensory intersection and convergence occur throughout the central nervous system. The neural processes for hearing perception undergo significant modulation by the two other major sensory systems, vision and somatosensation. This synthesis occurs at every level of the ascending auditory pathway: the cochlear nucleus, inferior colliculus, medial geniculate body and the auditory cortex. In this review, we explore the process of multisensory integration from (1) anatomical (inputs and connections), (2) physiological (cellular responses), (3) functional and (4) pathological aspects. We focus on the convergence between auditory and somatosensory inputs in each ascending auditory station. This review highlights the intricacy of sensory processing and offers a multisensory perspective regarding the understanding of sensory disorders.

A novel behavioral paradigm to assess multisensory processing in mice

Human psychophysical and animal behavioral studies have illustrated the benefits that can be conferred from having information available from multiple senses. Given the central role of multisensory integration for perceptual and cognitive function, it is important to design behavioral paradigms for animal models to provide mechanistic insights into the neural bases of these multisensory processes. Prior studies have focused on large mammals, yet the mouse offers a host of advantages, most importantly the wealth of available genetic manipulations relevant to human disease. To begin to employ this model species for multisensory research it is necessary to first establish and validate a robust behavioral assay for the mouse. Two common mouse strains (C57BL/6J and 129S6/SvEv) were first trained to respond to unisensory (visual and auditory) stimuli separately. Once trained, performance with paired audiovisual stimuli was then examined with a focus on response accuracy and behavioral gain. Stimulus durations varied from 50 ms to 1 s in order to modulate the effectiveness of the stimuli and to determine if the well-established "principle of inverse effectiveness" held in this model. Response accuracy in the multisensory condition was greater than for either unisensory condition for all stimulus durations, with significant gains observed at the 300 ms and 100 ms durations. Main effects of stimulus duration, stimulus modality and a significant interaction between these factors were observed. The greatest behavioral gain was seen for the 100 ms duration condition, with a trend observed that as the stimuli became less effective, larger behavioral gains were observed upon their pairing (i.e., inverse effectiveness). These results are the first to validate the mouse as a species that shows demonstrable behavioral facilitations under multisensory conditions and provides a platform for future mechanistically directed studies to examine the neural bases of multisensory integration.

The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities

Behavior, perception and cognition are strongly shaped by the synthesis of information across the different sensory modalities. Such multisensory integration often results in performance and perceptual benefits that reflect the additional information conferred by having cues from multiple senses providing redundant or complementary information. The spatial and temporal relationships of these cues provide powerful statistical information about how these cues should be integrated or "bound" in order to create a unified perceptual representation. Much recent work has examined the temporal factors that are integral in multisensory processing, with many focused on the construct of the multisensory temporal binding window - the epoch of time within which stimuli from different modalities is likely to be integrated and perceptually bound. Emerging evidence suggests that this temporal window is altered in a series of neurodevelopmental disorders, including autism, dyslexia and schizophrenia. In addition to their role in sensory processing, these deficits in multisensory temporal function may play an important role in the perceptual and cognitive weaknesses that characterize these clinical disorders. Within this context, focus on improving the acuity of multisensory temporal function may have important implications for the amelioration of the "higher-order" deficits that serve as the defining features of these disorders.

Divided multimodal attention sensory trace and context coding strategies in spatially congruent auditory and visual presentation

Previous research involving both unimodal and multimodal studies suggests that single-response change detection is a capacity-free process while a discriminatory up or down identification is capacity-limited. The trace/context model assumes that this reflects different memory strategies rather than inherent differences between identification and detection. To perform such tasks, one of two strategies is used, a sensory trace or a context coding strategy, and if one is blocked, people will automatically use the other. A drawback to most preceding studies is that stimuli are presented at separate locations, creating the possibility of a spatial confound, which invites alternative interpretations of the results. We describe a series of experiments, investigating divided multimodal attention, without the spatial confound. The results challenge the trace/context model. Our critical experiment involved a gap before a change in volume and brightness, which according to the trace/context model blocks the sensory trace strategy, simultaneously with a roaming pedestal, which should block the context coding strategy. The results clearly show that people can use strategies other than sensory trace and context coding in the tasks and conditions of these experiments, necessitating changes to the trace/context model.

Development of multisensory integration from the perspective of the individual neuron

The ability to use cues from multiple senses in concert is a fundamental aspect of brain function. It maximizes the brain’s use of the information available to it at any given moment and enhances the physiological salience of external events. Because each sense conveys a unique perspective of the external world, synthesizing information across senses affords computational benefits that cannot otherwise be achieved. Multisensory integration not only has substantial survival value but can also create unique experiences that emerge when signals from different sensory channels are bound together. However, neurons in a newborn’s brain are not capable of multisensory integration, and studies in the midbrain have shown that the development of this process is not predetermined. Rather, its emergence and maturation critically depend on cross-modal experiences that alter the underlying neural circuit in such a way that optimizes multisensory integrative capabilities for the environment in which the animal will function.

Dissociation of psychophysical and EEG steady-state response measures of cross-modal temporal correspondence for amplitude modulated acoustic and vibrotactile stimulation

Research examining multisensory integration suggests that the correspondence of stimulus characteristics across modalities (cross-modal correspondence) can have a dramatic influence on both neurophysiological and perceptual responses to multimodal stimulation. The current study extends prior research by examining the cross-modal correspondence of amplitude modulation rate for simultaneous acoustic and vibrotactile stimulation using EEG and perceptual measures of sensitivity to amplitude modulation. To achieve this, psychophysical thresholds and steady-state responses (SSRs) were measured for acoustic and vibrotactile amplitude modulated (AM) stimulation for 21 and 40 Hz AM rates as a function of the cross-modal correspondence. The study design included three primary conditions to determine whether the changes in the SSR and psychophysical thresholds were due to the cross-modal temporal correspondence of amplitude modulated stimuli: NONE (AM in one modality only), SAME (the same AM rate for each modality) and DIFF (different AM rates for each modality). The results of the psychophysical analysis showed that AM detection thresholds for the simultaneous AM conditions (i.e., SAME and DIFF) were significantly higher (i.e., lower sensitivity) than AM detection thresholds for the stimulation of a single modality (i.e., NONE). SSR results showed significant effects of SAME and DIFF conditions on SSR activity. The different pattern of results for perceptual and SSR measures of cross-modal correspondence of AM rate indicates a dissociation between entrained cortical activity (i.e., SSR) and perception.

Dissociation of reach-related and visual signals in the human superior colliculus

Electrophysiological and micro-stimulation studies in non-human animal species indicated that the superior colliculus (SC) plays a role in the control of upper limb movements. In our previous work we found reach-related signals in the deep superior colliculus in humans. Here we show that also signals in more dorsal locations are correlated with the execution of arm movements. We instructed healthy participants to reach for visual targets either presented in the left or in the right visual hemifield during an fMRI measurement. Visual stimulation was dissociated from movement execution using a pro- and anti-reaching task. Thereby, we successfully differentiated between signals at these locations induced by the visual input of target presentations on the one hand and by the execution of arm movements on the other hand. Extending our previous report, the results of this study are in good agreement with the observed anatomical distribution of reach-related neurons in macaques. Obviously, reach-related signals can be found across a considerable depth range also in humans.

Identifying the computational requirements of an integrated top-down-bottom-up model for overt visual attention within an active vision system

Computational visual attention systems have been constructed in order for robots and other devices to detect and locate regions of interest in their visual world. Such systems often attempt to take account of what is known of the human visual system and employ concepts, such as 'active vision', to gain various perceived advantages. However, despite the potential for gaining insights from such experiments, the computational requirements for visual attention processing are often not clearly presented from a biological perspective. This was the primary objective of this study, attained through two specific phases of investigation: 1) conceptual modeling of a top-down-bottom-up framework through critical analysis of the psychophysical and neurophysiological literature, 2) implementation and validation of the model into robotic hardware (as a representative of an active vision system). Seven computational requirements were identified: 1) transformation of retinotopic to egocentric mappings, 2) spatial memory for the purposes of medium-term inhibition of return, 3) synchronization of 'where' and 'what' information from the two visual streams, 4) convergence of top-down and bottom-up information to a centralized point of information processing, 5) a threshold function to elicit saccade action, 6) a function to represent task relevance as a ratio of excitation and inhibition, and 7) derivation of excitation and inhibition values from object-associated feature classes. The model provides further insight into the nature of data representation and transfer between brain regions associated with the vertebrate 'active' visual attention system. In particular, the model lends strong support to the functional role of the lateral intraparietal region of the brain as a primary area of information consolidation that directs putative action through the use of a 'priority map'.

Signals from the deep: reach-related activity in the human superior colliculus

Neurophysiological studies in nonhuman species indicated that neurons in the superior colliculus (SC) are involved in the control of upper limb movements. These findings suggested that the SC represents a crucial hub in a general sensorimotor network, including skeletomotor as much as oculomotor functions. In contrast to the SC in the various animal models, the human SC is largely unknown territory. In particular, it is unknown whether findings of reach-related activity in the nonhuman SC can be extrapolated to humans. Using fMRI we found signal increases at superficial/intermediate and deep locations at the SC during the execution of arm movements. In contrast, signals related to saccade execution were confined to the superficial and intermediate locations. Although targets for reaching were presented in the left and right hemifields under central fixation, we found a lateralization of reach-related signals with respect to the active arm. In contrast, saccade-related activity was bilateral, in agreement with the bilateral target presentation and the resulting directions of saccades. Our results suggest that the human SC not only contributes to the coordination of eye movements and spatial shifts of attentions but also to the sensorimotor control of arm movements.

Unilateral cerebral ischemia inhibits optomotor responses of the ipsilateral eye in mice

A reduction in blood flow to the brain causes stroke and damage to neuronal networks. Cerebral ischemia is frequently associated with loss of visual functions. Because retinal and small cerebral vessels are vulnerable to similar risk factors, the loss of vision could result from concurrent retinal ischemia, and it is not clear if visual functions may be inhibited by cerebral ischemia directly. In this study, the distal middle cerebral artery in the right hemisphere of mice was occluded to produce unilateral cerebral ischemia and subsequent infarction. The layer V neurons expressing YFP in transgenic yellow fluorescent protein in transgenic B6.Cg-Tg(Thy1-YFPH)2Jrs/J mice disappeared in the motor and somatosensory cortex, but not in the visual area. The latencies of flash visual evoked potential recorded from two hemispheres were imbalanced, but did not differ markedly from the latencies recorded in controls. However, the optomotor responses of the ipsilateral eye were significantly reduced by 48 h after occlusion. Our results suggest that focused cerebral ischemia may inhibit ipsilateral eye movement in the absence of damage to the visual cortex. This study may provide a platform for further investigation of the relationship between cortical ischemia and visual function.

Crossmodal reorganization in the early deaf switches sensory, but not behavioral roles of auditory cortex

It is well known that early disruption of sensory input from one modality can induce crossmodal reorganization of a deprived cortical area, resulting in compensatory abilities in the remaining senses. Compensatory effects, however, occur in selected cortical regions and it is not known whether such compensatory phenomena have any relation to the original function of the reorganized area. In the cortex of hearing cats, the auditory field of the anterior ectosylvian sulcus (FAES) is largely responsive to acoustic stimulation and its unilateral deactivation results in profound contralateral acoustic orienting deficits. Given these functional and behavioral roles, the FAES was studied in early-deafened cats to examine its crossmodal sensory properties as well as to assess the behavioral role of that reorganization. Recordings in the FAES of early-deafened adults revealed robust responses to visual stimulation as well as receptive fields that collectively represented the contralateral visual field. A second group of early-deafened cats was trained to localize visual targets in a perimetry array. In these animals, cooling loops were surgically placed on the FAES to reversibly deactivate the region, which resulted in substantial contralateral visual orienting deficits. These results demonstrate that crossmodal plasticity can substitute one sensory modality for another while maintaining the functional repertoire of the reorganized region.

Onset timing of cross-sensory activations and multisensory interactions in auditory and visual sensory cortices

Here we report early cross-sensory activations and audiovisual interactions at the visual and auditory cortices using magnetoencephalography (MEG) to obtain accurate timing information. Data from an identical fMRI experiment were employed to support MEG source localization results. Simple auditory and visual stimuli (300-ms noise bursts and checkerboards) were presented to seven healthy humans. MEG source analysis suggested generators in the auditory and visual sensory cortices for both within-modality and cross-sensory activations. fMRI cross-sensory activations were strong in the visual but almost absent in the auditory cortex; this discrepancy with MEG possibly reflects the influence of acoustical scanner noise in fMRI. In the primary auditory cortices (Heschl's gyrus) the onset of activity to auditory stimuli was observed at 23 ms in both hemispheres, and to visual stimuli at 82 ms in the left and at 75 ms in the right hemisphere. In the primary visual cortex (Calcarine fissure) the activations to visual stimuli started at 43 ms and to auditory stimuli at 53 ms. Cross-sensory activations thus started later than sensory-specific activations, by 55 ms in the auditory cortex and by 10 ms in the visual cortex, suggesting that the origins of the cross-sensory activations may be in the primary sensory cortices of the opposite modality, with conduction delays (from one sensory cortex to another) of 30-35 ms. Audiovisual interactions started at 85 ms in the left auditory, 80 ms in the right auditory and 74 ms in the visual cortex, i.e., 3-21 ms after inputs from the two modalities converged.

Adult plasticity of spatiotemporal receptive fields of multisensory superior colliculus neurons following early visual deprivation

PURPOSE

Previous work has established that the integrative capacity of multisensory neurons in the superior colliculus (SC) matures over a protracted period of postnatal life (Wallace and Stein, 1997), and that the development of normal patterns of multisensory integration depends critically on early sensory experience (Wallace et al., 2004). Although these studies demonstrated the importance of early sensory experience in the creation of mature multisensory circuits, it remains unknown whether the reestablishment of sensory experience in adulthood can reverse these effects and restore integrative capacity.

METHODS

The current study tested this hypothesis in cats that were reared in absolute darkness until adulthood and then returned to a normal housing environment for an equivalent period of time. Single unit extracellular recordings targeted multisensory neurons in the deep layers of the SC, and analyses were focused on both conventional measures of multisensory integration and on more recently developed methods designed to characterize spatiotemporal receptive fields (STRF).

RESULTS

Analysis of the STRF structure and integrative capacity of multisensory SC neurons revealed significant modifications in the temporal response dynamics of multisensory responses (e.g., discharge durations, peak firing rates, and mean firing rates), as well as significant changes in rates of spontaneous activation and degrees of multisensory integration.

CONCLUSIONS

These results emphasize the importance of early sensory experience in the establishment of normal multisensory processing architecture and highlight the limited plastic potential of adult multisensory circuits.

The neural basis of multisensory integration in the midbrain: its organization and maturation

Multisensory integration describes a process by which information from different sensory systems is combined to influence perception, decisions, and overt behavior. Despite a widespread appreciation of its utility in the adult, its developmental antecedents have received relatively little attention. Here we review what is known about the development of multisensory integration, with a focus on the circuitry and experiential antecedents of its development in the model system of the multisensory (i.e., deep) layers of the superior colliculus. Of particular interest here are two sets of experimental observations: (1) cortical influences appear essential for multisensory integration in the SC, and (2) postnatal experience guides its maturation. The current belief is that the experience normally gained during early life is instantiated in the cortico-SC projection, and that this is the primary route by which ecological pressures adapt SC multisensory integration to the particular environment in which it will be used.

Postnatal experiences influence how the brain integrates information from different senses

Sensory processing disorder (SPD) is characterized by anomalous reactions to, and integration of, sensory cues. Although the underlying etiology of SPD is unknown, one brain region likely to reflect these sensory and behavioral anomalies is the superior colliculus (SC), a structure involved in the synthesis of information from multiple sensory modalities and the control of overt orientation responses. In the present review we describe normal functional properties of this structure, the manner in which its individual neurons integrate cues from different senses, and the overt SC-mediated behaviors that are believed to manifest this “multisensory integration.” Of particular interest here is how SC neurons develop their capacity to engage in multisensory integration during early postnatal life as a consequence of early sensory experience, and the intimate communication between cortex and the midbrain that makes this developmental process possible.

Task-relevance and temporal synchrony between tactile and visual stimuli modulates cortical activity and motor performance during sensory-guided movement

Sensory-guided movements require the analysis and integration of task-relevant sensory inputs from multiple modalities. This article sought to: (1) assess effects of intermodal temporal synchrony upon modulation of primary somatosensory cortex (S1) during continuous sensorimotor transformations, (2) identify cortical areas sensitive to temporal synchrony, and (3) provide further insight into the reduction of S1 activity during continuous vibrotactile tracking previously observed by our group (Meehan and Staines 2007: Brain Res 1138:148-158). Functional MRI was acquired while participants received simultaneous bimodal (visuospatial/vibrotactile) stimulation and continuously tracked random changes in one modality, by applying graded force to a force-sensing resistor. Effects of intermodal synchrony were investigated, unbeknownst to the participants, by varying temporal synchrony so that sensorimotor transformations dictated by the distracter modality either conflicted (low synchrony) or supplemented (high synchrony) those of the target modality. Temporal synchrony differentially influenced tracking performance dependent upon tracking modality. Physiologically, synchrony did not influence S1 activation; however, the insula and superior temporal gyrus were influenced regardless of tracking modality. The left temporal-parietal junction demonstrated increased activation during high synchrony specific to vibrotactile tracking. The superior parietal lobe and superior temporal gyrus demonstrated increased activation during low synchrony specific to visuospatial tracking. As previously reported, vibrotactile tracking resulted in decreased S1 activation relative to when it was task-irrelevant. We conclude that while temporal synchrony is represented at higher levels than S1, interactions between inter- and intramodal mechanisms determines sensory processing at the level of S1.

Multisensory enhancement in the optic tectum of the barn owl: spike count and spike timing

Temporal and spatial correlations between auditory and visual stimuli facilitate the perception of unitary events and improve behavioral responses. However, it is not clear how combined visual and auditory information is processed in single neurons. Here we studied responses of multisensory neurons in the barn owl's optic tectum (the avian homologue of the superior colliculus) to visual, auditory, and bimodal stimuli. We specifically focused on responses to sequences of repeated stimuli. We first report that bimodal stimulation tends to elicit more spikes than in the responses to its unimodal components (a phenomenon known as multisensory enhancement). However, this tendency was found to be history-dependent; multisensory enhancement was mostly apparent in the first stimulus of the sequence and to a much lesser extent in the subsequent stimuli. Next, a vector-strength analysis was applied to quantify the phase locking of the responses to the stimuli. We report that in a substantial number of multisensory neurons responses to sequences of bimodal stimuli elicited spike trains that were better phase locked to the stimulus than spike trains elicited by stimulating with the unimodal counterparts (visual or auditory). We conclude that multisensory enhancement can be manifested in better phase locking to the stimulus as well as in more spikes.

Multimodal chemosensory integration through the maxillary palp in Drosophila

Drosophila melanogaster has an olfactory organ called the maxillary palp. It is smaller and numerically simpler than the antenna, and its specific role in behavior has long been unclear. Because of its proximity to the mouthparts, I explored the possibility of a role in taste behavior. Maxillary palp was tuned to mediate odor-induced taste enhancement: a sucrose solution was more appealing when simultaneously presented with the odorant 4-methylphenol. The same result was observed with other odors that stimulate other types of olfactory receptor neuron in the maxillary palp. When an antennal olfactory receptor was genetically introduced in the maxillary palp, the fly interpreted a new odor as a sweet-enhancing smell. These results all point to taste enhancement as a function of the maxillary palp. It also opens the door for studying integration of multiple senses in a model organism.

Time course of auditory masker effects: tapping the locus of audiovisual integration?

In a focused attention paradigm, saccadic reaction time (SRT) to a visual target tends to be shorter when an auditory accessory stimulus is presented in close temporal and spatial proximity. Observed SRT reductions typically diminish as spatial disparity between the stimuli increases. Here a visual target LED (500 ms duration) was presented above or below the fixation point and a simultaneously presented auditory accessory (2 ms duration) could appear at the same or the opposite vertical position. SRT enhancement was about 35 ms in the coincident and 10 ms in the disparate condition. In order to further probe the audiovisual integration mechanism, in addition to the auditory non-target an auditory masker (200 ms duration) was presented before, simultaneous to, or after the accessory stimulus. In all interstimulus interval (ISI) conditions, SRT enhancement went down both in the coincident and disparate configuration, but this decrement was fairly stable across the ISI values. If multisensory integration solely relied on a feed-forward process, one would expect a monotonic decrease of the masker effect with increasing ISI in the backward masking condition. It is therefore conceivable that the relatively high-energetic masker causes a broad excitatory response of SC neurons. During this state, the spatial audio-visual information from multisensory association areas is fed back and merged with the spatially unspecific excitation pattern induced by the masker. Assuming that a certain threshold of activation has to be achieved in order to generate a saccade in the correct direction, the blurred joint output of noise and spatial audio-visual information needs more time to reach this threshold prolonging SRT to an audio-visual object.

Multisensory integration: current issues from the perspective of the single neuron

For thousands of years science philosophers have been impressed by how effectively the senses work together to enhance the salience of biologically meaningful events. However, they really had no idea how this was accomplished. Recent insights into the underlying physiological mechanisms reveal that, in at least one circuit, this ability depends on an intimate dialogue among neurons at multiple levels of the neuraxis; this dialogue cannot take place until long after birth and might require a specific kind of experience. Understanding the acquisition and usage of multisensory integration in the midbrain and cerebral cortex of mammals has been aided by a multiplicity of approaches. Here we examine some of the fundamental advances that have been made and some of the challenging questions that remain.

Microsaccadic responses in a bimodal oddball task

In a visual oddball task the presentation of rare targets induces a prolonged microsaccadic inhibition as compared to standards. Here, we replicated this effect also in the auditory modality. In addition, although auditory standards induced a more limited modulation of microsaccadic frequency as compared to visual standards, auditory oddballs induced a prolonged microsaccadic inhibition. With bimodal standard stimuli the microsaccadic response was determined by the attended modality, resembling that produced by attended unimodal stimuli. The present findings support the idea that the microsaccadic response to oddball and standard stimuli is partly driven by cognitive mechanisms common to both the visual and the auditory modality, and that microsaccades can be used as an implicit behavioral measure of ongoing cognitive processes.

Tri-modal integration of visual, tactile and auditory signals for the perception of sequences of events

We investigated the interactions between visual, tactile and auditory sensory signals for the perception of sequences of events. Sequences of flashes, taps and beeps were presented simultaneously. For each session, subjects were instructed to count the number of events presented in one modality (Target) and to ignore the stimuli presented in the other modalities (Background). The number of events presented in the background sequence could differ from the number of events in the target sequence. For each session, we quantified the Background-evoked bias by comparing subjects' responses with and without Background (Target presented alone). Nine combinations between vision, touch and audition were tested. In each session but two, the Background significantly biased the Target. Vision was the most susceptible to Background-evoked bias and the least efficient in biasing the other two modalities. By contrast, audition was the least susceptible to Background-evoked bias and the most efficient in biasing the other two modalities. These differences were strongly correlated to the relative reliability of each modality. In line with this, the evoked biases were larger when the Background consisted of two instead of only one modality. These results show that for the perception of sequences of events: (1) vision, touch and audition are automatically integrated; (2) the respective contributions of the three modalities to the integrated percept differ; (3) the relative contribution of each modality depends on its relative reliability (1/variability); (4) task-irrelevant stimuli have more weight when presented in two rather than only one modality.

Pain and stress in a systems perspective: reciprocal neural, endocrine, and immune interactions

This paper advances a psychophysiological systems view of pain in which physical injury, or wounding, generates a complex stress response that extends beyond the nervous system and contributes to the experience of pain. Through a common chemical language comprising neurotransmitters, peptides, endocannabinoids, cytokines, and hormones, an ensemble of interdependent nervous, endocrine, and immune processes operates in concert to cope with the injury. These processes act as a single agent and comprise a supersystem. Acute pain in its multiple dimensions, and the related symptoms that commonly occur with it, are products of the supersystem. Chronic pain can develop as a result of unusual stress. Social stressors can compound the stress resulting from a wound or act alone to dysregulate the supersystem. When the supersystem suffers dysregulation, health, function, and sense of well-being suffer. Some chronic pain conditions are the product of supersystem dysregulation. Individuals vary and are vulnerable to dysregulation and dysfunction in particular organ systems due to the unique interactions of genetic, epigenetic and environmental factors, as well as the past experiences that characterize each person.

PERSPECTIVE

Acute tissue injury activates an ensemble of interdependent nervous, endocrine, and immune processes that operate in concert and comprise a supersystem. Some chronic pain conditions result from supersystem dysregulation. Individuals vary and are vulnerable to dysregulation due to the unique interactions of genetic, epigenetic, and environmental factors and past experiences that characterize each person. This perspective can potentially assist clinicians in assessing and managing chronic pain patients.

Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions

BACKGROUND

Viewing a speaker's articulatory movements substantially improves a listener's ability to understand spoken words, especially under noisy environmental conditions. In this study we investigated the ability of patients with schizophrenia to integrate visual and auditory speech. Our objective was to determine to what extent they experience benefit from visual articulation and to detail under what listening conditions they might show the greatest impairments.

METHODS

We assessed the ability to recognize auditory and audiovisual speech in different levels of noise in 18 patients with schizophrenia and compared their performance with that of 18 healthy volunteers. We used a large set of monosyllabic words as our stimuli in order to more closely approximate performance in everyday situations.

RESULTS

Patients with schizophrenia showed deficits in their ability to derive benefit from visual articulatory motion. This impairment was most pronounced at signal-to-noise levels where multisensory gain is known to be maximal in healthy control subjects. A surprising finding was that despite known early auditory sensory processing deficits and reports of impairments in speech processing in schizophrenia, patients' performance in unisensory auditory speech perception remained fully intact.

CONCLUSIONS

Thus, the results showed a specific deficit in multisensory speech processing in the absence of any measurable deficit in unisensory speech processing and suggest that sensory integration dysfunction may be an important and, to date, rather overlooked aspect of schizophrenia.

Multisensory integration produces an initial response enhancement

The brain has evolved the ability to integrate information across the senses in order to improve the detection and disambiguation of biologically significant events. This multisensory synthesis of information leads to faster (and more accurate) behavioral responses, yet the underlying neural mechanisms by which these responses are speeded are as yet unclear. The aim of these experiments was to evaluate the temporal properties of multisensory enhancement in the physiological responses of neurons in the superior colliculus (SC). Of specific interest was the temporal evolution of their responses to individual modality-specific stimuli as well as to cross-modal combinations of these stimuli. The results demonstrate that cross-modal stimuli typically elicit faster, more robust, and more reliable physiological responses than do their modality-specific component stimuli. Response measures sensitive to the time domain showed that these multisensory responses were enhanced from their very onset, and that the acceleration of the enhancement was greatest within the first 40ms (or 50% of the response). The latter half of the multisensory response was typically only as robust and informative as predicted by a linear combination of the unisensory component responses. These results may reveal some of the key physiological changes underlying many of the SC-mediated behavioral benefits of multisensory integration.

Signal reliability modulates auditory-tactile integration for event counting

Sequences of auditory beeps and tactile taps were simultaneously presented and participants were instructed to focus on one of these modalities and to ignore the other. We tested whether (i) the two sensory channels bias one another and (ii) the interaction depends on the relative reliability of the channels. Audition biased tactile perception and touch biased auditory perception. Lowering the reliability of the auditory channel (i.e. the intensity of the beeps) decreased the effect of audition on touch and increased the effect of touch on audition. These results show that simultaneous auditory and tactile stimuli tend to be automatically integrated in a reliability-dependent manner.