Development of multisensory integration following prolonged early-onset visual deprivation

The Status of Vernier Acuity Following Late Sight Onset

We possess a remarkably acute ability to detect even small misalignments between extended line segments. This "vernier acuity" significantly exceeds our "resolution acuity"-the ability to resolve closely separated stimuli-and is generally considered a "hyperacuity," since the detectable misalignments are markedly finer than the diameter of single retinal cones. Vernier acuity has, thus, often been proposed to reflect spatial organization and multi-unit cortical processing, rendering it an important index of visual function. Notably, vernier acuity exhibits a characteristic developmental signature: it is inferior to resolution acuity early in life but eventually exceeds it by up to one order of magnitude. However, vernier acuity may be disproportionately sensitive to developmental disruptions. Here, we examined the resilience of acquiring this visual proficiency to early-onset, prolonged deprivation by longitudinally tracking vernier and resolution acuities in children with dense congenital cataracts who gained sight late in life as part of Project Prakash. Our data reveal marked longitudinal improvements in both acuity measures and also demonstrate that, like the normally-sighted, late-sighted individuals' vernier acuity exceeds their resolution acuity, thereby rendering it a hyperacuity. However, the extent of this hyperacuity is weaker than observed in normally-sighted controls, pointing to partial limitations in postsurgical skill acquisition. Despite these constraints, our findings point to the feasibility of forming some integrative circuits in the visual system even when inputs are severely compromised, and to the availability of some residual plasticity late in childhood, with implications for the rehabilitation prospects of children following treatment for congenital cataracts.

The brain can develop conflicting multisensory principles to guide behavior

Midbrain multisensory neurons undergo a significant postnatal transition in how they process cross-modal (e.g. visual-auditory) signals. In early stages, signals derived from common events are processed competitively; however, at later stages they are processed cooperatively such that their salience is enhanced. This transition reflects adaptation to cross-modal configurations that are consistently experienced and become informative about which correspond to common events. Tested here was the assumption that overt behaviors follow a similar maturation. Cats were reared in omnidirectional sound thereby compromising the experience needed for this developmental process. Animals were then repeatedly exposed to different configurations of visual and auditory stimuli (e.g. spatiotemporally congruent or spatially disparate) that varied on each side of space and their behavior was assessed using a detection/localization task. Animals showed enhanced performance to stimuli consistent with the experience provided: congruent stimuli elicited enhanced behaviors where spatially congruent cross-modal experience was provided, and spatially disparate stimuli elicited enhanced behaviors where spatially disparate cross-modal experience was provided. Cross-modal configurations not consistent with experience did not enhance responses. The presumptive benefit of such flexibility in the multisensory developmental process is to sensitize neural circuits (and the behaviors they control) to the features of the environment in which they will function. These experiments reveal that these processes have a high degree of flexibility, such that two (conflicting) multisensory principles can be implemented by cross-modal experience on opposite sides of space even within the same animal.

Correctly establishing evidence for cue combination via gains in sensory precision: Why the choice of comparator matters

Studying how sensory signals from different sources (sensory cues) are integrated within or across multiple senses allows us to better understand the perceptual computations that lie at the foundation of adaptive behaviour. As such, determining the presence of precision gains - the classic hallmark of cue combination - is important for characterising perceptual systems, their development and functioning in clinical conditions. However, empirically measuring precision gains to distinguish cue combination from alternative perceptual strategies requires careful methodological considerations. Here, we note that the majority of existing studies that tested for cue combination either omitted this important contrast, or used an analysis approach that, unknowingly, strongly inflated false positives. Using simulations, we demonstrate that this approach enhances the chances of finding significant cue combination effects in up to 100% of cases, even when cues are not combined. We establish how this error arises when the wrong cue comparator is chosen and recommend an alternative analysis that is easy to implement but has only been adopted by relatively few studies. By comparing combined-cue perceptual precision with the best single-cue precision, determined for each observer individually rather than at the group level, researchers can enhance the credibility of their reported effects. We also note that testing for deviations from optimal predictions alone is not sufficient to ascertain whether cues are combined. Taken together, to correctly test for perceptual precision gains, we advocate for a careful comparator selection and task design to ensure that cue combination is tested with maximum power, while reducing the inflation of false positives.

Multisensory training improves the development of spatial cognition after sight restoration from congenital cataracts

Spatial cognition and mobility are typically impaired in congenitally blind individuals, as vision usually calibrates space perception by providing the most accurate distal spatial cues. We have previously shown that sight restoration from congenital bilateral cataracts guides the development of more accurate space perception, even when cataract removal occurs years after birth. However, late cataract-treated individuals do not usually reach the performance levels of the typically sighted population. Here, we developed a brief multisensory training that associated audiovisual feedback with body movements. Late cataract-treated participants quickly improved their space representation and mobility, performing as well as typically sighted controls in most tasks. Their improvement was comparable with that of a group of blind participants, who underwent training coupling their movements with auditory feedback alone. These findings suggest that spatial cognition can be enhanced by a training program that strengthens the association between bodily movements and their sensory feedback (either auditory or audiovisual).

Sound suppresses earliest visual cortical processing after sight recovery in congenitally blind humans

Neuroscientific research has consistently shown more extensive non-visual activity in the visual cortex of congenitally blind humans compared to sighted controls; a phenomenon known as crossmodal plasticity. Whether or not crossmodal activation of the visual cortex retracts if sight can be restored is still unknown. The present study, involving a rare group of sight-recovery individuals who were born pattern vision blind, employed visual event-related potentials to investigate persisting crossmodal modulation of the initial visual cortical processing stages. Here we report that the earliest, stimulus-driven retinotopic visual cortical activity (<100 ms) was suppressed in a spatially specific manner in sight-recovery individuals when concomitant sounds accompanied visual stimulation. In contrast, sounds did not modulate the earliest visual cortical response in two groups of typically sighted controls, nor in a third control group of sight-recovery individuals who had suffered a transient phase of later (rather than congenital) visual impairment. These results provide strong evidence for persisting crossmodal activity in the visual cortex after sight recovery following a period of congenital visual deprivation. Based on the time course of this modulation, we speculate on a role of exuberant crossmodal thalamic input which may arise during a sensitive phase of brain development.

Development and experience-dependence of multisensory spatial processing

Multisensory spatial processes are fundamental for efficient interaction with the world. They include not only the integration of spatial cues across sensory modalities, but also the adjustment or recalibration of spatial representations to changing cue reliabilities, crossmodal correspondences, and causal structures. Yet how multisensory spatial functions emerge during ontogeny is poorly understood. New results suggest that temporal synchrony and enhanced multisensory associative learning capabilities first guide causal inference and initiate early coarse multisensory integration capabilities. These multisensory percepts are crucial for the alignment of spatial maps across sensory systems, and are used to derive more stable biases for adult crossmodal recalibration. The refinement of multisensory spatial integration with increasing age is further promoted by the inclusion of higher-order knowledge.

Visual experience shapes the Bouba-Kiki effect and the size-weight illusion upon sight restoration from congenital blindness

The Bouba-Kiki effect is the systematic mapping between round/spiky shapes and speech sounds ("Bouba"/"Kiki"). In the size-weight illusion, participants judge the smaller of two equally-weighted objects as being heavier. Here we investigated the contribution of visual experience to the development of these phenomena. We compared three groups: early blind individuals (no visual experience), individuals treated for congenital cataracts years after birth (late visual experience), and typically sighted controls (visual experience from birth). We found that, in cataract-treated participants (tested visually/visuo-haptically), both phenomena are absent shortly after sight onset, just like in blind individuals (tested haptically). However, they emerge within months following surgery, becoming statistically indistinguishable from the sighted controls. This suggests a pivotal role of visual experience and refutes the existence of an early sensitive period: A short period of experience, even when gained only years after birth, is sufficient for participants to visually pick-up regularities in the environment, contributing to the development of these phenomena.

Grasping behavior does not recover after sight restoration from congenital blindness

We investigated whether early visual input is essential for establishing the ability to use predictions in the control of actions and for perception. To successfully interact with objects, it is necessary to pre-program bodily actions such as grasping movements (feedforward control). Feedforward control requires a model for making predictions, which is typically shaped by previous sensory experience and interaction with the environment.¹ Vision is the most crucial sense for establishing such predictions.^2,3 We typically rely on visual estimations of the to-be-grasped object's size and weight in order to scale grip force and hand aperture accordingly.^4,5,6 Size-weight expectations play a role also for perception, as evident in the size-weight illusion (SWI), in which the smaller of two equal-weight objects is misjudged to be heavier.^7,8 Here, we investigated predictions for action and perception by testing the development of feedforward controlled grasping and of the SWI in young individuals surgically treated for congenital cataracts several years after birth. Surprisingly, what typically developing individuals do easily within the first years of life, namely to adeptly grasp new objects based on visually predicted properties, cataract-treated individuals did not learn after years of visual experience. Contrary, the SWI exhibited significant development. Even though the two tasks differ in substantial ways, these results may suggest a potential dissociation in using visual experience to make predictions about an object's features for perception or action. What seems a very simple task-picking up small objects-is in truth a highly complex computation that necessitates early structured visual input to develop.

Noise-rearing precludes the behavioral benefits of multisensory integration

Concordant visual-auditory stimuli enhance the responses of individual superior colliculus (SC) neurons. This neuronal capacity for "multisensory integration" is not innate: it is acquired only after substantial cross-modal (e.g. auditory-visual) experience. Masking transient auditory cues by raising animals in omnidirectional sound ("noise-rearing") precludes their ability to obtain this experience and the ability of the SC to construct a normal multisensory (auditory-visual) transform. SC responses to combinations of concordant visual-auditory stimuli are depressed, rather than enhanced. The present experiments examined the behavioral consequence of this rearing condition in a simple detection/localization task. In the first experiment, the auditory component of the concordant cross-modal pair was novel, and only the visual stimulus was a target. In the second experiment, both component stimuli were targets. Noise-reared animals failed to show multisensory performance benefits in either experiment. These results reveal a close parallel between behavior and single neuron physiology in the multisensory deficits that are induced when noise disrupts early visual-auditory experience.

The power of vision: calibration of auditory space after sight restoration from congenital cataracts

Early visual deprivation typically results in spatial impairments in other sensory modalities. It has been suggested that, since vision provides the most accurate spatial information, it is used for calibrating space in the other senses. Here we investigated whether sight restoration after prolonged early onset visual impairment can lead to the development of more accurate auditory space perception. We tested participants who were surgically treated for congenital dense bilateral cataracts several years after birth. In Experiment 1 we assessed participants' ability to understand spatial relationships among sounds, by asking them to spatially bisect three consecutive, laterally separated sounds. Participants performed better after surgery than participants tested before. However, they still performed worse than sighted controls. In Experiment 2, we demonstrated that single sound localization in the two-dimensional frontal plane improves quickly after surgery, approaching performance levels of sighted controls. Such recovery seems to be mediated by visual acuity, as participants gaining higher post-surgical visual acuity performed better in both experiments. These findings provide strong support for the hypothesis that vision calibrates auditory space perception. Importantly, this also demonstrates that this process can occur even when vision is restored after years of visual deprivation.

Visual uncertainty unveils the distinct role of haptic cues in multisensory grasping

Human multisensory grasping movements (i.e., seeing and feeling a handheld object while grasping it with the contralateral hand) are superior to movements guided by each separate modality. This multisensory advantage might be driven by the integration of vision with either the haptic position only or with both position and size cues. To contrast these two hypotheses, we manipulated visual uncertainty (central vs. peripheral vision) and the availability of haptic cues during multisensory grasping. We showed a multisensory benefit irrespective of the degree of visual uncertainty suggesting that the integration process involved in multisensory grasping can be flexibly modulated by the contribution of each modality. Increasing visual uncertainty revealed the role of the distinct haptic cues. The haptic position cue was sufficient to promote multisensory benefits evidenced by faster actions with smaller grip apertures, whereas the haptic size was fundamental in fine-tuning the grip aperture scaling. These results support the hypothesis that, in multisensory grasping, vision is integrated with all haptic cues, with the haptic position cue playing the key part. Our findings highlight the important role of non-visual sensory inputs in sensorimotor control and hint at the potential contributions of the haptic modality in developing and maintaining visuomotor functions.Significance statementThe longstanding view that vision is considered the primary sense we rely on to guide grasping movements relegates the equally important haptic inputs, such as touch and proprioception, to a secondary role. Here we show that by increasing visual uncertainty during visuo-haptic grasping, the central nervous system exploits distinct haptic inputs about the object position and size to optimize grasping performance. Specifically, we demonstrate that haptic inputs about the object position are fundamental to support vision in enhancing grasping performance, whereas haptic size inputs can further refine hand shaping. Our results provide strong evidence that non-visual inputs serve an important, previously under-appreciated, functional role in grasping.