Sensory experience during early sensitive periods shapes cross-modal temporal biases

Altered visual cortex excitatory/inhibitory ratio following transient congenital visual deprivation in humans

Non-human animal models have indicated that the ratio of excitation to inhibition (E/I) in neural circuits is experience dependent, and changes across development. Here, we assessed 3T Magnetic Resonance Spectroscopy (MRS) and electroencephalography (EEG) markers of cortical E/I ratio in 10 individuals who had been treated for dense bilateral congenital cataracts, after an average of 12 years of blindness, to test for dependence of the E/I ratio on early visual experience in humans. First, participants underwent MRS scanning at rest with their eyes open and eyes closed, to obtain visual cortex Gamma-Aminobutyric Acid (GABA+) concentration, Glutamate/Glutamine (Glx) concentration, and the concentration ratio of Glx/GABA+, as measures of inhibition, excitation, and E/I ratio, respectively. Subsequently, EEG was recorded to assess aperiodic activity (1-20 Hz) as a neurophysiological measure of the cortical E/I ratio, during rest with eyes open and eyes closed, and during flickering stimulation. Across conditions, congenital cataract-reversal individuals demonstrated a significantly lower visual cortex Glx/GABA+ ratio, and a higher intercept and steeper aperiodic slope at occipital electrodes, compared to age-matched sighted controls. In the congenital cataract-reversal group, a lower Glx/GABA+ ratio was associated with better visual acuity, and Glx concentration correlated positively with the aperiodic intercept in the conditions with visual input. We speculate that these findings result from an increased E/I ratio of the visual cortex as a consequence of congenital blindness, which might require commensurately increased inhibition in order to balance the additional excitation from restored visual input. The lower E/I ratio in congenital cataract-reversal individuals would thus be a consequence of homeostatic plasticity.

Precision-based causal inference modulates audiovisual temporal recalibration

Cross-modal temporal recalibration guarantees stable temporal perception across ever-changing environments. Yet, the mechanisms of cross-modal temporal recalibration remain unknown. Here, we conducted an experiment to measure how participants' temporal perception was affected by exposure to audiovisual stimuli with constant temporal delays that we varied across sessions. Consistent with previous findings, recalibration effects plateaued with increasing audiovisual asynchrony (nonlinearity) and varied by which modality led during the exposure phase (asymmetry). We compared six observer models that differed in how they update the audiovisual temporal bias during the exposure phase and in whether they assume a modality-specific or modality-independent precision of arrival latency. The causal-inference observer shifts the audiovisual temporal bias to compensate for perceived asynchrony, which is inferred by considering two causal scenarios: when the audiovisual stimuli have a common cause or separate causes. The asynchrony-contingent observer updates the bias to achieve simultaneity of auditory and visual measurements, modulating the update rate by the likelihood of the audiovisual stimuli originating from a simultaneous event. In the asynchrony-correction model, the observer first assesses whether the sensory measurement is asynchronous; if so, she adjusts the bias proportionally to the magnitude of the measured asynchrony. Each model was paired with either modality-specific or modality-independent precision of arrival latency. A Bayesian model comparison revealed that both the causal-inference process and modality-specific precision in arrival latency are required to capture the nonlinearity and asymmetry observed in audiovisual temporal recalibration. Our findings support the hypothesis that audiovisual temporal recalibration relies on the same causal-inference processes that govern cross-modal perception.

Precision-based causal inference modulates audiovisual temporal recalibration

Cross-modal temporal recalibration guarantees stable temporal perception across ever-changing environments. Yet, the mechanisms of cross-modal temporal recalibration remain unknown. Here, we conducted an experiment to measure how participants' temporal perception was affected by exposure to audiovisual stimuli with consistent temporal delays. Consistent with previous findings, recalibration effects plateaued with increasing audiovisual asynchrony and varied by which modality led during the exposure phase. We compared six observer models that differed in how they update the audiovisual temporal bias during the exposure phase and whether they assume modality-specific or modality-independent precision of arrival latency. The causal-inference observer shifts the audiovisual temporal bias to compensate for perceived asynchrony, which is inferred by considering two causal scenarios: when the audiovisual stimuli have a common cause or separate causes. The asynchrony-contingent observer updates the bias to achieve simultaneity of auditory and visual measurements, modulating the update rate by the likelihood of the audiovisual stimuli originating from a simultaneous event. In the asynchrony-correction model, the observer first assesses whether the sensory measurement is asynchronous; if so, she adjusts the bias proportionally to the magnitude of the measured asynchrony. Each model was paired with either modality-specific or modality-independent precision of arrival latency. A Bayesian model comparison revealed that both the causal-inference process and modality-specific precision in arrival latency are required to capture the nonlinearity and asymmetry observed in audiovisual temporal recalibration. Our findings support the hypothesis that audiovisual temporal recalibration relies on the same causal-inference processes that govern cross-modal perception.

Audiovisual simultaneity windows reflect temporal sensory uncertainty

The ability to judge the temporal alignment of visual and auditory information is a prerequisite for multisensory integration and segregation. However, each temporal measurement is subject to error. Thus, when judging whether a visual and auditory stimulus were presented simultaneously, observers must rely on a subjective decision boundary to distinguish between measurement error and truly misaligned audiovisual signals. Here, we tested whether these decision boundaries are relaxed with increasing temporal sensory uncertainty, i.e., whether participants make the same type of adjustment an ideal observer would make. Participants judged the simultaneity of audiovisual stimulus pairs with varying temporal offset, while being immersed in different virtual environments. To obtain estimates of participants' temporal sensory uncertainty and simultaneity criteria in each environment, an independent-channels model was fitted to their simultaneity judgments. In two experiments, participants' simultaneity decision boundaries were predicted by their temporal uncertainty, which varied unsystematically with the environment. Hence, observers used a flexibly updated estimate of their own audiovisual temporal uncertainty to establish subjective criteria of simultaneity. This finding implies that, under typical circumstances, audiovisual simultaneity windows reflect an observer's cross-modal temporal uncertainty.

Independent-channels models of temporal-order judgment revisited: A model comparison

The perception of temporal order or simultaneity of stimuli is almost always explained in terms of independent-channels models, such as perceptual-moment, triggered-moment, and attention-switching models. Independent-channels models generally posit that stimuli are processed in separate peripheral channels and that their arrival-time difference at a central location is translated into an internal state of order (simultaneity) if it reaches (misses) a certain threshold. Non-monotonic and non-parallel psychometric functions in a ternary-response task provided critical evidence against a wide range of independent-channels models. However, two independent-channels models have been introduced in the last decades that can account for such shapes by considering misreports of internal states (response-error model) or by assuming that simultaneity and order judgments rely on distinct sensory and decisional processes (two-stage model). Based on previous ideas, we also consider a two-threshold model, according to which the same arrival-time difference may need to reach a higher threshold for order detection than for successiveness detection. All three models were fitted to various data sets collected over a period of more than a century. The two-threshold model provided the best balance between goodness of fit and parsimony. This preference for the two-threshold model over the two-stage model and the response-error model aligns well with several lines of evidence from cognitive modeling, psychophysics, mental chronometry, and psychophysiology. We conclude that the seemingly deviant shapes of psychometric functions can be explained within the framework of independent-channels models in a simpler way than previously assumed.

Early-life maturation of the somatosensory cortex: sensory experience and beyond

Early life experiences shape physical and behavioral outcomes throughout lifetime. Sensory circuits are especially susceptible to environmental and physiological changes during development. However, the impact of different types of early life experience are often evaluated in isolation. In this mini review, we discuss the specific effects of postnatal sensory experience, sleep, social isolation, and substance exposure on barrel cortex development. Considering these concurrent factors will improve understanding of the etiology of atypical sensory perception in many neuropsychiatric and neurodevelopmental disorders.

Body schema: Resolving the conundrum of the distorted body

Humans show perceptual biases that suggest distorted internal representations of their own body. New research reveals that these perceptual biases can reflect integration of prior assumptions about body posture rather than a misshaped representation of the body's geometry.

Quantifying accuracy and precision from continuous response data in studies of spatial perception and crossmodal recalibration

The ability to detect the absolute location of sensory stimuli can be quantified with either error-based metrics derived from single-trial localization errors or regression-based metrics derived from a linear regression of localization responses on the true stimulus locations. Here we tested the agreement between these two approaches in estimating accuracy and precision in a large sample of 188 subjects who localized auditory stimuli from different azimuthal locations. A subsample of 57 subjects was subsequently exposed to audiovisual stimuli with a consistent spatial disparity before performing the sound localization test again, allowing us to additionally test which of the different metrics best assessed correlations between the amount of crossmodal spatial recalibration and baseline localization performance. First, our findings support a distinction between accuracy and precision. Localization accuracy was mainly reflected in the overall spatial bias and was moderately correlated with precision metrics. However, in our data, the variability of single-trial localization errors (variable error in error-based metrics) and the amount by which the eccentricity of target locations was overestimated (slope in regression-based metrics) were highly correlated, suggesting that intercorrelations between individual metrics need to be carefully considered in spatial perception studies. Secondly, exposure to spatially discrepant audiovisual stimuli resulted in a shift in bias toward the side of the visual stimuli (ventriloquism aftereffect) but did not affect localization precision. The size of the aftereffect shift in bias was at least partly explainable by unspecific test repetition effects, highlighting the need to account for inter-individual baseline differences in studies of spatial learning.

Visual experience shapes functional connectivity between occipital and non-visual networks

Comparisons across adults with different sensory histories (blind vs. sighted) have uncovered effects of experience on human brain function. In people born blind visual cortices are responsive to non-visual tasks and show altered functional connectivity at rest. Since almost all research has been done with adults, little is known about the developmental origins of this plasticity. Are infant visual cortices initially functionally like those of sighted adults and blindness causes reorganization? Alternatively, do infants start like blind adults, with vision required to set up the sighted pattern? To distinguish between these possibilities, we compare resting state functional connectivity across blind (n = 30) and blindfolded sighted (n = 50) adults to a large cohort of sighted infants (Developing Human Connectome Project, n = 475). Remarkably, we find that infant secondary visual cortices functionally resemble those of blind more than sighted adults, consistent with the idea that visual experience is required to set up long-range functional connectivity. Primary visual cortices show a mixture of instructive effects of vision and reorganizing effects of blindness. Specifically, in sighted adults, visual cortices show stronger functional coupling with nonvisual sensory-motor networks (i.e., auditory, somatosensory/motor) than with higher-cognitive prefrontal cortices (PFC). In blind adults, visual cortices show stronger coupling with PFC. In infants, connectivity of secondary visual cortices is stronger with PFC, while V1 shows equal sensory-motor/PFC connectivity. In contrast, lateralization of occipital-to-frontal connectivity resembles the sighted adults at birth and is reorganized by blindness, possibly due to recruitment of occipital networks for lateralized cognitive functions, such as language.

Sight restoration in congenitally blind humans does not restore visual brain structure

It is unknown whether impaired brain structure after congenital blindness is reversible if sight is restored later in life. Using structural magnetic resonance imaging, visual cortical surface area and cortical thickness were assessed in a large group of 21 sight-recovery individuals who had been born blind and who months or years later gained sight through cataract removal surgery. As control groups, we included 27 normally sighted individuals, 10 individuals with permanent congenital blindness, and 11 sight-recovery individuals with a late onset of cataracts. Congenital cataract-reversal individuals had a lower visual cortical surface area and a higher visual cortical thickness than normally sighted controls. These results corresponded to those of congenitally permanently blind individuals suggesting that impaired brain structure did not recover. Crucially, structural brain alterations in congenital-cataract reversal individuals were associated with a lower post-surgery visual acuity. No significant changes in visual cortex structure were observed in sight-recovery individuals with late onset cataracts. The results demonstrate that impaired structural brain development due to visual deprivation from birth is not fully reversible and limits functional recovery. Additionally, they highlight the crucial importance of prevention measures in the context of other types of aberrant childhood environments including low socioeconomic status and adversity.

Critical Periods in Vision Revisited

For four decades, investigations of the biological basis of critical periods in the developing mammalian visual cortex were dominated by study of the consequences of altered early visual experience in cats and nonhuman primates. The neural deficits thus revealed also provided insight into the origin and neural basis of human amblyopia that in turn motivated additional studies of humans with abnormal early visual input. Recent human studies point to deficits arising from alterations in all visual cortical areas and even in nonvisual cortical regions. As the new human data accumulated in parallel with a near-complete shift toward the use of rodent animal models for the study of neural mechanisms, it is now essential to review the human data and the earlier animal data obtained from cats and monkeys to infer general conclusions and to optimize future choice of the most appropriate animal model. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Brief Postnatal Visual Deprivation Triggers Long-Lasting Interactive Structural and Functional Reorganization of the Human Cortex

Patients treated for bilateral congenital cataracts provide a unique model to test the role of early visual input in shaping the development of the human cortex. Previous studies showed that brief early visual deprivation triggers long-lasting changes in the human visual cortex. However, it remains unknown if such changes interact with the development of other parts of the cortex. With high-resolution structural and resting-state fMRI images, we found changes in cortical thickness within, but not limited to, the visual cortex in adult patients, who experienced transient visual deprivation early in life as a result of congenital cataracts. Importantly, the covariation of cortical thickness across regions was also altered in the patients. The areas with altered cortical thickness in patients also showed differences in functional connectivity between patients and normally sighted controls. Together, the current findings suggest an impact of early visual deprivation on the interactive development of the human cortex.

Causal inference regulates audiovisual spatial recalibration via its influence on audiovisual perception

To obtain a coherent perception of the world, our senses need to be in alignment. When we encounter misaligned cues from two sensory modalities, the brain must infer which cue is faulty and recalibrate the corresponding sense. We examined whether and how the brain uses cue reliability to identify the miscalibrated sense by measuring the audiovisual ventriloquism aftereffect for stimuli of varying visual reliability. To adjust for modality-specific biases, visual stimulus locations were chosen based on perceived alignment with auditory stimulus locations for each participant. During an audiovisual recalibration phase, participants were presented with bimodal stimuli with a fixed perceptual spatial discrepancy; they localized one modality, cued after stimulus presentation. Unimodal auditory and visual localization was measured before and after the audiovisual recalibration phase. We compared participants’ behavior to the predictions of three models of recalibration: (a) Reliability-based: each modality is recalibrated based on its relative reliability—less reliable cues are recalibrated more; (b) Fixed-ratio: the degree of recalibration for each modality is fixed; (c) Causal-inference: recalibration is directly determined by the discrepancy between a cue and its estimate, which in turn depends on the reliability of both cues, and inference about how likely the two cues derive from a common source. Vision was hardly recalibrated by audition. Auditory recalibration by vision changed idiosyncratically as visual reliability decreased: the extent of auditory recalibration either decreased monotonically, peaked at medium visual reliability, or increased monotonically. The latter two patterns cannot be explained by either the reliability-based or fixed-ratio models. Only the causal-inference model of recalibration captures the idiosyncratic influences of cue reliability on recalibration. We conclude that cue reliability, causal inference, and modality-specific biases guide cross-modal recalibration indirectly by determining the perception of audiovisual stimuli.

Audiovisual recalibration of spatial perception occurs when we receive audiovisual stimuli with a systematic spatial discrepancy. The brain must determine to which extent both modalities should be recalibrated. In this study, we scrutinized the mechanisms the brain employs to do so. To this aim, we conducted a classical audiovisual recalibration experiment in which participants were adapted to spatially discrepant audiovisual stimuli. The visual component of the bimodal stimulus was either less, equally, or more reliable than the auditory component. We measured the amount of recalibration by computing the difference between participants’ unimodal localization responses before and after the audiovisual recalibration. Across participants, the influence of visual reliability on auditory recalibration varied fundamentally. We compared three models of recalibration. Only a causal-inference model of recalibration captured the diverse influences of cue reliability on recalibration found in our study, this model is also able to replicate contradictory results found in previous studies. In this model, recalibration depends on the discrepancy between a sensory measurement and the perceptual estimate for the same sensory modality. Cue reliability, perceptual biases, and the degree to which participants infer that the two cues come from a common source govern audiovisual perception and therefore audiovisual recalibration.

Development of multisensory integration following prolonged early-onset visual deprivation

Adult humans make effortless use of multisensory signals and typically integrate them in an optimal fashion.¹ This remarkable ability takes many years for normally sighted children to develop.^2,3 Would individuals born blind or with extremely low vision still be able to develop multisensory integration later in life when surgically treated for sight restoration? Late acquisition of such capability would be a vivid example of the brain's ability to retain high levels of plasticity. We studied the development of multisensory integration in individuals suffering from congenital dense bilateral cataract, surgically treated years after birth. We assessed cataract-treated individuals' reliance on their restored visual abilities when estimating the size of an object simultaneously explored by touch. Within weeks to months after surgery, when combining information from vision and touch, they developed a multisensory weighting behavior similar to matched typically sighted controls. Next, we tested whether cataract-treated individuals benefited from integrating vision with touch by increasing the precision of size estimates, as it occurs when integrating signals in a statistically optimal fashion.¹ For participants retested multiple times, such a benefit developed within months after surgery to levels of precision indistinguishable from optimal behavior. To summarize, the development of multisensory integration does not merely depend on age, but requires extensive multisensory experience with the world, rendered possible by the improved post-surgical visual acuity. We conclude that early exposure to multisensory signals is not essential for the development of multisensory integration, which can still be acquired even after many years of visual deprivation.

The size-weight illusion is unimpaired in individuals with a history of congenital visual deprivation

Visual deprivation in childhood can lead to lifelong impairments in multisensory processing. Here, the Size-Weight Illusion (SWI) was used to test whether visuo-haptic integration recovers after early visual deprivation. Normally sighted individuals perceive larger objects to be lighter than smaller objects of the same weight. In Experiment 1, individuals treated for dense bilateral congenital cataracts (who had no patterned visual experience at birth), individuals treated for developmental cataracts (who had patterned visual experience at birth, but were visually impaired), congenitally blind individuals and normally sighted individuals had to rate the weight of manually explored cubes that differed in size (Small, Medium, Large) across two possible weights (350 g, 700 g). In Experiment 2, individuals treated for dense bilateral congenital cataracts were compared to sighted individuals in a similar task using a string set-up, which removed haptic size cues. In both experiments, indistinguishable SWI effects were observed across all groups. These results provide evidence that early aberrant vision does not interfere with the development of the SWI, and suggest a recovery of the integration of size and weight cues provided by the visual and haptic modality.

Visual experience dependent plasticity in humans

While sensitive periods in brain development have often been studied by investigating the recovery of visual functions after a congenital phase of visual deprivation in non-human animals, research in humans who had recovered sight after a transient phase of congenital blindness is still scarce. Here, we discuss the hypothesis put forward based on non-human primate work which states that the effects of experience increase downstream the visual processing hierarchy. Recent results from behavioral and neuroscience studies in sight recovery individuals are discussed in the context of research findings from permanently congenitally blind humans as well as from prospective studies in infants and children.

Predictive modeling of neurobehavioral state and trait variation across development

A key goal of human neurodevelopmental research is to map neural and behavioral trajectories across both health and disease. A growing number of developmental consortia have begun to address this gap by providing open access to cross-sectional and longitudinal 'big data' repositories. However, it remains challenging to develop models that enable prediction of both within-subject and between-subject neurodevelopmental variation. Here, we present a conceptual and analytical perspective of two essential ingredients for mapping neurodevelopmental trajectories: state and trait components of variance. We focus on mapping variation across a range of neural and behavioral measurements and consider concurrent alterations of state and trait variation across development. We present a quantitative framework for combining both state- and trait-specific sources of neurobehavioral variation across development. Specifically, we argue that non-linear mixed growth models that leverage state and trait components of variance and consider environmental factors are necessary to comprehensively map brain-behavior relationships. We discuss this framework in the context of mapping language neurodevelopmental changes in early childhood, with an emphasis on measures of functional connectivity and their reliability for establishing robust neurobehavioral relationships. The ultimate goal is to statistically unravel developmental trajectories of neurobehavioral relationships that involve a combination of individual differences and age-related changes.

Sensory experience during early sensitive periods shapes cross-modal temporal biases

Typical human perception features stable biases such as perceiving visual events as later than synchronous auditory events. The origin of such perceptual biases is unknown. To investigate the role of early sensory experience, we tested whether a congenital, transient loss of pattern vision, caused by bilateral dense cataracts, has sustained effects on audio-visual and tactile-visual temporal biases and resolution. Participants judged the temporal order of successively presented, spatially separated events within and across modalities. Individuals with reversed congenital cataracts showed a bias towards perceiving visual stimuli as occurring earlier than auditory (Expt. 1) and tactile (Expt. 2) stimuli. This finding stood in stark contrast to normally sighted controls and sight-recovery individuals who had developed cataracts later in childhood: both groups exhibited the typical bias of perceiving vision as delayed compared to audition. These findings provide strong evidence that cross-modal temporal biases depend on sensory experience during an early sensitive period.