Using the past to estimate sensory uncertainty

FrAMBI: A Software Framework for Auditory Modeling Based on Bayesian Inference

Research in hearing science often relies on auditory models to describe listener's behaviour and its neural underpinning in acoustic environments. These models gather empirical evidence from behavioural data to address research questions on the neural mechanisms underlying sound perception. Despite seemingly similar statistical methods, auditory models are often implemented for each study separately, which hinders reproducibility and across-study comparisons, thus limiting the advancement at a field level. Here, we introduce a framework for studying neural mechanisms of sound perception by employing auditory modeling based on Bayesian inference (FrAMBI), a MATLAB/Octave toolbox. FrAMBI provides a standardized structure to implement an auditory model following the perception-action cycle and enables the automatic application of statistical analysis with behavioural data. We show FrAMBI's capabilities in several examples with increasing levels of complexity within the context of sound source localisation tasks: a basic implementation for a static scenario, iterating over the perception-action cycle with a moving sound source, the definition of multiple model variants testing different neural mechanisms, and the procedure for parameter estimation and model comparison. Being integrated into the widely used auditory modelling toolbox (AMT), FrAMBI is planned to be maintained in the long term and expanded accordingly, fostering reproducible research in the field of neuroscience.

Perceived Multisensory Common Cause Relations Shape the Ventriloquism Effect but Only Marginally the Trial-Wise Aftereffect

Combining multisensory cues is fundamental for perception and action and reflected by two frequently studied phenomena: multisensory integration and sensory recalibration. In the context of audio-visual spatial signals, these phenomena are exemplified by the ventriloquism effect and its aftereffect. The ventriloquism effect occurs when the perceived location of a sound is biased by a concurrent visual stimulus, while the aftereffect manifests as a recalibration of perceived sound location after exposure to spatially discrepant stimuli. The relationship between these processes-whether recalibration is a direct consequence of integration or operates independently-remains debated. We investigate the role of causal inference in these processes by examining whether trial-wise judgements about a common-cause underlying audio-visual stimuli influence both the ventriloquism effect and the immediate aftereffect. In a spatial paradigm, participants made explicit judgements about the common cause of stimulus pairs, and their influence on both perceptual biases was assessed. Results obtained across two experiments indicate that while multisensory integration is contingent on common cause judgements, the immediate recalibration effect is not. This suggests that recalibration can occur independently of the perceived commonality of the multisensory stimuli, challenging the notion that recalibration is solely a by-product of integration.

The interplay of uncertainty, relevance and learning influences auditory categorization

Auditory perception requires categorizing sound sequences, such as speech or music, into classes, such as syllables or notes. Auditory categorization depends not only on the acoustic waveform, but also on variability and uncertainty in how the listener perceives the sound - including sensory and stimulus uncertainty, the listener's estimated relevance of the particular sound to the task, and their ability to learn the past statistics of the acoustic environment. Whereas these factors have been studied in isolation, whether and how these factors interact to shape categorization remains unknown. Here, we measured human participants' performance on a multi-tone categorization task and modeled each participant's behavior using a Bayesian framework. Task-relevant tones contributed more to category choice than task-irrelevant tones, confirming that participants combined information about sensory features with task relevance. Conversely, participants' poor estimates of task-relevant tones or high-sensory uncertainty adversely impacted category choice. Learning the statistics of sound category over both short and long timescales also affected decisions, biasing the decisions toward the overrepresented category. The magnitude of this effect correlated inversely with participants' relevance estimates. Our results demonstrate that individual participants idiosyncratically weigh sensory uncertainty, task relevance, and statistics over both short and long timescales, providing a novel understanding of and a computational framework for how sensory decisions are made under several simultaneous behavioral demands.

Sustained EEG responses to rapidly unfolding stochastic sounds reflect Bayesian inferred reliability tracking

How does the brain track and process rapidly changing sensory information? Current computational accounts suggest that our sensations and decisions arise from the intricate interplay between bottom-up sensory signals and constantly changing expectations regarding the statistics of the surrounding world. A significant focus of recent research is determining which statistical properties are tracked by the brain as it monitors the rapid progression of sensory information. Here, by combining EEG (three experiments N ≥ 22 each) and computational modelling, we examined how the brain processes rapid and stochastic sound sequences that simulate key aspects of dynamic sensory environments. Passively listening participants were exposed to structured tone-pip arrangements that contained transitions between a range of stochastic patterns. Predictions were guided by a Bayesian predictive inference model. We demonstrate that listeners automatically track the statistics of unfolding sounds, even when these are irrelevant to behaviour. Transitions between sequence patterns drove a shift in the sustained EEG response. This was observed to a range of distributional statistics, and even in situations where behavioural detection of these transitions was at floor. These observations suggest that the modulation of the EEG sustained response reflects a process of belief updating within the brain. By establishing a connection between the outputs of the computational model and the observed brain responses, we demonstrate that the dynamics of these transition-related responses align with the tracking of "precision" - the confidence or reliability assigned to a predicted sensory signal - shedding light on the intricate interplay between the brain's statistical tracking mechanisms and its response dynamics.

Individual Differences in Bodily Self-Consciousness and Its Neural Basis

Bodily self-consciousness (BSC), a subject of interdisciplinary interest, refers to the awareness of one's bodily states. Previous studies have noted the existence of individual differences in BSC, while neglecting the underlying factors and neural basis of such individual differences. Considering that BSC relied on integration from both internal and external self-relevant information, we here review previous findings on individual differences in BSC through a three-level-self model, which includes interoceptive, exteroceptive, and mental self-processing. The data show that cross-level factors influenced individual differences in BSC, involving internal bodily signal perceptibility, multisensory processing principles, personal traits shaped by environment, and interaction modes that integrate multiple levels of self-processing. Furthermore, in interoceptive processing, regions like the anterior cingulate cortex and insula show correlations with different perceptions of internal sensations. For exteroception, the parietal lobe integrates sensory inputs, coordinating various BSC responses. Mental self-processing modulates differences in BSC through areas like the medial prefrontal cortex. For interactions between multiple levels of self-processing, regions like the intraparietal sulcus involve individual differences in BSC. We propose that diverse experiences of BSC can be attributed to different levels of self-processing, which moderates one's perception of their body. Overall, considering individual differences in BSC is worth amalgamating diverse methodologies for the diagnosis and treatment of some diseases.

Profile of Sensory Integration Disorders in Migraine Patients-New Perspectives of Therapy

Background: The involvement of sensory integration disorders in the pathophysiology of migraine has been suggested. This study aims to analyze the relationship between symptoms of sensory integration disorders and migraine in a broad scope, including all sensory domains, and examine its impact on migraine attacks. Methods: The study included 372 people diagnosed with migraine. The Daniel Travis Questionnaire was used to assess symptoms of sensory integration disorders and their severity across six domains. The relationships between the severity of these symptoms and headache features, as well as accompanying headache symptoms, were the subject of statistical analysis. Results: Current impairment in all sensory domains was significantly associated with headaches exacerbated by everyday life activities. A significant inverse relationship was found between the occurrence of throbbing headaches and symptoms of sensory integration disorders in terms of current sensory discrimination, current motor skills, and current emotional/social skills. Past under-responsiveness and past disturbances in emotional/social abilities were significantly associated with migraine aura. Conclusions: The severity of symptoms of sensory integration disorders affects the clinical picture of migraine. The significant association between migraine and emotional/social disorders, as well as under-responsiveness in the past, needs further research to assess whether this is a cause-and-effect relationship. There is a need for in-depth diagnostics of sensory integration disorders in migraine patients, which could be an additional target of their therapy.

Development and experience-dependence of multisensory spatial processing

Multisensory spatial processes are fundamental for efficient interaction with the world. They include not only the integration of spatial cues across sensory modalities, but also the adjustment or recalibration of spatial representations to changing cue reliabilities, crossmodal correspondences, and causal structures. Yet how multisensory spatial functions emerge during ontogeny is poorly understood. New results suggest that temporal synchrony and enhanced multisensory associative learning capabilities first guide causal inference and initiate early coarse multisensory integration capabilities. These multisensory percepts are crucial for the alignment of spatial maps across sensory systems, and are used to derive more stable biases for adult crossmodal recalibration. The refinement of multisensory spatial integration with increasing age is further promoted by the inclusion of higher-order knowledge.

Different time scales of common-cause evidence shape multisensory integration, recalibration and motor adaptation

Perceptual coherence in the face of discrepant multisensory signals is achieved via the processes of multisensory integration, recalibration and sometimes motor adaptation. These supposedly operate on different time scales, with integration reducing immediate sensory discrepancies and recalibration and motor adaptation reflecting the cumulative influence of their recent history. Importantly, whether discrepant signals are bound during perception is guided by the brains' inference of whether they originate from a common cause. When combined, these two notions lead to the hypothesis that the time scales on which integration and recalibration (or motor adaptation) operate are associated with different time scales of evidence about a common cause underlying two signals. We tested this prediction in a well-established visuo-motor paradigm, in which human participants performed visually guided hand movements. The kinematic correlation between hand and cursor movements indicates their common origin, which allowed us to manipulate the common-cause evidence by titrating this correlation. Specifically, we dissociated hand and cursor signals during individual movements while preserving their correlation across the series of movement endpoints. Following our hypothesis, this manipulation reduced integration compared with a condition in which visual and proprioceptive signals were perfectly correlated. In contrast, recalibration and motor adaption were not affected by this manipulation. This supports the notion that multisensory integration and recalibration deal with sensory discrepancies on different time scales guided by common-cause evidence: Integration is prompted by local common-cause evidence and reduces immediate discrepancies, whereas recalibration and motor adaptation are prompted by global common-cause evidence and reduce persistent discrepancies.

Cumulative multisensory discrepancies shape the ventriloquism aftereffect but not the ventriloquism bias

Multisensory integration and recalibration are two processes by which perception deals with discrepant signals. Both are often studied in the spatial ventriloquism paradigm. There, integration is probed by the presentation of discrepant audio-visual stimuli, while recalibration manifests as an aftereffect in subsequent judgements of unisensory sounds. Both biases are typically quantified against the degree of audio-visual discrepancy, reflecting the possibility that both may arise from common underlying multisensory principles. We tested a specific prediction of this: that both processes should also scale similarly with the history of multisensory discrepancies, i.e. the sequence of discrepancies in several preceding audio-visual trials. Analyzing data from ten experiments with randomly varying spatial discrepancies we confirmed the expected dependency of each bias on the immediately presented discrepancy. And in line with the aftereffect being a cumulative process, this scaled with the discrepancies presented in at least three preceding audio-visual trials. However, the ventriloquism bias did not depend on this three-trial history of multisensory discrepancies and also did not depend on the aftereffect biases in previous trials - making these two multisensory processes experimentally dissociable. These findings support the notion that the ventriloquism bias and the aftereffect reflect distinct functions, with integration maintaining a stable percept by reducing immediate sensory discrepancies and recalibration maintaining an accurate percept by accounting for consistent discrepancies.

Modeling facial perception in group context from a serial perception perspective

By utilizing statistical properties and summary statistics, the visual system can efficiently integrate perception of spatially and temporally adjacent stimuli into perception of a given target. For instance, perception of a target face can either be biased positively toward previous faces (e.g. the serial dependence effect) or be biased negatively by surrounding faces in the same trial/space (e.g. spatial ensemble averaging). However, both aspects were investigated separately. As spatial and temporal processing share the same purpose to reduce redundancy in visual processing, if one statistical processing occurs, would the statistical processing in the other domain still exist or be discarded? We investigated this question by exploring whether serial dependence of face perception (of attractiveness and averageness) survives when the changed face perception in the group context occurs. The results of Markov Chain modeling and conventional methods suggested that serial dependence (the temporal aspect) co-occurs with changed face perception in the group context (the spatial aspect). We also utilized the Hidden Markov modeling, as a new mathematical method, to model statistical processing from both domains. The results confirmed the co-occurrence of temporal effect and changed face perception in the group context for both attractiveness and averageness, suggesting potentially different spatial and temporal compression mechanisms in high-level vision. Further modeling and cluster analysis further revealed that the detailed computation of spatially and temporally adjacent faces in the attractiveness and averageness processing were similar yet different among different individuals. This work builds a bridge to understanding mathematical principles underlying changed face perception in the group context from the serial perspective.

Short-term effects of visuomotor discrepancies on multisensory integration, proprioceptive recalibration, and motor adaptation

Information about the position of our hand is provided by multisensory signals that are often not perfectly aligned. Discrepancies between the seen and felt hand position or its movement trajectory engage the processes of 1) multisensory integration, 2) sensory recalibration, and 3) motor adaptation, which adjust perception and behavioral responses to apparently discrepant signals. To foster our understanding of the coemergence of these three processes, we probed their short-term dependence on multisensory discrepancies in a visuomotor task that has served as a model for multisensory perception and motor control previously. We found that the well-established integration of discrepant visual and proprioceptive signals is tied to the immediate discrepancy and independent of the outcome of the integration of discrepant signals in immediately preceding trials. However, the strength of integration was context dependent, being stronger in an experiment featuring stimuli that covered a smaller range of visuomotor discrepancies (±15°) compared with one covering a larger range (±30°). Both sensory recalibration and motor adaptation for nonrepeated movement directions were absent after two bimodal trials with same or opposite visuomotor discrepancies. Hence our results suggest that short-term sensory recalibration and motor adaptation are not an obligatory consequence of the integration of preceding discrepant multisensory signals.NEW & NOTEWORTHY The functional relation between multisensory integration and recalibration remains debated. We here refute the notion that they coemerge in an obligatory manner and support the hypothesis that they serve distinct goals of perception.

The temporal context in bayesian models of interval timing: Recent advances and future directions

Sensory perception, motor control, and cognition necessitate reliable timing in the range of milliseconds to seconds, which implies the existence of a highly accurate timing system. Yet, partly owing to the fact that temporal processing is modulated by contextual factors, perceived time is not isomorphic to physical time. Temporal estimates exhibit regression to the mean of an interval distribution (global context) and are also affected by preceding trials (local context). Recent Bayesian models of interval timing have provided important insights regarding these observations, but questions remain as to how exposure to past intervals shapes perceived time. In this article, we provide a brief overview of Bayesian models of interval timing and their contribution to current understanding of context effects. We then proceed to highlight recent developments in the field concerning precision weighting of Bayesian evidence in both healthy timing and disease and the neurophysiological and neurochemical signatures of timing prediction errors. We further aim to bring attention to current outstanding questions for Bayesian models of interval timing, such as the likelihood conceptualization. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Bayesian causal inference: A unifying neuroscience theory

Understanding of the brain and the principles governing neural processing requires theories that are parsimonious, can account for a diverse set of phenomena, and can make testable predictions. Here, we review the theory of Bayesian causal inference, which has been tested, refined, and extended in a variety of tasks in humans and other primates by several research groups. Bayesian causal inference is normative and has explained human behavior in a vast number of tasks including unisensory and multisensory perceptual tasks, sensorimotor, and motor tasks, and has accounted for counter-intuitive findings. The theory has made novel predictions that have been tested and confirmed empirically, and recent studies have started to map its algorithms and neural implementation in the human brain. The parsimony, the diversity of the phenomena that the theory has explained, and its illuminating brain function at all three of Marr's levels of analysis make Bayesian causal inference a strong neuroscience theory. This also highlights the importance of collaborative and multi-disciplinary research for the development of new theories in neuroscience.

Perceptual Inference, Learning, and Attention in a Multisensory World

Adaptive behavior in a complex, dynamic, and multisensory world poses some of the most fundamental computational challenges for the brain, notably inference, decision-making, learning, binding, and attention. We first discuss how the brain integrates sensory signals from the same source to support perceptual inference and decision-making by weighting them according to their momentary sensory uncertainties. We then show how observers solve the binding or causal inference problem-deciding whether signals come from common causes and should hence be integrated or else be treated independently. Next, we describe the multifarious interplay between multisensory processing and attention. We argue that attentional mechanisms are crucial to compute approximate solutions to the binding problem in naturalistic environments when complex time-varying signals arise from myriad causes. Finally, we review how the brain dynamically adapts multisensory processing to a changing world across multiple timescales.

Ageing and multisensory integration: A review of the evidence, and a computational perspective

The processing of multisensory signals is crucial for effective interaction with the environment, but our ability to perform this vital function changes as we age. In the first part of this review, we summarise existing research into the effects of healthy ageing on multisensory integration. We note that age differences vary substantially with the paradigms and stimuli used: older adults often receive at least as much benefit (to both accuracy and response times) as younger controls from congruent multisensory stimuli, but are also consistently more negatively impacted by the presence of intersensory conflict. In the second part, we outline a normative Bayesian framework that provides a principled and computationally informed perspective on the key ingredients involved in multisensory perception, and how these are affected by ageing. Applying this framework to the existing literature, we conclude that changes to sensory reliability, prior expectations (together with attentional control), and decisional strategies all contribute to the age differences observed. However, we find no compelling evidence of any age-related changes to the basic inference mechanisms involved in multisensory perception.

The Neurophysiological Basis of the Trial-Wise and Cumulative Ventriloquism Aftereffects

Our senses often receive conflicting multisensory information, which our brain reconciles by adaptive recalibration. A classic example is the ventriloquism aftereffect, which emerges following both cumulative (long-term) and trial-wise exposure to spatially discrepant multisensory stimuli. Despite the importance of such adaptive mechanisms for interacting with environments that change over multiple timescales, it remains debated whether the ventriloquism aftereffects observed following trial-wise and cumulative exposure arise from the same neurophysiological substrate. We address this question by probing electroencephalography recordings from healthy humans (both sexes) for processes predictive of the aftereffect biases following the exposure to spatially offset audiovisual stimuli. Our results support the hypothesis that discrepant multisensory evidence shapes aftereffects on distinct timescales via common neurophysiological processes reflecting sensory inference and memory in parietal-occipital regions, while the cumulative exposure to consistent discrepancies additionally recruits prefrontal processes. During the subsequent unisensory trial, both trial-wise and cumulative exposure bias the encoding of the acoustic information, but do so distinctly. Our results posit a central role of parietal regions in shaping multisensory spatial recalibration, suggest that frontal regions consolidate the behavioral bias for persistent multisensory discrepancies, but also show that the trial-wise and cumulative exposure bias sound position encoding via distinct neurophysiological processes.

SIGNIFICANCE STATEMENT Our brain easily reconciles conflicting multisensory information, such as seeing an actress on screen while hearing her voice over headphones. These adaptive mechanisms exert a persistent influence on the perception of subsequent unisensory stimuli, known as the ventriloquism aftereffect. While this aftereffect emerges following trial-wise or cumulative exposure to multisensory discrepancies, it remained unclear whether both arise from a common neural substrate. We here rephrase this hypothesis using human electroencephalography recordings. Our data suggest that parietal regions involved in multisensory and spatial memory mediate the aftereffect following both trial-wise and cumulative adaptation, but also show that additional and distinct processes are involved in consolidating and implementing the aftereffect following prolonged exposure.

Using the past to estimate sensory uncertainty

To form a more reliable percept of the environment, the brain needs to estimate its own sensory uncertainty. Current theories of perceptual inference assume that the brain computes sensory uncertainty instantaneously and independently for each stimulus. We evaluated this assumption in four psychophysical experiments, in which human observers localized auditory signals that were presented synchronously with spatially disparate visual signals. Critically, the visual noise changed dynamically over time continuously or with intermittent jumps. Our results show that observers integrate audiovisual inputs weighted by sensory uncertainty estimates that combine information from past and current signals consistent with an optimal Bayesian learner that can be approximated by exponential discounting. Our results challenge leading models of perceptual inference where sensory uncertainty estimates depend only on the current stimulus. They demonstrate that the brain capitalizes on the temporal dynamics of the external world and estimates sensory uncertainty by combining past experiences with new incoming sensory signals.