Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons

When alertness fades: Drowsiness-induced visual dominance and oscillatory recalibration in audiovisual integration

Multisensory integration allows the brain to align inputs from different sensory modalities, enhancing perception and behavior. However, transitioning into drowsiness, a state marked by decreased attentional control and altered cortical dynamics, offers a unique opportunity to examine adaptations in these multisensory processes. In this study, we investigated how drowsiness influences reaction times (RTs) and neural oscillations during audiovisual multisensory integration. Participants performed a task where auditory and visual stimuli were presented either in a coordinated manner or with temporal misalignment (visual-first or auditory-first uncoordinated conditions). Behavioral results showed that drowsiness slowed RTs overall but revealed a clear sensory dominance effect: visual-first uncoordination facilitated RTs compared to auditory-first uncoordination, reflecting vision's dominant role in recalibrating sensory conflicts. In contrast, RTs in coordinated conditions remained stable across alert and drowsy states, suggesting that multisensory redundancy compensates for reduced cortical integration during drowsiness. At the neural level, distinct patterns of oscillatory activity emerged. Alpha oscillations supported attentional realignment and temporal alignment in visual-first conditions, while Gamma oscillations were recruited during auditory-first uncoordination, reflecting heightened sensory-specific processing demands. These effects were state-dependent, becoming more pronounced during drowsiness. Our findings demonstrate that drowsiness fundamentally reshapes multisensory integration by amplifying sensory dominance mechanisms, particularly vision. Compensatory neural mechanisms involving Alpha and Gamma oscillations maintain perceptual coherence under conditions of reduced cortical interaction. These results provide critical insights into how the brain adapts to sensory conflicts during states of diminished awareness, with broader implications for performance and decision-making in real-world drowsy states.

Effect of inverted visual acceleration profile on vestibular heading perception

Visual motion is ambiguous in that it can either represent object motion or self-motion. Visual-vestibular integration is most advantageous during self-motion. The current experiment tests the hypothesis that the visual motion needs to have a motion profile consistent with the inertial motion. To test this, we examined the effect on heading perception when the visual stimulus was consistent with the inertial motion compared to an inverted visual stimulus, which was thus inconsistent with inertial motion. Twenty healthy human subjects (mean age 20 ± 3 years, 13 female) experienced 2s of translation, which they reported as left or right. A synchronized 2s visual heading was offset by 0°, ± 45°, ± 60°, or ±75°. In randomly interleaved trials, the visual motion was consistent with the inertial motion or inverted - it started at the peak velocity, decreased to zero mid-stimulus, and then accelerated back to the peak velocity at the end. When the velocity profile of the visual stimulus matched the velocity profile of inertial motion, the inertial stimulus was biased 10.0 ± 1.8° (mean ± SE) with a 45° visual offset, 8.9 ± 1.7° with a 60° offset, and 9.3° ± 2.5 ± with a 75° offset. When the visual stimulus was inverted so it was inconsistent with the inertial motion, the respective biases were 6.5 ± 1.5°, 5.6 ± 1.7°, and 5.9 ± 2.0°. The biases with the inverted stimulus were significantly smaller (p < 0.0001), demonstrating that the visual motion profile is considered in multisensory integration rather than simple trajectory endpoints.

The Effects of Deep Brain Stimulation on Balance in Parkinson's Disease as Measured Using Posturography-A Narrative Review

BACKGROUND

Postural imbalance with falls affects 80% of patients with Parkinson's disease (PD) at 10 years. Standard PD therapies (e.g., levodopa and/or deep brain stimulation-DBS) are poor at improving postural imbalance. Additionally, the mechanistic complexity of interpreting postural control is a major barrier to improving our understanding of treatment effects. In this paper, we review the effects of DBS on balance as measured using posturography. We also critically appraise the quantitative measures and analyses used in these studies.

METHODS

A literature search was performed independently by 2 researchers using the PUBMED database. Thirty-eight studies are included in this review, with DBS at the subthalamic nucleus (STN-) (n = 25), globus pallidus internus (GPi-) (n = 6), ventral intermediate nucleus (VIM)/thalamus (n = 2), and pedunculopontine nucleus (PPN) (n = 5).

RESULTS

STN- and GPi-DBS reduce static sway in PD and mitigate the increased sway from levodopa. STN-DBS impairs automatic responses to perturbations, whilst GPi-DBS has a more neutral effect. STN-DBS may promote protective strategies following external perturbations but does not improve adaptation. The evidence regarding the effects on gait initiation is less clear. Insufficient evidence exists to make conclusions regarding VIM- and PPN-DBS.

CONCLUSIONS

STN- and GPi-DBS have differing effects on posturography, which suggests site-specific and possibly non-dopaminergic mechanisms. Posturography tests should be utilised to answer specific questions regarding the mechanisms of and effects on postural control following DBS. We recommend standardising posturography measures and test conditions by expert consensus and greater long-term data collection, utilising ongoing DBS registries.

The role of explicit knowledge in compensating for a visuo-proprioceptive cue conflict

It is unclear how explicit knowledge of an externally imposed mismatch between visual and proprioceptive cues of hand position affects perceptual recalibration. The Bayesian causal inference framework might suggest such knowledge should abolish the visual and proprioceptive recalibration that occurs when individuals perceive these cues as coming from the same source (their hand), while the visuomotor adaptation literature suggests explicit knowledge of a cue conflict does not eliminate implicit compensatory processes. Here we compared visual and proprioceptive recalibration in three groups with varying levels of knowledge about the visuo-proprioceptive cue conflict. All participants estimated the position of visual, proprioceptive, or combined targets related to their left index fingertip, with a 70 mm visuo-proprioceptive offset gradually imposed. Groups 1, 2, and 3 received no information, medium information, and high information, respectively, about the offset. Information was manipulated using instructional and visual cues. All groups performed the task similarly at baseline in terms of variance, weighting, and integration. Results suggest the three groups recalibrated vision and proprioception differently, but there was no difference in variance or weighting. Participants who received only instructional cues about the mismatch (Group 2) did not recalibrate less, on average, than participants provided no information about the mismatch (Group 1). However, participants provided instructional cues and extra visual cues of their hands during the perturbation (Group 3) demonstrated significantly less recalibration than other groups. These findings are consistent with the idea that instructional cues alone are insufficient to override participants' intrinsic belief in common cause and reduce recalibration.

Crossmodal sensory neurons based on high-performance flexible memristors for human-machine in-sensor computing system

Constructing crossmodal in-sensor processing system based on high-performance flexible devices is of great significance for the development of wearable human-machine interfaces. A bio-inspired crossmodal in-sensor computing system can perform real-time energy-efficient processing of multimodal signals, alleviating data conversion and transmission between different modules in conventional chips. Here, we report a bio-inspired crossmodal spiking sensory neuron (CSSN) based on a flexible VO2 memristor, and demonstrate a crossmodal in-sensor encoding and computing system for wearable human-machine interfaces. We demonstrate excellent performance in the VO2 memristor including endurance (>1012), uniformity (0.72% for cycle-to-cycle variations and 3.73% for device-to-device variations), speed (<30 ns), and flexibility (bendable to a curvature radius of 1 mm). A flexible hardware processing system is implemented based on the CSSN, which can directly perceive and encode pressure and temperature bimodal information into spikes, and then enables the real-time haptic-feedback for human-machine interaction. We successfully construct a crossmodal in-sensor spiking reservoir computing system via the CSSNs, which can achieve dynamic objects identification with a high accuracy of 98.1% and real-time signal feedback. This work provides a feasible approach for constructing flexible bio-inspired crossmodal in-sensor computing systems for wearable human-machine interfaces.

Control and recalibration of path integration in place cells using optic flow

Hippocampal place cells are influenced by both self-motion (idiothetic) signals and external sensory landmarks as an animal navigates its environment. To continuously update a position signal on an internal 'cognitive map', the hippocampal system integrates self-motion signals over time, a process that relies on a finely calibrated path integration gain that relates movement in physical space to movement on the cognitive map. It is unclear whether idiothetic cues alone, such as optic flow, exert sufficient influence on the cognitive map to enable recalibration of path integration, or if polarizing position information provided by landmarks is essential for this recalibration. Here, we demonstrate both recalibration of path integration gain and systematic control of place fields by pure optic flow information in freely moving rats. These findings demonstrate that the brain continuously rebalances the influence of conflicting idiothetic cues to fine-tune the neural dynamics of path integration, and that this recalibration process does not require a top-down, unambiguous position signal from landmarks.

Bioinspired In-Sensor Multimodal Fusion for Enhanced Spatial and Spatiotemporal Association

Multimodal perception can capture more precise and comprehensive information compared with unimodal approaches. However, current sensory systems typically merge multimodal signals at computing terminals following parallel processing and transmission, which results in the potential loss of spatial association information and requires time stamps to maintain temporal coherence for time-series data. Here we demonstrate bioinspired in-sensor multimodal fusion, which effectively enhances comprehensive perception and reduces the level of data transfer between sensory terminal and computation units. By adopting floating gate phototransistors with reconfigurable photoresponse plasticity, we realize the agile spatial and spatiotemporal fusion under nonvolatile and volatile photoresponse modes. To realize an optimal spatial estimation, we integrate spatial information from visual-tactile signals. For dynamic events, we capture and fuse in real time spatiotemporal information from visual-audio signals, realizing a dance-music synchronization recognition task without a time-stamping process. This in-sensor multimodal fusion approach provides the potential to simplify the multimodal integration system, extending the in-sensor computing paradigm.

Conductance-based dendrites perform Bayes-optimal cue integration

A fundamental function of cortical circuits is the integration of information from different sources to form a reliable basis for behavior. While animals behave as if they optimally integrate information according to Bayesian probability theory, the implementation of the required computations in the biological substrate remains unclear. We propose a novel, Bayesian view on the dynamics of conductance-based neurons and synapses which suggests that they are naturally equipped to optimally perform information integration. In our approach apical dendrites represent prior expectations over somatic potentials, while basal dendrites represent likelihoods of somatic potentials. These are parametrized by local quantities, the effective reversal potentials and membrane conductances. We formally demonstrate that under these assumptions the somatic compartment naturally computes the corresponding posterior. We derive a gradient-based plasticity rule, allowing neurons to learn desired target distributions and weight synaptic inputs by their relative reliabilities. Our theory explains various experimental findings on the system and single-cell level related to multi-sensory integration, which we illustrate with simulations. Furthermore, we make experimentally testable predictions on Bayesian dendritic integration and synaptic plasticity.

Conditional independence as a statistical assessment of evidence integration processes

Intuitively, combining multiple sources of evidence should lead to more accurate decisions than considering single sources of evidence individually. In practice, however, the proper computation may be difficult, or may require additional data that are inaccessible. Here, based on the concept of conditional independence, we consider expressions that can serve either as recipes for integrating evidence based on limited data, or as statistical benchmarks for characterizing evidence integration processes. Consider three events, A, B, and C. We find that, if A and B are conditionally independent with respect to C, then the probability that C occurs given that both A and B are known, P(C|A, B), can be easily calculated without the need to measure the full three-way dependency between A, B, and C. This simplified approach can be used in two general ways: to generate predictions by combining multiple (conditionally independent) sources of evidence, or to test whether separate sources of evidence are functionally independent of each other. These applications are demonstrated with four computer-simulated examples, which include detecting a disease based on repeated diagnostic testing, inferring biological age based on multiple biomarkers of aging, discriminating two spatial locations based on multiple cue stimuli (multisensory integration), and examining how behavioral performance in a visual search task depends on selection histories. Besides providing a sound prescription for predicting outcomes, this methodology may be useful for analyzing experimental data of many types.

Sensory perception of rivals has trait-dependent effects on plasticity in Drosophila melanogaster

The social environment has myriad effects on individuals, altering reproduction, immune function, cognition, and aging. Phenotypic plasticity enables animals to respond to heterogeneous environments such as the social environment but requires that they assess those environments accurately. It has been suggested that combinations of sensory cues allow animals to respond rapidly and accurately to changeable environments, but it is unclear whether the same sensory inputs are required in all traits that respond to a particular environmental cue. Drosophila melanogaster males, in the presence of rival males, exhibit a consistent behavioral response by extending mating duration. However, exposure to a rival also results in a reduction in their lifespan, a phenomenon interpreted as a trade-off associated with sperm competition strategies. D. melanogaster perceive their rivals by using multiple sensory cues; interfering with at least two olfactory, auditory, or tactile cues eliminates the extension of mating duration. Here, we assessed whether these same cues were implicated in the lifespan reduction. Removal of combinations of auditory and olfactory cues removed the extended mating duration response to a rival, as previously found. However, we found that these manipulations did not alter the reduction in lifespan of males exposed to rivals or induce any changes in activity patterns, grooming, or male-male aggression. Therefore, our analysis suggests that lifespan reduction is not a cost associated with the behavioral responses to sperm competition. Moreover, this highlights the trait-specific nature of the mechanisms underlying plasticity in response to the same environmental conditions.

Cortical cellular encoding of thermotactile integration

Recent evidence suggests that primary sensory cortical regions play a role in the integration of information from multiple sensory modalities. How primary cortical neurons integrate different sources of sensory information is unclear, partly because non-primary sensory input to a cortical sensory region is often weak or modulatory. To address this question, we take advantage of the robust representation of thermal (cooling) and tactile stimuli in mouse forelimb primary somatosensory cortex (fS1). Using a thermotactile detection task, we show that the perception of threshold-level cool or tactile information is enhanced when they are presented simultaneously, compared with presentation alone. To investigate the cortical cellular correlates of thermotactile integration, we performed in vivo extracellular recordings from fS1 in awake resting and anesthetized mice during unimodal and bimodal stimulation of the forepaw. Unimodal stimulation evoked thermal- or tactile- specific excitatory and inhibitory responses of fS1 neurons. The most prominent features of combined thermotactile stimulation are the recruitment of unimodally silent fS1 neurons, non-linear integration features, and response dynamics that favor longer response durations with additional spikes. Together, we identify quantitative and qualitative changes in cortical encoding that may underlie the improvement in perception of thermotactile surfaces during haptic exploration.

The hidden structure of consciousness

According to Loorits, if we want consciousness to be explained in terms of natural sciences, we should be able to analyze its seemingly non-structural aspects, like qualia, in structural terms. However, the studies conducted over the last three decades do not seem to be able to bridge the explanatory gap between physical phenomena and phenomenal experience. One possible way to bridge the explanatory gap is to seek the structure of consciousness within consciousness itself, through a phenomenal analysis of the qualitative aspects of experience. First, this analysis leads us to identify the explanandum concerning the simplest forms of experience not in qualia but in the unitary set of qualities found in early vision. Second, it leads us to hypothesize that consciousness is also made up of non-apparent parts, and that there exists a hidden structure of consciousness. This structure, corresponding to a simple early visual experience, is constituted by a Hierarchy of Spatial Belongings nested within each other. Each individual Spatial Belonging is formed by a primary content and a primary space. The primary content can be traced in the perceptibility of the contents we can distinguish in the phenomenal field. The primary space is responsible for the perceptibility of the content and is not perceptible in itself. However, the phenomenon I refer to as subtraction of visibility allows us to characterize it as phenomenally negative. The hierarchical relationships between Spatial Belongings can ensure the qualitative nature of components of perceptual organization, such as object, background, and detail. The hidden structure of consciousness presents aspects that are decidedly counterintuitive compared to our idea of phenomenal experience. However, on the one hand, the Hierarchy of Spatial Belongings can explain the qualities of early vision and their appearance as a unitary whole, while on the other hand, it might be more easily explicable in terms of brain organization. In other words, the hidden structure of consciousness can be considered a bridge structure which, placing itself at an intermediate level between experience and physical properties, can contribute to bridging the explanatory gap.

Triple dissociation of visual, auditory and motor processing in mouse primary visual cortex

Primary sensory cortices respond to crossmodal stimuli-for example, auditory responses are found in primary visual cortex (V1). However, it remains unclear whether these responses reflect sensory inputs or behavioral modulation through sound-evoked body movement. We address this controversy by showing that sound-evoked activity in V1 of awake mice can be dissociated into auditory and behavioral components with distinct spatiotemporal profiles. The auditory component began at approximately 27 ms, was found in superficial and deep layers and originated from auditory cortex. Sound-evoked orofacial movements correlated with V1 neural activity starting at approximately 80-100 ms and explained auditory frequency tuning. Visual, auditory and motor activity were expressed by different laminar profiles and largely segregated subsets of neuronal populations. During simultaneous audiovisual stimulation, visual representations remained dissociable from auditory-related and motor-related activity. This three-fold dissociability of auditory, motor and visual processing is central to understanding how distinct inputs to visual cortex interact to support vision.

More Than the Sum of Its Parts: Visual-Tactile Integration in the Behaving Rat

We experience the world by constantly integrating cues from multiple modalities to form unified sensory percepts. Once familiar with multimodal properties of an object, we can recognize it regardless of the modality involved. In this chapter we will examine the case of a visual-tactile orientation categorization experiment in rats. We will explore the involvement of the cerebral cortex in recognizing objects through multiple sensory modalities. In the orientation categorization task, rats learned to examine and judge the orientation of a raised, black and white grating using touch, vision, or both. Their multisensory performance was better than the predictions of linear models for cue combination, indicating synergy between the two sensory channels. Neural recordings made from a candidate associative cortical area, the posterior parietal cortex (PPC), reflected the principal neuronal correlates of the behavioral results: PPC neurons encoded both graded information about the object and categorical information about the animal's decision. Intriguingly single neurons showed identical responses under each of the three modality conditions providing a substrate for a neural circuit in the cortex that is involved in modality-invariant processing of objects.

Multisensory Integration and Causal Inference in Typical and Atypical Populations

Multisensory perception is critical for effective interaction with the environment, but human responses to multisensory stimuli vary across the lifespan and appear changed in some atypical populations. In this review chapter, we consider multisensory integration within a normative Bayesian framework. We begin by outlining the complex computational challenges of multisensory causal inference and reliability-weighted cue integration, and discuss whether healthy young adults behave in accordance with normative Bayesian models. We then compare their behaviour with various other human populations (children, older adults, and those with neurological or neuropsychiatric disorders). In particular, we consider whether the differences seen in these groups are due only to changes in their computational parameters (such as sensory noise or perceptual priors), or whether the fundamental computational principles (such as reliability weighting) underlying multisensory perception may also be altered. We conclude by arguing that future research should aim explicitly to differentiate between these possibilities.

From Multisensory Integration to Multisensory Decision-Making

Organisms live in a dynamic environment in which sensory information from multiple sources is ever changing. A conceptually complex task for the organisms is to accumulate evidence across sensory modalities and over time, a process known as multisensory decision-making. This is a new concept, in terms of that previous researches have been largely conducted in parallel disciplines. That is, much efforts have been put either in sensory integration across modalities using activity summed over a duration of time, or in decision-making with only one sensory modality that evolves over time. Recently, a few studies with neurophysiological measurements emerge to study how different sensory modality information is processed, accumulated, and integrated over time in decision-related areas such as the parietal or frontal lobes in mammals. In this review, we summarize and comment on these studies that combine the long-existed two parallel fields of multisensory integration and decision-making. We show how the new findings provide insight into our understanding about neural mechanisms mediating multisensory information processing in a more complete way.

Brain-inspired bodily self-perception model for robot rubber hand illusion

The core of bodily self-consciousness involves perceiving ownership of one's body. A central question is how body illusions like the rubber hand illusion (RHI) occur. Existing theoretical models still lack satisfying computational explanations from connectionist perspectives, especially for how the brain encodes body perception and generates illusions from neuronal interactions. Moreover, the integration of disability experiments is also neglected. Here, we integrate biological findings of bodily self-consciousness to propose a brain-inspired bodily self-perception model by which perceptions of bodily self are autonomously constructed without any supervision signals. We successfully validated the model with six RHI experiments and a disability experiment on an iCub humanoid robot and simulated environments. The results show that our model can not only well-replicate the behavioral and neural data of monkeys in biological experiments but also reasonably explain the causes and results of RHI at the neuronal level, thus contributing to the revelation of mechanisms underlying RHI.

How deep is the brain? The shallow brain hypothesis

Deep learning and predictive coding architectures commonly assume that inference in neural networks is hierarchical. However, largely neglected in deep learning and predictive coding architectures is the neurobiological evidence that all hierarchical cortical areas, higher or lower, project to and receive signals directly from subcortical areas. Given these neuroanatomical facts, today's dominance of cortico-centric, hierarchical architectures in deep learning and predictive coding networks is highly questionable; such architectures are likely to be missing essential computational principles the brain uses. In this Perspective, we present the shallow brain hypothesis: hierarchical cortical processing is integrated with a massively parallel process to which subcortical areas substantially contribute. This shallow architecture exploits the computational capacity of cortical microcircuits and thalamo-cortical loops that are not included in typical hierarchical deep learning and predictive coding networks. We argue that the shallow brain architecture provides several critical benefits over deep hierarchical structures and a more complete depiction of how mammalian brains achieve fast and flexible computational capabilities.

Bayesian encoding and decoding as distinct perspectives on neural coding

The Bayesian brain hypothesis is one of the most influential ideas in neuroscience. However, unstated differences in how Bayesian ideas are operationalized make it difficult to draw general conclusions about how Bayesian computations map onto neural circuits. Here, we identify one such unstated difference: some theories ask how neural circuits could recover information about the world from sensory neural activity (Bayesian decoding), whereas others ask how neural circuits could implement inference in an internal model (Bayesian encoding). These two approaches require profoundly different assumptions and lead to different interpretations of empirical data. We contrast them in terms of motivations, empirical support and relationship to neural data. We also use a simple model to argue that encoding and decoding models are complementary rather than competing. Appreciating the distinction between Bayesian encoding and Bayesian decoding will help to organize future work and enable stronger empirical tests about the nature of inference in the brain.

Conditional independence as a statistical assessment of evidence integration processes

Intuitively, combining multiple sources of evidence should lead to more accurate decisions than considering single sources of evidence individually. In practice, however, the proper computation may be difficult, or may require additional data that are inaccessible. Here, based on the concept of conditional independence, we consider expressions that can serve either as recipes for integrating evidence based on limited data, or as statistical benchmarks for characterizing evidence integration processes. Consider three events, A , B , and C . We find that, if A and B are conditionally independent with respect to C , then the probability that C occurs given that both A and B are known, P C | A , B , can be easily calculated without the need to measure the full three-way dependency between A , B , and C . This simplified approach can be used in two general ways: to generate predictions by combining multiple (conditionally independent) sources of evidence, or to test whether separate sources of evidence are functionally independent of each other. These applications are demonstrated with four computer-simulated examples, which include detecting a disease based on repeated diagnostic testing, inferring biological age based on multiple biomarkers of aging, discriminating two spatial locations based on multiple cue stimuli (multisensory integration), and examining how behavioral performance in a visual search task depends on selection histories. Besides providing a sound prescription for predicting outcomes, this methodology may be useful for analyzing experimental data of many types.

Statistically Optimal Cue Integration During Human Spatial Navigation

In 2007, Cheng and colleagues published their influential review wherein they analyzed the literature on spatial cue interaction during navigation through a Bayesian lens, and concluded that models of optimal cue integration often applied in psychophysical studies could explain cue interaction during navigation. Since then, numerous empirical investigations have been conducted to assess the degree to which human navigators are optimal when integrating multiple spatial cues during a variety of navigation-related tasks. In the current review, we discuss the literature on human cue integration during navigation that has been published since Cheng et al.'s original review. Evidence from most studies demonstrate optimal navigation behavior when humans are presented with multiple spatial cues. However, applications of optimal cue integration models vary in their underlying assumptions (e.g., uninformative priors and decision rules). Furthermore, cue integration behavior depends in part on the nature of the cues being integrated and the navigational task (e.g., homing versus non-home goal localization). We discuss the implications of these models and suggest directions for future research.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Visuo-vestibular heading perception: a model system to study multi-sensory decision making

Integrating noisy signals across time as well as sensory modalities, a process named multi-sensory decision making (MSDM), is an essential strategy for making more accurate and sensitive decisions in complex environments. Although this field is just emerging, recent extraordinary works from different perspectives, including computational theory, psychophysical behaviour and neurophysiology, begin to shed new light onto MSDM. In the current review, we focus on MSDM by using a model system of visuo-vestibular heading. Combining well-controlled behavioural paradigms on virtual-reality systems, single-unit recordings, causal manipulations and computational theory based on spiking activity, recent progress reveals that vestibular signals contain complex temporal dynamics in many brain regions, including unisensory, multi-sensory and sensory-motor association areas. This challenges the brain for cue integration across time and sensory modality such as optic flow which mainly contains a motion velocity signal. In addition, new evidence from the higher-level decision-related areas, mostly in the posterior and frontal/prefrontal regions, helps revise our conventional thought on how signals from different sensory modalities may be processed, converged, and moment-by-moment accumulated through neural circuits for forming a unified, optimal perceptual decision. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

How 'visual' is the visual cortex? The interactions between the visual cortex and other sensory, motivational and motor systems as enabling factors for visual perception

The definition of the visual cortex is primarily based on the evidence that lesions of this area impair visual perception. However, this does not exclude that the visual cortex may process more information than of retinal origin alone, or that other brain structures contribute to vision. Indeed, research across the past decades has shown that non-visual information, such as neural activity related to reward expectation and value, locomotion, working memory and other sensory modalities, can modulate primary visual cortical responses to retinal inputs. Nevertheless, the function of this non-visual information is poorly understood. Here we review recent evidence, coming primarily from studies in rodents, arguing that non-visual and motor effects in visual cortex play a role in visual processing itself, for instance disentangling direct auditory effects on visual cortex from effects of sound-evoked orofacial movement. These findings are placed in a broader framework casting vision in terms of predictive processing under control of frontal, reward- and motor-related systems. In contrast to the prevalent notion that vision is exclusively constructed by the visual cortical system, we propose that visual percepts are generated by a larger network-the extended visual system-spanning other sensory cortices, supramodal areas and frontal systems. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

How the brain controls decision making in a multisensory world

Sensory systems evolved to provide the organism with information about the environment to guide adaptive behaviour. Neuroscientists and psychologists have traditionally considered each sense independently, a legacy of Aristotle and a natural consequence of their distinct physical and anatomical bases. However, from the point of view of the organism, perception and sensorimotor behaviour are fundamentally multi-modal; after all, each modality provides complementary information about the same world. Classic studies revealed much about where and how sensory signals are combined to improve performance, but these tended to treat multisensory integration as a static, passive, bottom-up process. It has become increasingly clear how this approach falls short, ignoring the interplay between perception and action, the temporal dynamics of the decision process and the many ways by which the brain can exert top-down control of integration. The goal of this issue is to highlight recent advances on these higher order aspects of multisensory processing, which together constitute a mainstay of our understanding of complex, natural behaviour and its neural basis. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Multisensory causal inference is feature-specific, not object-based

Multisensory integration depends on causal inference about the sensory signals. We tested whether implicit causal-inference judgements pertain to entire objects or focus on task-relevant object features. Participants in our study judged virtual visual, haptic and visual-haptic surfaces with respect to two features-slant and roughness-against an internal standard in a two-alternative forced-choice task. Modelling of participants' responses revealed that the degree to which their perceptual judgements were based on integrated visual-haptic information varied unsystematically across features. For example, a perceived mismatch between visual and haptic roughness would not deter the observer from integrating visual and haptic slant. These results indicate that participants based their perceptual judgements on a feature-specific selection of information, suggesting that multisensory causal inference proceeds not at the object level but at the level of single object features. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Proprioceptive uncertainty promotes the rubber hand illusion

Body ownership is the multisensory perception of a body as one's own. Recently, the emergence of body ownership illusions like the visuotactile rubber hand illusion has been described by Bayesian causal inference models in which the observer computes the probability that visual and tactile signals come from a common source. Given the importance of proprioception for the perception of one's body, proprioceptive information and its relative reliability should impact this inferential process. We used a detection task based on the rubber hand illusion where participants had to report whether the rubber hand felt like their own or not. We manipulated the degree of asynchrony of visual and tactile stimuli delivered to the rubber hand and the real hand under two levels of proprioceptive noise using tendon vibration applied to the lower arm's antagonist extensor and flexor muscles. As hypothesized, the probability of the emergence of the rubber hand illusion increased with proprioceptive noise. Moreover, this result, well fitted by a Bayesian causal inference model, was best described by a change in the a priori probability of a common cause for vision and touch. These results offer new insights into how proprioceptive uncertainty shapes the multisensory perception of one's own body.

Coherent mapping of position and head direction across auditory and visual cortex

Neurons in primary visual cortex (V1) may not only signal current visual input but also relevant contextual information such as reward expectancy and the subject's spatial position. Such contextual representations need not be restricted to V1 but could participate in a coherent mapping throughout sensory cortices. Here, we show that spiking activity coherently represents a location-specific mapping across auditory cortex (AC) and lateral, secondary visual cortex (V2L) of freely moving rats engaged in a sensory detection task on a figure-8 maze. Single-unit activity of both areas showed extensive similarities in terms of spatial distribution, reliability, and position coding. Importantly, reconstructions of subject position based on spiking activity displayed decoding errors that were correlated between areas. Additionally, we found that head direction, but not locomotor speed or head angular velocity, was an important determinant of activity in AC and V2L. By contrast, variables related to the sensory task cues or to trial correctness and reward were not markedly encoded in AC and V2L. We conclude that sensory cortices participate in coherent, multimodal representations of the subject's sensory-specific location. These may provide a common reference frame for distributed cortical sensory and motor processes and may support crossmodal predictive processing.

Bimodular continuous attractor neural networks with static and moving stimuli

We investigated the dynamical behaviors of bimodular continuous attractor neural networks, each processing a modality of sensory input and interacting with each other. We found that when bumps coexist in both modules, the position of each bump is shifted towards the other input when the intermodular couplings are excitatory and is shifted away when inhibitory. When one intermodular coupling is excitatory while another is moderately inhibitory, temporally modulated population spikes can be generated. On further increase of the inhibitory coupling, momentary spikes will emerge. In the regime of bump coexistence, bump heights are primarily strengthened by excitatory intermodular couplings, but there is a lesser weakening effect due to a bump being displaced from the direct input. When bimodular networks serve as decoders of multisensory integration, we extend the Bayesian framework to show that excitatory and inhibitory couplings encode attractive and repulsive priors, respectively. At low disparity, the bump positions decode the posterior means in the Bayesian framework, whereas at high disparity, multiple steady states exist. In the regime of multiple steady states, the less stable state can be accessed if the input causing the more stable state arrives after a sufficiently long delay. When one input is moving, the bump in the corresponding module is pinned when the moving stimulus is weak, unpinned at intermediate stimulus strength, and tracks the input at strong stimulus strength, and the stimulus strengths for these transitions increase with the velocity of the moving stimulus. These results are important to understanding multisensory integration of static and dynamic stimuli.

Explaining human interactions on the road by large-scale integration of computational psychological theory

When humans share space in road traffic, as drivers or as vulnerable road users, they draw on their full range of communicative and interactive capabilities. Much remains unknown about these behaviors, but they need to be captured in models if automated vehicles are to coexist successfully with human road users. Empirical studies of human road user behavior implicate a large number of underlying cognitive mechanisms, which taken together are well beyond the scope of existing computational models. Here, we note that for all of these putative mechanisms, computational theories exist in different subdisciplines of psychology, for more constrained tasks. We demonstrate how these separate theories can be generalized from abstract laboratory paradigms and integrated into a computational framework for modeling human road user interaction, combining Bayesian perception, a theory of mind regarding others' intentions, behavioral game theory, long-term valuation of action alternatives, and evidence accumulation decision-making. We show that a model with these assumptions-but not simpler versions of the same model-can account for a number of previously unexplained phenomena in naturalistic driver-pedestrian road-crossing interactions, and successfully predicts interaction outcomes in an unseen data set. Our modeling results contribute to demonstrating the real-world value of the theories from which we draw, and address calls in psychology for cumulative theory-building, presenting human road use as a suitable setting for work of this nature. Our findings also underscore the formidable complexity of human interaction in road traffic, with strong implications for the requirements to set on development and testing of vehicle automation.

Altered visual conscious awareness in patients with vestibular dysfunctions; a cross-sectional observation study

BACKGROUND

Patients with vestibular dysfunctions often experience visual-induced symptoms. Here we asked whether such visual dependence can be related to alterations in visual conscious awareness in these patients.

METHODS

To measure visual conscious awareness, we used the effect of motion-induced blindness (MIB,) in which the perceptual awareness of the visual stimulus alternates despite its unchanged physical characteristics. In this phenomenon, a salient visual target spontaneously disappears and subsequently reappears from visual perception when presented against a moving visual background. The number of perceptual switches during the experience of the MIB stimulus was measured for 120 s in 15 healthy controls, 15 patients with vestibular migraine, 15 patients with benign positional paroxysmal vertigo (BPPV) and 15 with migraine without vestibular symptoms.

RESULTS

Patients with vestibular dysfunctions (i.e., both vestibular migraine and BPPV) exhibited increased perceptual fluctuations during MIB compared to healthy controls and migraine patients without vertigo. In VM patients, those with more severe symptoms exhibited higher fluctuations of visual awareness (i.e., positive correlation), whereas, in BPPV patients, those with more severe symptoms had lower fluctuations of visual awareness (i.e., negative correlation).

IMPLICATIONS

Taken together, these findings show that fluctuations of visual awareness are linked to the severity of visual-induced symptoms in patients with vestibular dysfunctions, and distinct pathophysiological mechanisms may mediate visual vertigo in peripheral versus central vestibular dysfunctions.

Multisensory alarm to benefit alarm identification and decrease workload: a feasibility study

The poor design of conventional auditory medical alarms has contributed to alarm desensitization, and eventually, alarm fatigue in medical personnel. This study tested a novel multisensory alarm system which aims to help medical personnel better interpret and respond to alarm annunciation during periods of high cognitive load such as those found within intensive care units. We tested a multisensory alarm that combined auditory and vibrotactile cues to convey alarm type, alarm priority, and patient identity. Testing was done in three phases: control (conventional auditory), Half (limited multisensory alarm), and Full (complete multisensory alarm). Participants (N = 19, undergraduates) identified alarm type, priority, and patient identity (patient 1 or 2) using conventional and multisensory alarms, while simultaneously completing a cognitively demanding task. Performance was based on reaction time (RT) and identification accuracy of alarm type and priority. Participants also reported their perceived workload. RT was significantly faster for the Control phase (p < 0.05). Participant performance in identifying alarm type, priority, and patient did not differ significantly between the three phase conditions (p = 0.87, 0.37, and 0.14 respectively). The Half multisensory phase produced the lowest mental demand, temporal demand, and overall perceived workload score. These data suggest that implementation of a multisensory alarm with alarm and patient information may decrease perceived workload without significant changes in alarm identification performance. Additionally, a ceiling effect may exist for multisensory stimuli, with only part of an alarm benefitting from multisensory integration.

Multisensory learning binds neurons into a cross-modal memory engram

Associating multiple sensory cues with objects and experience is a fundamental brain process that improves object recognition and memory performance. However, neural mechanisms that bind sensory features during learning and augment memory expression are unknown. Here we demonstrate multisensory appetitive and aversive memory in Drosophila. Combining colours and odours improved memory performance, even when each sensory modality was tested alone. Temporal control of neuronal function revealed visually selective mushroom body Kenyon cells (KCs) to be required for enhancement of both visual and olfactory memory after multisensory training. Voltage imaging in head-fixed flies showed that multisensory learning binds activity between streams of modality-specific KCs so that unimodal sensory input generates a multimodal neuronal response. Binding occurs between regions of the olfactory and visual KC axons, which receive valence-relevant dopaminergic reinforcement, and is propagated downstream. Dopamine locally releases GABAergic inhibition to permit specific microcircuits within KC-spanning serotonergic neurons to function as an excitatory bridge between the previously 'modality-selective' KC streams. Cross-modal binding thereby expands the KCs representing the memory engram for each modality into those representing the other. This broadening of the engram improves memory performance after multisensory learning and permits a single sensory feature to retrieve the memory of the multimodal experience.

Sensory fusion in the hoverfly righting reflex

We study how falling hoverflies use sensory cues to trigger appropriate roll righting behavior. Before being released in a free fall, flies were placed upside-down with their legs contacting the substrate. The prior leg proprioceptive information about their initial orientation sufficed for the flies to right themselves properly. However, flies also use visual and antennal cues to recover faster and disambiguate sensory conflicts. Surprisingly, in one of the experimental conditions tested, hoverflies flew upside-down while still actively flapping their wings. In all the other conditions, flies were able to right themselves using two roll dynamics: fast ([Formula: see text]50ms) and slow ([Formula: see text]110ms) in the presence of consistent and conflicting cues, respectively. These findings suggest that a nonlinear sensory integration of the three types of sensory cues occurred. A ring attractor model was developed and discussed to account for this cue integration process.

Factors influencing clinical outcome in vestibular neuritis - A focussed review and reanalysis of prospective data

Following vestibular neuritis (VN), long term prognosis is not dependent on the magnitude of the residual peripheral function as measured with either caloric or the video head-impulse test. Rather, recovery is determined by a combination of visuo-vestibular (visual dependence), psychological (anxiety) and vestibular perceptual factors. Our recent research in healthy individuals has also revealed a strong association between the degree of lateralisation of vestibulo-cortical processing and gating of vestibular signals, anxiety and visual dependence. In the context of several functional brain changes occurring in the interaction between visual, vestibular and emotional cortices, which underpin the aforementioned psycho-physiological features in patients with VN, we re-examined our previously published findings focusing on additional factors impacting long term clinical outcome and function. These included: (i) the role of concomitant neuro-otological dysfunction (i.e. migraine and benign paroxysmal positional vertigo (BPPV)) and (ii) the degree to which brain lateralisation of vestibulo-cortical processing influences gating of vestibular function in the acute stage. We found that migraine and BPPV interfere with symptomatic recovery following VN. That is, dizziness handicap at short-term recovery stage was significantly predicted by migraine (r = 0.523, n = 28, p = .002), BPPV (r = 0.658, n = 31, p < .001) and acute visual dependency (r = 0.504, n = 28, p = .003). Moreover, dizziness handicap in the long-term recovery stage continued to be predicted by migraine (r = 0.640, n = 22, p = .001), BPPV (r = 0.626, n = 24, p = .001) and acute visual dependency (r = 0.667, n = 22, p < .001). Furthermore, surrogate measures of vestibulo-cortical lateralisation were predictive of the amount of cortical suppression exerted over vestibular thresholds. That is, in right-sided VN patients, we observed a positive correlation between visual dependence and acute ipsilesional oculomotor thresholds (R² 0.497; p < .001), but not contralateral thresholds (R² 0.017: p > .05). In left-sided VN patients, we observed a negative correlation between visual dependence and ipsilesional oculomotor thresholds (R² 0.459; p < .001), but not for contralateral thresholds (R² 0.013; p > .05). To surmise, our findings illustrate that in VN, neuro-otological co-morbidities retard recovery, and that measures of the peripheral vestibular system are an aggregate of residual function and cortically mediated gating of vestibular input.

Being active over one's own motion: Considering predictive mechanisms in self-motion perception

Self-motion perception is a key element guiding pilots' behavior. Its importance is mostly revealed when impaired, leading in most cases to spatial disorientation which is still today a major factor of accidents occurrence. Self-motion perception is known as mainly based on visuo-vestibular integration and can be modulated by the physical properties of the environment with which humans interact. For instance, several studies have shown that the respective weight of visual and vestibular information depends on their reliability. More recently, it has been suggested that the internal state of an operator can also modulate multisensory integration. Interestingly, the systems' automation can interfere with this internal state through the loss of the intentional nature of movements (i.e., loss of agency) and the modulation of associated predictive mechanisms. In this context, one of the new challenges is to better understand the relationship between automation and self-motion perception. The present review explains how linking the concepts of agency and self-motion is a first approach to address this issue.

Walking humans and running mice: perception and neural encoding of optic flow during self-motion

Locomotion produces full-field optic flow that often dominates the visual motion inputs to an observer. The perception of optic flow is in turn important for animals to guide their heading and interact with moving objects. Understanding how locomotion influences optic flow processing and perception is therefore essential to understand how animals successfully interact with their environment. Here, we review research investigating how perception and neural encoding of optic flow are altered during self-motion, focusing on locomotion. Self-motion has been found to influence estimation and sensitivity for optic flow speed and direction. Nonvisual self-motion signals also increase compensation for self-driven optic flow when parsing the visual motion of moving objects. The integration of visual and nonvisual self-motion signals largely follows principles of Bayesian inference and can improve the precision and accuracy of self-motion perception. The calibration of visual and nonvisual self-motion signals is dynamic, reflecting the changing visuomotor contingencies across different environmental contexts. Throughout this review, we consider experimental research using humans, non-human primates and mice. We highlight experimental challenges and opportunities afforded by each of these species and draw parallels between experimental findings. These findings reveal a profound influence of locomotion on optic flow processing and perception across species. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Multisensory integration of orally-sourced gustatory and olfactory inputs to the posterior piriform cortex in awake rats

Flavour refers to the sensory experience of food, which is a combination of sensory inputs sourced from multiple modalities during consumption, including taste and odour. Previous work has demonstrated that orally-sourced taste and odour cues interact to determine perceptual judgements of flavour stimuli, although the underlying cellular- and circuit-level neural mechanisms remain unknown. We recently identified a region of the piriform olfactory cortex in rats that responds to both taste and odour stimuli. Here, we investigated how converging taste and odour inputs to this area interact to affect single neuron responsiveness ensemble coding of flavour identity. To accomplish this, we recorded spiking activity from ensembles of single neurons in the posterior piriform cortex (pPC) in awake, tasting rats while delivering taste solutions, odour solutions and taste + odour mixtures directly into the oral cavity. Our results show that taste and odour inputs evoke highly selective, temporally-overlapping responses in multisensory pPC neurons. Comparing responses to mixtures and their unisensory components revealed that taste and odour inputs interact in a non-linear manner to produce unique response patterns. Taste input enhances trial-by-trial decoding of odour identity from small ensembles of simultaneously recorded neurons. Together, these results demonstrate that taste and odour inputs to pPC interact in complex, non-linear ways to form amodal flavour representations that enhance identity coding. KEY POINTS: Experience of food involves taste and smell, although how information from these different senses is combined by the brain to create our sense of flavour remains unknown. We recorded from small groups of neurons in the olfactory cortex of awake rats while they consumed taste solutions, odour solutions and taste + odour mixtures. Taste and smell solutions evoke highly selective responses. When presented in a mixture, taste and smell inputs interacted to alter responses, resulting in activation of unique sets of neurons that could not be predicted by the component responses. Synergistic interactions increase discriminability of odour representations. The olfactory cortex uses taste and smell to create new information representing multisensory flavour identity.

Active self-motion control and the role of agency under ambiguity

Purpose

Self-motion perception is a key factor in daily behaviours such as driving a car or piloting an aircraft. It is mainly based on visuo-vestibular integration, whose weighting mechanisms are modulated by the reliability properties of sensory inputs. Recently, it has been shown that the internal state of the operator can also modulate multisensory integration and may sharpen the representation of relevant inputs. In line with the concept of agency, it thus appears relevant to evaluate the impact of being in control of our own action on self-motion perception.

Methodology

Here, we tested two conditions of motion control (active/manual trigger versus passive/ observer condition), asking participants to discriminate between two consecutive longitudinal movements by identifying the larger displacement (displacement of higher intensity). We also tested motion discrimination under two levels of ambiguity by applying acceleration ratios that differed from our two "standard" displacements (i.e., 3 s; 0.012 m.s^-2 and 0.030 m.s^-2).

Results

We found an effect of control condition, but not of the level of ambiguity on the way participants perceived the standard displacement, i.e., perceptual bias (Point of Subjective Equality; PSE). Also, we found a significant effect of interaction between the active condition and the level of ambiguity on the ability to discriminate between displacements, i.e., sensitivity (Just Noticeable Difference; JND).

Originality

Being in control of our own motion through a manual intentional trigger of self-displacement maintains overall motion sensitivity when ambiguity increases.

Temporal scaling and computing time in neural circuits: Should we stop watching the clock and look for its gears?

Timing underlies a variety of functions, from walking to perceiving causality. Neural timing models typically fall into one of two categories-"ramping" and "population-clock" theories. According to ramping models, individual neurons track time by gradually increasing or decreasing their activity as an event approaches. To time different intervals, ramping neurons adjust their slopes, ramping steeply for short intervals and vice versa. In contrast, according to "population-clock" models, multiple neurons track time as a group, and each neuron can fire nonlinearly. As each neuron changes its rate at each point in time, a distinct pattern of activity emerges across the population. To time different intervals, the brain learns the population patterns that coincide with key events. Both model categories have empirical support. However, they often differ in plausibility when applied to certain behavioral effects. Specifically, behavioral data indicate that the timing system has a rich computational capacity, allowing observers to spontaneously compute novel intervals from previously learned ones. In population-clock theories, population patterns map to time arbitrarily, making it difficult to explain how different patterns can be computationally combined. Ramping models are viewed as more plausible, assuming upstream circuits can set the slope of ramping neurons according to a given computation. Critically, recent studies suggest that neurons with nonlinear firing profiles often scale to time different intervals-compressing for shorter intervals and stretching for longer ones. This "temporal scaling" effect has led to a hybrid-theory where, like a population-clock model, population patterns encode time, yet like a ramping neuron adjusting its slope, the speed of each neuron's firing adapts to different intervals. Here, we argue that these "relative" population-clock models are as computationally plausible as ramping theories, viewing population-speed and ramp-slope adjustments as equivalent. Therefore, we view identifying these "speed-control" circuits as a key direction for evaluating how the timing system performs computations. Furthermore, temporal scaling highlights that a key distinction between different neural models is whether they propose an absolute or relative time-representation. However, we note that several behavioral studies suggest the brain processes both scales, cautioning against a dichotomy.

Conscious awareness of a visuo-proprioceptive mismatch: Effect on cross-sensory recalibration

The brain estimates hand position using vision and position sense (proprioception). The relationship between visual and proprioceptive estimates is somewhat flexible: visual information about the index finger can be spatially displaced from proprioceptive information, resulting in cross-sensory recalibration of the visual and proprioceptive unimodal position estimates. According to the causal inference framework, recalibration occurs when the unimodal estimates are attributed to a common cause and integrated. If separate causes are perceived, then recalibration should be reduced. Here we assessed visuo-proprioceptive recalibration in response to a gradual visuo-proprioceptive mismatch at the left index fingertip. Experiment 1 asked how frequently a 70 mm mismatch is consciously perceived compared to when no mismatch is present, and whether awareness is linked to reduced visuo-proprioceptive recalibration, consistent with causal inference predictions. However, conscious offset awareness occurred rarely. Experiment 2 tested a larger displacement, 140 mm, and asked participants about their perception more frequently, including at 70 mm. Experiment 3 confirmed that participants were unbiased at estimating distances in the 2D virtual reality display. Results suggest that conscious awareness of the mismatch was indeed linked to reduced cross-sensory recalibration as predicted by the causal inference framework, but this was clear only at higher mismatch magnitudes (70–140 mm). At smaller offsets (up to 70 mm), conscious perception of an offset may not override unconscious belief in a common cause, perhaps because the perceived offset magnitude is in range of participants’ natural sensory biases. These findings highlight the interaction of conscious awareness with multisensory processes in hand perception.

A theoretical model and empirical analysis of university library readers' spatial cognition

Purpose

The practice of renovation and construction of university libraries is flourishing, but how to attract readers to use the library is an issue that urgently needs to be explored. Spatial cognition is a subjective judgment of a person's tendency to take action in the future and implies behavioral intention. Based on the sensory–image–cognition relationship, a theoretical model of university library readers' spatial cognition is conducted, and the influencing factors and mechanisms of spatial cognition are explored based on empirical data to provide theoretical references for spatial practices in university libraries.

Design/methodology/approach

A visual and art-based mental map approach is introduced based on a questionnaire survey. The questionnaire is mainly used for the specific evaluation of spatial use and the breakdown of the detailed elements, while the mental map method is mainly used for the evaluation of readers' spatial cognition. Relevant empirical data are collected from the library of the Zhejiang University of Technology.

Findings

The results indicate that readers' spatial sensory experience and mental imagery have positive effects on readers' behavior via the mediator spatial cognition, readers' spatial sensory experience and mental imagery have a positive effect on readers' spatial cognition and spatial cognition has a significant effect on readers' behavior.

Originality/value

The main contribution of this study is to construct a theoretical model of readers' spatial cognition and to explore the factors that have an impact on spatial cognition and the influence of cognition on behavior. This provides a more rational and in-depth thinking paradigm for the study of university library space and provides theoretical references for library practice.

Cortical Mechanisms of Multisensory Linear Self-motion Perception

Accurate self-motion perception, which is critical for organisms to survive, is a process involving multiple sensory cues. The two most powerful cues are visual (optic flow) and vestibular (inertial motion). Psychophysical studies have indicated that humans and nonhuman primates integrate the two cues to improve the estimation of self-motion direction, often in a statistically Bayesian-optimal way. In the last decade, single-unit recordings in awake, behaving animals have provided valuable neurophysiological data with a high spatial and temporal resolution, giving insight into possible neural mechanisms underlying multisensory self-motion perception. Here, we review these findings, along with new evidence from the most recent studies focusing on the temporal dynamics of signals in different modalities. We show that, in light of new data, conventional thoughts about the cortical mechanisms underlying visuo-vestibular integration for linear self-motion are challenged. We propose that different temporal component signals may mediate different functions, a possibility that requires future studies.

What is neurorepresentationalism? From neural activity and predictive processing to multi-level representations and consciousness

This review provides an update on Neurorepresentationalism, a theoretical framework that defines conscious experience as multimodal, situational survey and explains its neural basis from brain systems constructing best-guess representations of sensations originating in our environment and body [1]. It posits that conscious experience is characterized by five essential hallmarks: (i) multimodal richness, (ii) situatedness and immersion, (iii) unity and integration, (iv) dynamics and stability, and (v) intentionality. Consciousness is furthermore proposed to have a biological function, framed by the contrast between reflexes and habits (not requiring consciousness) versus goal-directed, planned behavior (requiring multimodal, situational survey). Conscious experience is therefore understood as a sensorily rich, spatially encompassing representation of body and environment, while we nevertheless have the impression of experiencing external reality directly. Contributions to understanding neural mechanisms underlying consciousness are derived from models for predictive processing, which are trained in an unsupervised manner, do not necessarily require overt action, and have been extended to deep neural networks. Even with predictive processing in place, however, the question remains why this type of neural network activity would give rise to phenomenal experience. Here, I propose to tackle the Hard Problem with the concept of multi-level representations which emergently give rise to multimodal, spatially wide superinferences corresponding to phenomenal experiences. Finally, Neurorepresentationalism is compared to other neural theories of consciousness, and its implications for defining indicators of consciousness in animals, artificial intelligence devices and immobile or unresponsive patients with disorders of consciousness are discussed.

Do Congruent Auditory Stimuli Facilitate Visual Search in Dynamic Environments? An Experimental Study Based on Multisensory Interaction

The purpose of this study was to investigate the cue congruency effect of auditory stimuli during visual search in dynamic environments. Twenty-eight participants were recruited to conduct a visual search experiment. The experiment applied auditory stimuli to understand whether they could facilitate visual search in different types of background. Additionally, target location and target orientation were manipulated to clarify their influences on visual search. Target location was related to horizontal visual search and target orientation was associated with visual search for an inverted target. The results regarding dynamic backgrounds reported that target-congruent auditory stimuli could speed up the visual search time. In addition, the cue congruency effect of auditory stimuli was critical for the center of the visual display but declined for the edge, indicating the inhibition of horizontal visual search behavior. Moreover, few improvements accompanying auditory stimuli were provided for the visual detection of non-inverted and inverted targets. The findings of this study suggested developing multisensory interaction with head-mounted displays, such as augmented reality glasses, in real life.

Expanding Aesthetics

This paper seeks to expand traditional aesthetic dimensions of design beyond the limits of human capability in order to encompass other species' sensory modalities. To accomplish this, the idea of inclusivity is extended beyond human cultural and personal identities and needs, to embrace multi-species experiences of places, events and interactions in the world. This involves drawing together academic perspectives from ecology, neuroscience, anthropology, philosophy and interaction design, as well as exploring artistic perspectives and demonstrating how these different frames of reference can inspire and complement each other. This begins with a rationale for the existence of non-human aesthetics, followed by an overview of existing research into non-human aesthetic dimensions. Novel aesthetic categories are proposed and the challenge of how to include non-human aesthetic sensibility in design is discussed.

Influence of Sensory Conflict on Perceived Timing of Passive Rotation in Virtual Reality

Integration of incoming sensory signals from multiple modalities is central in the determination of self-motion perception. With the emergence of consumer virtual reality (VR), it is becoming increasingly common to experience a mismatch in sensory feedback regarding motion when using immersive displays. In this study, we explored whether introducing various discrepancies between the vestibular and visual motion would influence the perceived timing of self-motion. Participants performed a series of temporal-order judgements between an auditory tone and a passive whole-body rotation on a motion platform accompanied by visual feedback using a virtual environment generated through a head-mounted display. Sensory conflict was induced by altering the speed and direction by which the movement of the visual scene updated relative to the observer's physical rotation. There were no differences in perceived timing of the rotation without vision, with congruent visual feedback and when the speed of the updating of the visual motion was slower. However, the perceived timing was significantly further from zero when the direction of the visual motion was incongruent with the rotation. These findings demonstrate the potential interaction between visual and vestibular signals in the temporal perception of self-motion. Additionally, we recorded cybersickness ratings and found that sickness severity was significantly greater when visual motion was present and incongruent with the physical motion. This supports previous research regarding cybersickness and the sensory conflict theory, where a mismatch between the visual and vestibular signals may lead to a greater likelihood for the occurrence of sickness symptoms.

Direct eye gaze enhances the ventriloquism effect

The “ventriloquism effect” describes an illusory phenomenon where the perceived location of an auditory stimulus is pulled toward the location of a visual stimulus. Ventriloquists use this phenomenon to create an illusion where an inanimate puppet is perceived to speak. Ventriloquists use the expression and suppression of their own and the puppet’s mouth movements as well the direction of their respective eye gaze to maximize the illusion. While the puppet’s often exaggerated mouth movements have been demonstrated to enhance the ventriloquism effect, the contribution of direct eye gaze remains unknown. In Experiment 1, participants viewed an image of a person’s face while hearing a temporally synchronous recording of a voice originating from different locations on the azimuthal plane. The eyes of the facial stimuli were either looking directly at participants or were closed. Participants were more likely to misperceive the location of a range of voice locations as coming from a central position when the eye gaze of the facial stimuli were directed toward them. Thus, direct gaze enhances the ventriloquist effect by attracting participants’ perception of the voice locations toward the location of the face. In an exploratory analysis, we furthermore found no evidence for an other-race effect between White vs Asian listeners. In Experiment 2, we replicated the effect of direct eye gaze on the ventriloquism effect, also showing that faces per se attract perceived sound locations compared with audio-only sound localization. Showing a modulation of the ventriloquism effect by socially-salient eye gaze information thus adds to previous findings reporting top-down influences on this effect.

Cognitive, Systems, and Computational Neurosciences of the Self in Motion

Navigating by path integration requires continuously estimating one's self-motion. This estimate may be derived from visual velocity and/or vestibular acceleration signals. Importantly, these senses in isolation are ill-equipped to provide accurate estimates, and thus visuo-vestibular integration is an imperative. After a summary of the visual and vestibular pathways involved, the crux of this review focuses on the human and theoretical approaches that have outlined a normative account of cue combination in behavior and neurons, as well as on the systems neuroscience efforts that are searching for its neural implementation. We then highlight a contemporary frontier in our state of knowledge: understanding how velocity cues with time-varying reliabilities are integrated into an evolving position estimate over prolonged time periods. Further, we discuss how the brain builds internal models inferring when cues ought to be integrated versus segregated-a process of causal inference. Lastly, we suggest that the study of spatial navigation has not yet addressed its initial condition: self-location.