Spatially uninformative sounds increase sensitivity for visual motion change

Audiovisual correspondence facilitates the visual search for biological motion

Hearing synchronous sounds may facilitate the visual search for the concurrently changed visual targets. Evidence for this audiovisual attentional facilitation effect mainly comes from studies using artificial stimuli with relatively simple temporal dynamics, indicating a stimulus-driven mechanism whereby synchronous audiovisual cues create a salient object to capture attention. Here, we investigated the crossmodal attentional facilitation effect on biological motion (BM), a natural, biologically significant stimulus with complex and unique dynamic profiles. We found that listening to temporally congruent sounds, compared with incongruent sounds, enhanced the visual search for BM targets. More intriguingly, such a facilitation effect requires the presence of distinctive local motion cues (especially the accelerations in feet movement) independent of the global BM configuration, suggesting a crossmodal mechanism triggered by specific biological features to enhance the salience of BM signals. These findings provide novel insights into how audiovisual integration boosts attention to biologically relevant motion stimuli and extend the function of a proposed life detection system driven by local kinematics of BM to multisensory life motion perception.

The Beep-Speed Illusion Cannot Be Explained With a Simple Selection Bias

An object appears to move at higher speed than another equally fast object when brief nonspatial tones coincide with its changes in motion direction. We refer to this phenomenon as the beep-speed illusion (Meyerhoff et al., 2022, Cognition, 219, 104978). The origin of this illusion is unclear; however, attentional explanations and potential biases in the response behavior appear to be plausible candidates. In this report, we test a simple bias explanation that emerges from the way the dependent variable is assessed. As the participants have to indicate the faster of the two objects, participants possibly always indicate the audio-visually synchronized object in situations of perceptual uncertainty. Such a response behavior potentially could explain the observed shift in perceived speed. We therefore probed the magnitude of the beep-speed illusion when the participants indicated either the object that appeared to move faster or the object that appeared to move slower. If a simple selection bias would explain the beep-speed illusion, the response pattern should be inverted with the instruction to indicate the slower object. However, contrary to this bias hypothesis, illusion emerged indistinguishably under both instructions. Therefore, simple selection biases cannot explain the beep-speed illusion.

Exploring the effectiveness of auditory, visual, and audio-visual sensory cues in a multiple object tracking environment

Maintaining object correspondence among multiple moving objects is an essential task of the perceptual system in many everyday life activities. A substantial body of research has confirmed that observers are able to track multiple target objects amongst identical distractors based only on their spatiotemporal information. However, naturalistic tasks typically involve the integration of information from more than one modality, and there is limited research investigating whether auditory and audio-visual cues improve tracking. In two experiments, we asked participants to track either five target objects or three versus five target objects amongst similarly indistinguishable distractor objects for 14 s. During the tracking interval, the target objects bounced occasionally against the boundary of a centralised orange circle. A visual cue, an auditory cue, neither or both coincided with these collisions. Following the motion interval, the participants were asked to indicate all target objects. Across both experiments and both set sizes, our results indicated that visual and auditory cues increased tracking accuracy although visual cues were more effective than auditory cues. Audio-visual cues, however, did not increase tracking performance beyond the level of purely visual cues for both high and low load conditions. We discuss the theoretical implications of our findings for multiple object tracking as well as for the principles of multisensory integration.

The beep-speed illusion: Non-spatial tones increase perceived speed of visual objects in a forced-choice paradigm

We introduce a new audio-visual illusion revealing the interplay between audio-visual integration and selective visual attention. This illusion involves two simultaneously moving objects that change their motion trajectory occasionally, but only the direction changes of one object are accompanied by spatially uninformative tones. We observed a selective increase in perceived object speed of the audio-visually synchronized object by measuring the point of subjective equality in a forced-choice paradigm. The illusory increase in perceived speed of the audio-visually synchronized object persisted when preventing eye movements. Using temporally matched color changes of the synchronized object also increased the perceived speed. Yet, using color changes of a surrounding frame instead of tones had no effect on perceived speed ruling out simple alertness explanations. Thus, in contrast to coinciding tones, visual coincidences only elicit illusory increases in perceived speed when the coincidence provided spatial information. Taken together, our pattern of results suggests that audio-visual synchrony attracts visual attention toward the coinciding visual object, leading to an increase in speed-perception and thus shedding new light on the interplay between attention and multisensory feature integration. We discuss potential limitations such as the choice of paradigm and outline prospective research question to further investigate the effect of audio-visual integration on perceived object speed.

Effects of Audiovisual Presentations on Visual Localization Errors: One or Several Multisensory Mechanisms?

The present study examines the extent to which temporal and spatial properties of sound modulate visual motion processing in spatial localization tasks. Participants were asked to locate the place at which a moving visual target unexpectedly vanished. Across different tasks, accompanying sounds were factorially varied within subjects as to their onset and offset times and/or positions relative to visual motion. Sound onset had no effect on the localization error. Sound offset was shown to modulate the perceived visual offset location, both for temporal and spatial disparities. This modulation did not conform to attraction toward the timing or location of the sounds but, demonstrably in the case of temporal disparities, to bimodal enhancement instead. Favorable indications to a contextual effect of audiovisual presentations on interspersed visual-only trials were also found. The short sound-leading offset asynchrony had equivalent benefits to audiovisual offset synchrony, suggestive of the involvement of early-level mechanisms, constrained by a temporal window, at these conditions. Yet, we tentatively hypothesize that the whole of the results and how they compare with previous studies requires the contribution of additional mechanisms, including learning-detection of auditory-visual associations and cross-sensory spread of endogenous attention.

Visuo-perceptual capabilities predict sensitivity for coinciding auditory and visual transients in multi-element displays

In order to obtain a coherent representation of the outside world, auditory and visual information are integrated during human information processing. There is remarkable variance among observers in the capability to integrate auditory and visual information. Here, we propose that visuo-perceptual capabilities predict detection performance for audiovisually coinciding transients in multi-element displays due to severe capacity limitations in audiovisual integration. In the reported experiment, we employed an individual differences approach in order to investigate this hypothesis. Therefore, we measured performance in a useful-field-of-view task that captures detection performance for briefly presented stimuli across a large perceptual field. Furthermore, we measured sensitivity for visual direction changes that coincide with tones within the same participants. Our results show that individual differences in visuo-perceptual capabilities predicted sensitivity for the presence of audiovisually synchronous events among competing visual stimuli. To ensure that this correlation does not stem from superordinate factors, we also tested performance in an unrelated working memory task. Performance in this task was independent of sensitivity for the presence of audiovisually synchronous events. Our findings strengthen the proposed link between visuo-perceptual capabilities and audiovisual integration. The results also suggest that basic visuo-perceptual capabilities provide the basis for the subsequent integration of auditory and visual information.

Microdream neurophenomenology

Nightly transitions into sleep are usually uneventful and transpire in the blink of an eye. But in the laboratory these transitions afford a unique view of how experience is transformed from the perceptually grounded consciousness of wakefulness to the hallucinatory simulations of dreaming. The present review considers imagery in the sleep-onset transition-"microdreams" in particular-as an alternative object of study to dreaming as traditionally studied in the sleep lab. A focus on microdream phenomenology has thus far proven fruitful in preliminary efforts to (i) develop a classification for dreaming's core phenomenology (the "oneiragogic spectrum"), (ii) establish a structure for assessing dreaming's multiple memory inputs ("multi-temporal memory sources"), (iii) further Silberer's project for classifying sleep-onset images in relation to waking cognition by revealing two new imagery types ("autosensory imagery," "exosensory imagery"), and (iv) embed a potential understanding of microdreaming processes in a larger explanatory framework ("multisensory integration approach"). Such efforts may help resolve outstanding questions about dream neurophysiology and dreaming's role in memory consolidation during sleep but may also advance discovery in the neuroscience of consciousness more broadly.

Spatiotemporal Relationships among Audiovisual Stimuli Modulate Auditory Facilitation of Visual Target Discrimination

Sensory information is multimodal; through audiovisual interaction, task-irrelevant auditory stimuli tend to speed response times and increase visual perception accuracy. However, mechanisms underlying these performance enhancements have remained unclear. We hypothesize that task-irrelevant auditory stimuli might provide reliable temporal and spatial cues for visual target discrimination and behavioral response enhancement. Using signal detection theory, the present study investigated the effects of spatiotemporal relationships on auditory facilitation of visual target discrimination. Three experiments were conducted where an auditory stimulus maintained reliable temporal and/or spatial relationships with visual target stimuli. Results showed that perception sensitivity (d') to visual target stimuli was enhanced only when a task-irrelevant auditory stimulus maintained reliable spatiotemporal relationships with a visual target stimulus. When only reliable spatial or temporal information was contained, perception sensitivity was not enhanced. These results suggest that reliable spatiotemporal relationships between visual and auditory signals are required for audiovisual integration during a visual discrimination task, most likely due to a spread of attention. These results also indicate that auditory facilitation of visual target discrimination follows from late-stage cognitive processes rather than early stage sensory processes.

The Curious Incident of Attention in Multisensory Integration: Bottom-up vs. Top-down

The role attention plays in our experience of a coherent, multisensory world is still controversial. On the one hand, a subset of inputs may be selected for detailed processing and multisensory integration in a top-down manner, i.e., guidance of multisensory integration by attention. On the other hand, stimuli may be integrated in a bottom-up fashion according to low-level properties such as spatial coincidence, thereby capturing attention. Moreover, attention itself is multifaceted and can be describedviaboth top-down and bottom-up mechanisms. Thus, the interaction between attention and multisensory integration is complex and situation-dependent. The authors of this opinion paper are researchers who have contributed to this discussion from behavioural, computational and neurophysiological perspectives. We posed a series of questions, the goal of which was to illustrate the interplay between bottom-up and top-down processes in various multisensory scenarios in order to clarify the standpoint taken by each author and with the hope of reaching a consensus. Although divergence of viewpoint emerges in the current responses, there is also considerable overlap: In general, it can be concluded that the amount of influence that attention exerts on MSI depends on the current task as well as prior knowledge and expectations of the observer. Moreover stimulus properties such as the reliability and salience also determine how open the processing is to influences of attention.

The interactions of multisensory integration with endogenous and exogenous attention

Stimuli from multiple sensory organs can be integrated into a coherent representation through multiple phases of multisensory processing; this phenomenon is called multisensory integration. Multisensory integration can interact with attention. Here, we propose a framework in which attention modulates multisensory processing in both endogenous (goal-driven) and exogenous (stimulus-driven) ways. Moreover, multisensory integration exerts not only bottom-up but also top-down control over attention. Specifically, we propose the following: (1) endogenous attentional selectivity acts on multiple levels of multisensory processing to determine the extent to which simultaneous stimuli from different modalities can be integrated; (2) integrated multisensory events exert top-down control on attentional capture via multisensory search templates that are stored in the brain; (3) integrated multisensory events can capture attention efficiently, even in quite complex circumstances, due to their increased salience compared to unimodal events and can thus improve search accuracy; and (4) within a multisensory object, endogenous attention can spread from one modality to another in an exogenous manner.

Predictive coding and multisensory integration: an attentional account of the multisensory mind

Multisensory integration involves a host of different cognitive processes, occurring at different stages of sensory processing. Here I argue that, despite recent insights suggesting that multisensory interactions can occur at very early latencies, the actual integration of individual sensory traces into an internally consistent mental representation is dependent on both top–down and bottom–up processes. Moreover, I argue that this integration is not limited to just sensory inputs, but that internal cognitive processes also shape the resulting mental representation. Studies showing that memory recall is affected by the initial multisensory context in which the stimuli were presented will be discussed, as well as several studies showing that mental imagery can affect multisensory illusions. This empirical evidence will be discussed from a predictive coding perspective, in which a central top–down attentional process is proposed to play a central role in coordinating the integration of all these inputs into a coherent mental representation.

The attentional window modulates capture by audiovisual events

Visual search is markedly improved when a target color change is synchronized with a spatially non-informative auditory signal. This "pip and pop" effect is an automatic process as even a distractor captures attention when accompanied by a tone. Previous studies investigating visual attention have indicated that automatic capture is susceptible to the size of the attentional window. The present study investigated whether the pip and pop effect is modulated by the extent to which participants divide their attention across the visual field We show that participants were better in detecting a synchronized audiovisual event when they divided their attention across the visual field relative to a condition in which they focused their attention. We argue that audiovisual capture is reduced under focused conditions relative to distributed settings.