Sounds can boost the awareness of visual events through attention without cross-modal integration

Semantic audio-visual congruence modulates visual sensitivity to biological motion across awareness levels

Whether cross-modal interaction requires conscious awareness of multisensory information or whether it can occur in the absence of awareness, is still an open question. Here, we investigated if sounds can enhance detection sensitivity of semantically matching visual stimuli at varying levels of visual awareness. We presented biological motion stimuli of human actions (walking, rowing, sawing) during dynamic continuous flash suppression (CFS) to 80 participants and measured the effect of co-occurring, semantically matching or non-matching action sounds on visual sensitivity (d'). By individually thresholding stimulus contrast, we distinguished participants who detected motion either above or at chance level. Participants who reliably detected visual motion above chance showed higher sensitivity to upright versus inverted biological motion across all experimental conditions. In contrast, participants detecting visual motion at chance level, i.e. during successful suppression, demonstrated this upright advantage exclusively during trials with semantically congruent sounds. Across the whole sample, the impact of sounds on visual sensitivity increased as participants' visual detection performance decreased, revealing a systematic trade-off between auditory and visual processing. Our findings suggest that semantic congruence between auditory and visual information can selectively modulate biological motion perception when visual awareness is minimal or absent, while more robust visual signals enable perception of biological motion independent of auditory input. Thus, semantically congruent sounds may impact visual representations as a function of the level of visual awareness.

The impact of frequency-specific music stimulation on consciousness in patients with disorders of consciousness

Objective

This study aimed to evaluate the effects of frequency-specific music stimulation on the awareness and brain connectivity in patients with disorders of consciousness (DOC).

Methods

A total of 25 DOC patients were exposed to auditory stimulation through music at varying frequencies (low: <0.5 Hz, middle: 0.5 Hz-3.5 kHz, high: >3.5 kHz). Brain responses were assessed using Functional Near-Infrared Spectroscopy (fNIRS) to monitor objective markers of brain activity. The analysis focused on effective connectivity and coupling strength (CS) values in response to different frequency stimulations, targeting regions such as the motor and somatosensory cortices.

Results

The mean age of the patients was 49.4 years, with an average coma duration of 1.96 months. While no significant differences were observed in general brain arousal across different frequency stimuli, notable differences in effective connectivity were identified. High-frequency stimulation resulted in significantly higher CS values in the right primary motor cortex (p < 0.05), while middle-frequency stimulation showed significant effects in the right primary somatosensory cortex (p = 0.016).

Conclusion

The findings suggest that middle- and high-frequency music stimulation may enhance effective connectivity in specific brain regions, potentially contributing to the rehabilitation of DOC patients. These results indicate that frequency-specific music could stimulate motor networks and areas associated with autobiographical memory, highlighting its therapeutic potential in promoting awareness in this patient population.

Irrelevant sights and sounds require spatial suppression: ERP evidence

Both real-world experience and behavioral laboratory research suggest that entirely irrelevant stimuli (distractors) can interfere with a primary task. However, it is as yet unknown whether such interference reflects competition for spatial attention - indeed, prominent theories of attention predict that this should not be the case. Whilst electrophysiological indices of spatial capture and spatial suppression have been well-investigated, experiments have primarily utilized distractors which share a degree of task-relevance with targets, and are limited to the visual domain. The present research measured behavioral and ERP responses to test the ability of salient yet entirely task-irrelevant visual and auditory distractors to compete for spatial attention during a visual task, while also testing for potentially enhanced competition from multisensory distractors. Participants completed a central letter search task, while ignoring lateralized visual (e.g., image of a dog), auditory (e.g., barking), or multisensory (e.g., image + barking) distractors. Results showed that visual and multisensory distractors elicited a P_D component indicative of active lateralized suppression. We also establish for the first time an auditory analog of the P_D component, the P_AD , elicited by auditory and multisensory distractors. Interestingly, there was no evidence to suggest enhanced ability of multisensory distractors to compete for attentional selection, despite previous proposals of a "special" saliency status for such items. Our findings hence suggest that irrelevant multisensory and unisensory distractors are similarly capable of eliciting a spatial "attend-to-me" signal - a precursor of spatial attentional capture - but at least in the present data set did not elicit full spatial attentional capture.

Magnetoencephalography recordings reveal the neural mechanisms of auditory contributions to improved visual detection

Sounds enhance the detection of visual stimuli while concurrently biasing an observer's decisions. To investigate the neural mechanisms that underlie such multisensory interactions, we decoded time-resolved Signal Detection Theory sensitivity and criterion parameters from magneto-encephalographic recordings of participants that performed a visual detection task. We found that sounds improved visual detection sensitivity by enhancing the accumulation and maintenance of perceptual evidence over time. Meanwhile, criterion decoding analyses revealed that sounds induced brain activity patterns that resembled the patterns evoked by an actual visual stimulus. These two complementary mechanisms of audiovisual interplay differed in terms of their automaticity: Whereas the sound-induced enhancement in visual sensitivity depended on participants being actively engaged in a detection task, we found that sounds activated the visual cortex irrespective of task demands, potentially inducing visual illusory percepts. These results challenge the classical assumption that sound-induced increases in false alarms exclusively correspond to decision-level biases.

Tactile information affects alternating visual percepts during binocular rivalry using naturalistic objects

INTRODUCTION

Past studies have provided evidence that the effects of tactile stimulation on binocular rivalry are mediated by primitive features (orientation and spatial frequency) common in vision and touch. In this study, we examined whether such effects on binocular rivalry can be obtained through the roughness of naturalistic objects. In three experiments, the total dominant time of visual percepts of two objects was measured under binocular rivalry when participants touched one of the objects.

RESULT

In Experiment 1, the total dominant time for the image of artificial turf and bathmat was prolonged by congruent tactile stimulation and shortened by incongruent tactile stimulation. In Experiment 2, we used the same stimuli but rotated their visual images in opposite directions. The dominant time for either image was prolonged by congruent tactile stimulation. In Experiment 3, we used different types of stimuli, smooth marble and rough fabric, and noted significant effects of the congruent and incongruent tactile stimulation on the dominant time of visual percepts.

CONCLUSION

These three experiments demonstrated that visuo-tactile interaction on binocular rivalry can be mediated by roughness.

Modality-specific and modality-general electrophysiological correlates of visual and auditory awareness: Evidence from a bimodal ERP experiment

To date, most studies on the event-related potential (ERP) correlates of conscious perception have examined a single perceptual modality. We compared electrophysiological correlates of visual and auditory awareness in the same experiment to test whether there are modality-specific and modality-general correlates of conscious perception. We used near threshold stimulation and analyzed event-related potentials in response to aware and unaware trials in visual, auditory and bimodal conditions. The results showed modality-specific negative amplitude correlates of conscious perception between 200 and 300 ms after stimulus onset. A combination of these auditory and visual awareness negativities was observed in the bimodal condition. A later positive amplitude difference, whose early part was modality-specific, possibly reflecting access to global workspace, and later part shared modality-general features, possibly indicating higher level cognitive processing involving the decision making, was also observed.

Sound gaps boost emotional audiovisual integration independent of attention: Evidence from an ERP study

The emotion discrimination paradigm was adopted to study the effect of interrupted sound on visual emotional processing under different attentional states. There were two experiments: Experiment 1: judging facial expressions (explicit task), Experiment 2: judging the position of a bar (implicit task). In Experiment 1, ERP results showed that there was a sound gap accelerating the effect of P1 present only under neutral faces. In Experiment 2, the accelerating effect (P1) existed regardless of the emotional condition. Combining two experiments, P1 findings suggest that sound gap enhances bottom-up attention. The N170 and late positive component (LPC) were found to be regulated by emotion face in both experiments, with fear over the neutral. Comparing the two experiments, the explicit task induced a larger LPC than the implicit task. Overall, sound gaps boosted the audiovisual integration by bottom-up attention in early integration, while cognitive expectations led to top-down attention in late stages.

Combining Visual Contrast Information with Sound Can Produce Faster Decisions

Pieron’s and Chocholle’s seminal psychophysical work predicts that human response time to information relative to visual contrast and/or sound frequency decreases when contrast intensity or sound frequency increases. The goal of this study is to bring to the forefront the ability of individuals to use visual contrast intensity and sound frequency in combination for faster perceptual decisions of relative depth (“nearer”) in planar (2D) object configurations based on physical variations in luminance contrast. Computer controlled images with two abstract patterns of varying contrast intensity, one on the left and one on the right, preceded or not by a pure tone of varying frequency, were shown to healthy young humans in controlled experimental sequences. Their task (two-alternative, forced-choice) was to decide as quickly as possible which of two patterns, the left or the right one, in a given image appeared to “stand out as if it were nearer” in terms of apparent (subjective) visual depth. The results showed that the combinations of varying relative visual contrast with sounds of varying frequency exploited here produced an additive effect on choice response times in terms of facilitation, where a stronger visual contrast combined with a higher sound frequency produced shorter forced-choice response times. This new effect is predicted by audio-visual probability summation.

Effects of Multimodal Association on Ambiguous Perception in Binocular Rivalry

When two eyes view dissimilar images, an observer typically reports ambiguous perception called binocular rivalry where the subjective perception fluctuates between the two inputs. This perceptual instability is often comprised of exclusive dominance of each image and a transition state called piecemeal state where the two images are intermingled in patchwork manner. Herein, we investigated the effects of multimodal association of sensory congruent pair, arbitrary pair, and reverse pair on piecemeal state in order to see how each level of association affects the ambiguous perception during binocular rivalry. To induce the multisensory associations, we designed a matching task with audiovisual feedback where subjects were required to respond according to given pairing rules. We found that explicit audiovisual associations can substantially affect the piecemeal state during binocular rivalry and that this congruency effect that reduces the amount of visual ambiguity originates primarily from explicit audiovisual association training rather than common sensory features. Furthermore, when one information is associated with multiple information, recent and preexisting associations work collectively to influence the perceptual ambiguity during rivalry. Our findings show that learned multimodal association directly affects the temporal dynamics of ambiguous perception during binocular rivalry by modulating not only the exclusive dominance but also the piecemeal state in a systematic manner.

Multisensory enhancement of attention depends on whether you are already paying attention

Multisensory stimuli are argued to capture attention more effectively than unisensory stimuli due to their ability to elicit a super-additive neuronal response. However, behavioural evidence for enhanced multisensory attentional capture is mixed. Furthermore, the notion of multisensory enhancement of attention conflicts with findings suggesting that multisensory integration may itself be dependent upon top-down attention. The present research resolves this discrepancy by examining how both endogenous attentional settings and the availability of attentional capacity modulate capture by multisensory stimuli. Across a series of four studies, two measures of attentional capture were used which vary in their reliance on endogenous attention: facilitation and distraction. Perceptual load was additionally manipulated to determine whether multisensory stimuli are still able to capture attention when attention is occupied by a demanding primary task. Multisensory stimuli presented as search targets were consistently detected faster than unisensory stimuli regardless of perceptual load, although they are nevertheless subject to load modulation. In contrast, task irrelevant multisensory stimuli did not cause greater distraction than unisensory stimuli, suggesting that the enhanced attentional status of multisensory stimuli may be mediated by the availability of endogenous attention. Implications for multisensory alerts in practical settings such as driving and aviation are discussed, namely that these may be advantageous during demanding tasks, but may be less suitable to signaling unexpected events.

The Time Course of Audio-Visual Phoneme Identification: a High Temporal Resolution Study

Speech unfolds in time and, as a consequence, its perception requires temporal integration. Yet, studies addressing audio-visual speech processing have often overlooked this temporal aspect. Here, we address the temporal course of audio-visual speech processing in a phoneme identification task using a Gating paradigm. We created disyllabic Spanish word-like utterances (e.g., /pafa/, /paθa/, …) from high-speed camera recordings. The stimuli differed only in the middle consonant (/f/, /θ/, /s/, /r/, /g/), which varied in visual and auditory saliency. As in classical Gating tasks, the utterances were presented in fragments of increasing length (gates), here in 10 ms steps, for identification and confidence ratings. We measured correct identification as a function of time (at each gate) for each critical consonant in audio, visual and audio-visual conditions, and computed the Identification Point and Recognition Point scores. The results revealed that audio-visual identification is a time-varying process that depends on the relative strength of each modality (i.e., saliency). In some cases, audio-visual identification followed the pattern of one dominant modality (either A or V), when that modality was very salient. In other cases, both modalities contributed to identification, hence resulting in audio-visual advantage or interference with respect to unimodal conditions. Both unimodal dominance and audio-visual interaction patterns may arise within the course of identification of the same utterance, at different times. The outcome of this study suggests that audio-visual speech integration models should take into account the time-varying nature of visual and auditory saliency.