Attentional Interactions Between Vision and Hearing in Event-Related Responses to Crossmodal and Conjunct Oddballs

Early audiovisual integration in target processing under continuous noise: Behavioral and EEG evidence

Multisensory integration is interconnected across various information reception. The stage and mechanism of brain response to audiovisual integration have not been fully understood. In this study, we designed audiovisual and unisensory experiments to investigate task performance and electrophysiological characteristics associated with audiovisual integration in a continuous background interference environment using materials collected from the underwater environment. Behavioral results showed that the reaction time (RT) was shorter, and the accuracy was higher in the audiovisual experiment. The cumulative distribution function (CDF) results of RT indicated that audiovisual integration supported the co-activation model. Event-related potential (ERP) results revealed shorter latency of the P1 and N1 components in the audiovisual experiment. Microstate analysis indicated that the parietal-occipital area may play a key role in audiovisual integration. Moreover, event-related spectral perturbation (ERSP) results demonstrated the critical role of low-frequency oscillation in audiovisual integration at the early stage. Our findings support the view that the beneficial effect of audiovisual integration is predominantly upon the early stage of neural information processing, including task-independent information.

EEG decoding for effects of visual joint attention training on ASD patients with interpretable and lightweight convolutional neural network

Visual joint attention, the ability to track gaze and recognize intent, plays a key role in the development of social and language skills in health humans, which is performed abnormally hard in autism spectrum disorder (ASD). The traditional convolutional neural network, EEGnet, is an effective model for decoding technology, but few studies have utilized this model to address attentional training in ASD patients. In this study, EEGNet was used to decode the P300 signal elicited by training and the saliency map method was used to visualize the cognitive properties of ASD patients during visual attention. The results showed that in the spatial distribution, the parietal lobe was the main region of classification contribution, especially for Pz electrode. In the temporal information, the time period from 300 to 500 ms produced the greatest contribution to the electroencephalogram (EEG) classification, especially around 300 ms. After training for ASD patients, the gradient contribution was significantly enhanced at 300 ms, which was effective only in social scenarios. Meanwhile, with the increase of joint attention training, the P300 latency of ASD patients gradually shifted forward in social scenarios, but this phenomenon was not obvious in non-social scenarios. Our results indicated that joint attention training could improve the cognitive ability and responsiveness of social characteristics in ASD patients.

A multimodal cortical network of sensory expectation violation revealed by fMRI

The brain is subjected to multi-modal sensory information in an environment governed by statistical dependencies. Mismatch responses (MMRs), classically recorded with EEG, have provided valuable insights into the brain's processing of regularities and the generation of corresponding sensory predictions. Only few studies allow for comparisons of MMRs across multiple modalities in a simultaneous sensory stream and their corresponding cross-modal context sensitivity remains unknown. Here, we used a tri-modal version of the roving stimulus paradigm in fMRI to elicit MMRs in the auditory, somatosensory and visual modality. Participants (N = 29) were simultaneously presented with sequences of low and high intensity stimuli in each of the three senses while actively observing the tri-modal input stream and occasionally reporting the intensity of the previous stimulus in a prompted modality. The sequences were based on a probabilistic model, defining transition probabilities such that, for each modality, stimuli were more likely to repeat (p = .825) than change (p = .175) and stimulus intensities were equiprobable (p = .5). Moreover, each transition was conditional on the configuration of the other two modalities comprising global (cross-modal) predictive properties of the sequences. We identified a shared mismatch network of modality general inferior frontal and temporo-parietal areas as well as sensory areas, where the connectivity (psychophysiological interaction) between these regions was modulated during mismatch processing. Further, we found deviant responses within the network to be modulated by local stimulus repetition, which suggests highly comparable processing of expectation violation across modalities. Moreover, hierarchically higher regions of the mismatch network in the temporo-parietal area around the intraparietal sulcus were identified to signal cross-modal expectation violation. With the consistency of MMRs across audition, somatosensation and vision, our study provides insights into a shared cortical network of uni- and multi-modal expectation violation in response to sequence regularities.