1
|
Li Y, Noguchi Y. The role of beta band phase resetting in audio-visual temporal order judgment. Cogn Neurodyn 2025; 19:28. [PMID: 39823079 PMCID: PMC11735826 DOI: 10.1007/s11571-024-10183-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2024] [Revised: 10/26/2024] [Accepted: 12/13/2024] [Indexed: 01/19/2025] Open
Abstract
The integration of auditory and visual stimuli is essential for effective language processing and social perception. The present study aimed to elucidate the mechanisms underlying audio-visual (A-V) integration by investigating the temporal dynamics of multisensory regions in the human brain. Specifically, we evaluated inter-trial coherence (ITC), a neural index indicative of phase resetting, through scalp electroencephalography (EEG) while participants performed a temporal-order judgment task that involved auditory (beep, A) and visual (flash, V) stimuli. The results indicated that ITC phase resetting was greater for bimodal (A + V) stimuli compared to unimodal (A or V) stimuli in the posterior temporal region, which resembled the responses of A-V multisensory neurons reported in animal studies. Furthermore, the ITC got lager as the stimulus-onset asynchrony (SOA) between beep and flash approached 0 ms. This enhancement in ITC was most clearly seen in the beta band (13-30 Hz). Overall, these findings highlight the importance of beta rhythm activity in the posterior temporal cortex for the detection of synchronous audiovisual stimuli, as assessed through temporal order judgment tasks. Supplementary Information The online version contains supplementary material available at 10.1007/s11571-024-10183-0.
Collapse
Affiliation(s)
- Yueying Li
- Department of Psychology, Graduate School of Humanities, Kobe University, 1-1 Rokkodai- cho, Nada, Kobe, 657-8501 Japan
| | - Yasuki Noguchi
- Department of Psychology, Graduate School of Humanities, Kobe University, 1-1 Rokkodai- cho, Nada, Kobe, 657-8501 Japan
| |
Collapse
|
2
|
Parise CV, Ernst MO. Multisensory integration operates on correlated input from unimodal transient channels. eLife 2025; 12:RP90841. [PMID: 39841060 PMCID: PMC11753778 DOI: 10.7554/elife.90841] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/23/2025] Open
Abstract
Audiovisual information reaches the brain via both sustained and transient input channels, representing signals' intensity over time or changes thereof, respectively. To date, it is unclear to what extent transient and sustained input channels contribute to the combined percept obtained through multisensory integration. Based on the results of two novel psychophysical experiments, here we demonstrate the importance of the transient (instead of the sustained) channel for the integration of audiovisual signals. To account for the present results, we developed a biologically inspired, general-purpose model for multisensory integration, the multisensory correlation detectors, which combines correlated input from unimodal transient channels. Besides accounting for the results of our psychophysical experiments, this model could quantitatively replicate several recent findings in multisensory research, as tested against a large collection of published datasets. In particular, the model could simultaneously account for the perceived timing of audiovisual events, multisensory facilitation in detection tasks, causality judgments, and optimal integration. This study demonstrates that several phenomena in multisensory research that were previously considered unrelated, all stem from the integration of correlated input from unimodal transient channels.
Collapse
Affiliation(s)
- Cesare V Parise
- Cognitive Neuroscience Department, University of Bielefeld (DE)BielefeldGermany
- Department of Psychology, University of LiverpoolLiverpoolUnited Kingdom
| | - Marc O Ernst
- Cognitive Neuroscience Department, University of Bielefeld (DE)BielefeldGermany
- Applied Cognitive Psychology, University of Ulm (DE)UlmGermany
| |
Collapse
|
3
|
Senkowski D, Engel AK. Multi-timescale neural dynamics for multisensory integration. Nat Rev Neurosci 2024; 25:625-642. [PMID: 39090214 DOI: 10.1038/s41583-024-00845-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/02/2024] [Indexed: 08/04/2024]
Abstract
Carrying out any everyday task, be it driving in traffic, conversing with friends or playing basketball, requires rapid selection, integration and segregation of stimuli from different sensory modalities. At present, even the most advanced artificial intelligence-based systems are unable to replicate the multisensory processes that the human brain routinely performs, but how neural circuits in the brain carry out these processes is still not well understood. In this Perspective, we discuss recent findings that shed fresh light on the oscillatory neural mechanisms that mediate multisensory integration (MI), including power modulations, phase resetting, phase-amplitude coupling and dynamic functional connectivity. We then consider studies that also suggest multi-timescale dynamics in intrinsic ongoing neural activity and during stimulus-driven bottom-up and cognitive top-down neural network processing in the context of MI. We propose a new concept of MI that emphasizes the critical role of neural dynamics at multiple timescales within and across brain networks, enabling the simultaneous integration, segregation, hierarchical structuring and selection of information in different time windows. To highlight predictions from our multi-timescale concept of MI, real-world scenarios in which multi-timescale processes may coordinate MI in a flexible and adaptive manner are considered.
Collapse
Affiliation(s)
- Daniel Senkowski
- Department of Psychiatry and Neurosciences, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Andreas K Engel
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
| |
Collapse
|
4
|
Chen YJ, Sun Z, Nishida S. Feature-invariant processing of spatial segregation based on temporal asynchrony. J Vis 2024; 24:15. [PMID: 38814934 PMCID: PMC11146091 DOI: 10.1167/jov.24.5.15] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Accepted: 05/03/2024] [Indexed: 06/01/2024] Open
Abstract
Temporal asynchrony is a cue for the perceptual segregation of spatial regions. Past research found attribute invariance of this phenomenon such that asynchrony induces perceptual segmentation regardless of the changing attribute type, and it does so even when asynchrony occurs between different attributes. To test the generality of this finding and obtain insights into the underlying computational mechanism, we compared the segmentation performance for changes in luminance, color, motion direction, and their combinations. Our task was to detect the target quadrant in which a periodic alternation in attribute was phase-delayed compared to the remaining quadrants. When stimulus elements made a square-wave attribute change, target detection was not clearly attribute invariant, being more difficult for motion direction change than for luminance or color changes and nearly impossible for the combination of motion direction and luminance or color. We suspect that waveform mismatch might cause anomalous behavior of motion direction since a square-wave change in motion direction is a triangular-wave change in the spatial phase (i.e., a second-order change in the direction of the spatial phase change). In agreement with this idea, we found that the segregation performance was strongly affected by the waveform type (square wave, triangular wave, or their combination), and when this factor was controlled, the performance was nearly, though not perfectly, invariant against attribute type. The results were discussed with a model in which different visual attributes share a common asynchrony-based segmentation mechanism.
Collapse
Affiliation(s)
- Yen-Ju Chen
- Graduate School of Informatics, Kyoto University, Kyoto, Japan
| | - Zitang Sun
- Graduate School of Informatics, Kyoto University, Kyoto, Japan
| | - Shin'ya Nishida
- Graduate School of Informatics, Kyoto University, Kyoto, Japan
- NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Atsugi, Japan
| |
Collapse
|
5
|
Zhao H, Zhang Y, Han L, Qian W, Wang J, Wu H, Li J, Dai Y, Zhang Z, Bowen CR, Yang Y. Intelligent Recognition Using Ultralight Multifunctional Nano-Layered Carbon Aerogel Sensors with Human-Like Tactile Perception. NANO-MICRO LETTERS 2023; 16:11. [PMID: 37943399 PMCID: PMC10635924 DOI: 10.1007/s40820-023-01216-0] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Accepted: 09/11/2023] [Indexed: 11/10/2023]
Abstract
Humans can perceive our complex world through multi-sensory fusion. Under limited visual conditions, people can sense a variety of tactile signals to identify objects accurately and rapidly. However, replicating this unique capability in robots remains a significant challenge. Here, we present a new form of ultralight multifunctional tactile nano-layered carbon aerogel sensor that provides pressure, temperature, material recognition and 3D location capabilities, which is combined with multimodal supervised learning algorithms for object recognition. The sensor exhibits human-like pressure (0.04-100 kPa) and temperature (21.5-66.2 °C) detection, millisecond response times (11 ms), a pressure sensitivity of 92.22 kPa-1 and triboelectric durability of over 6000 cycles. The devised algorithm has universality and can accommodate a range of application scenarios. The tactile system can identify common foods in a kitchen scene with 94.63% accuracy and explore the topographic and geomorphic features of a Mars scene with 100% accuracy. This sensing approach empowers robots with versatile tactile perception to advance future society toward heightened sensing, recognition and intelligence.
Collapse
Affiliation(s)
- Huiqi Zhao
- CAS Center for Excellence in Nanoscience, Beijing Key Laboratory of Micro-Nano Energy and Sensor, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing, 101400, People's Republic of China
- School of Nanoscience and Technology, University of Chinese Academy of Sciences, Beijing, 100049, People's Republic of China
| | - Yizheng Zhang
- Tencent Robotics X, Shenzhen, 518054, People's Republic of China
| | - Lei Han
- Tencent Robotics X, Shenzhen, 518054, People's Republic of China
| | - Weiqi Qian
- CAS Center for Excellence in Nanoscience, Beijing Key Laboratory of Micro-Nano Energy and Sensor, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing, 101400, People's Republic of China
- School of Nanoscience and Technology, University of Chinese Academy of Sciences, Beijing, 100049, People's Republic of China
| | - Jiabin Wang
- CAS Center for Excellence in Nanoscience, Beijing Key Laboratory of Micro-Nano Energy and Sensor, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing, 101400, People's Republic of China
- Center on Nanoenergy Research, School of Physical Science and Technology, Guangxi University, Nanning, 530004, People's Republic of China
| | - Heting Wu
- CAS Center for Excellence in Nanoscience, Beijing Key Laboratory of Micro-Nano Energy and Sensor, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing, 101400, People's Republic of China
| | - Jingchen Li
- Tencent Robotics X, Shenzhen, 518054, People's Republic of China
| | - Yuan Dai
- Tencent Robotics X, Shenzhen, 518054, People's Republic of China.
| | - Zhengyou Zhang
- Tencent Robotics X, Shenzhen, 518054, People's Republic of China
| | - Chris R Bowen
- Department of Mechanical Engineering, University of Bath, Bath, BA2 7AK, UK
| | - Ya Yang
- CAS Center for Excellence in Nanoscience, Beijing Key Laboratory of Micro-Nano Energy and Sensor, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing, 101400, People's Republic of China.
- School of Nanoscience and Technology, University of Chinese Academy of Sciences, Beijing, 100049, People's Republic of China.
- Center on Nanoenergy Research, School of Physical Science and Technology, Guangxi University, Nanning, 530004, People's Republic of China.
| |
Collapse
|
6
|
Polgári P, Jovanovic L, van Wassenhove V, Giersch A. The processing of subthreshold visual temporal order is transitory and motivation-dependent. Sci Rep 2023; 13:7699. [PMID: 37169810 PMCID: PMC10175535 DOI: 10.1038/s41598-023-34392-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2022] [Accepted: 04/28/2023] [Indexed: 05/13/2023] Open
Abstract
Processing a sequence of events is different from encoding the relative order of the elements composing the sequence. Whether order processing arises automatically from the sequential processing of events is yet unknown, however the literature suggests that order processing can occur at an automatic level when the order of stimuli is not detected consciously. In the present study, we aimed to investigate the question of automatic order processing in a difficult visual task where participants identified one among two possible target luminances. The luminance of the targets was contingent on the order of presentation of two visual cues separated by a subthreshold asynchrony. Participants' performance was compared to that in a control condition where the cues were presented synchronously. In a first experiment, participants' performance benefited from the use of subthreshold order information compared to the control condition, however this facilitation effect was transient and disappeared over the course of the experiment. In a second experiment, we investigated and confirmed the role of motivation, via a monetary incentive, on the previously observed effect. Taken together, our results suggest that the processing of temporal order of sub-threshold asynchronies is possible, although fragile and likely dependent on task requirements.
Collapse
Affiliation(s)
- Patrik Polgári
- University of Strasbourg, INSERM U1114, Strasbourg, France
| | - Ljubica Jovanovic
- University of Strasbourg, INSERM U1114, Strasbourg, France
- Laboratoire des Systèmes Perceptifs, École Normale Supérieure, PSL University & CNRS, Paris, France
| | - Virginie van Wassenhove
- Cognitive Neuroimaging Unit, CEA, INSERM, CNRS, Neurospin, Université Paris-Saclay, 91191, Gif/Yvette, France
| | - Anne Giersch
- University of Strasbourg, INSERM U1114, Strasbourg, France.
- Department of Psychiatry, University Hospital of Strasbourg, 1, Pl de L'Hôpital, 67000, Strasbourg, France.
| |
Collapse
|
7
|
Quintero SI, Shams L, Kamal K. Changing the Tendency to Integrate the Senses. Brain Sci 2022; 12:1384. [PMID: 36291318 PMCID: PMC9599885 DOI: 10.3390/brainsci12101384] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2022] [Revised: 10/04/2022] [Accepted: 10/05/2022] [Indexed: 11/16/2022] Open
Abstract
Integration of sensory signals that emanate from the same source, such as the visual of lip articulations and the sound of the voice of a speaking individual, can improve perception of the source signal (e.g., speech). Because momentary sensory inputs are typically corrupted with internal and external noise, there is almost always a discrepancy between the inputs, facing the perceptual system with the problem of determining whether the two signals were caused by the same source or different sources. Thus, whether or not multisensory stimuli are integrated and the degree to which they are bound is influenced by factors such as the prior expectation of a common source. We refer to this factor as the tendency to bind stimuli, or for short, binding tendency. In theory, the tendency to bind sensory stimuli can be learned by experience through the acquisition of the probabilities of the co-occurrence of the stimuli. It can also be influenced by cognitive knowledge of the environment. The binding tendency varies across individuals and can also vary within an individual over time. Here, we review the studies that have investigated the plasticity of binding tendency. We discuss the protocols that have been reported to produce changes in binding tendency, the candidate learning mechanisms involved in this process, the possible neural correlates of binding tendency, and outstanding questions pertaining to binding tendency and its plasticity. We conclude by proposing directions for future research and argue that understanding mechanisms and recipes for increasing binding tendency can have important clinical and translational applications for populations or individuals with a deficiency in multisensory integration.
Collapse
Affiliation(s)
- Saul I. Quintero
- Department of Psychology, University of California, Los Angeles, CA 90095, USA
| | - Ladan Shams
- Department of Psychology, University of California, Los Angeles, CA 90095, USA
- Department of Bioengineering, University of California, Los Angeles, CA 90089, USA
- Neuroscience Interdepartmental Program, University of California, Los Angeles, CA 90089, USA
| | - Kimia Kamal
- Department of Psychology, University of California, Los Angeles, CA 90095, USA
| |
Collapse
|