1
|
Wang J, Guo M, Zhang J, Bai Y, Ni G. Early audiovisual integration in target processing under continuous noise: Behavioral and EEG evidence. Neuropsychologia 2025; 211:109128. [PMID: 40112909 DOI: 10.1016/j.neuropsychologia.2025.109128] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2025] [Revised: 03/17/2025] [Accepted: 03/17/2025] [Indexed: 03/22/2025]
Abstract
Multisensory integration is interconnected across various information reception. The stage and mechanism of brain response to audiovisual integration have not been fully understood. In this study, we designed audiovisual and unisensory experiments to investigate task performance and electrophysiological characteristics associated with audiovisual integration in a continuous background interference environment using materials collected from the underwater environment. Behavioral results showed that the reaction time (RT) was shorter, and the accuracy was higher in the audiovisual experiment. The cumulative distribution function (CDF) results of RT indicated that audiovisual integration supported the co-activation model. Event-related potential (ERP) results revealed shorter latency of the P1 and N1 components in the audiovisual experiment. Microstate analysis indicated that the parietal-occipital area may play a key role in audiovisual integration. Moreover, event-related spectral perturbation (ERSP) results demonstrated the critical role of low-frequency oscillation in audiovisual integration at the early stage. Our findings support the view that the beneficial effect of audiovisual integration is predominantly upon the early stage of neural information processing, including task-independent information.
Collapse
Affiliation(s)
- Junjie Wang
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072 China; State Key Laboratory of Advanced Medical Materials and Devices, Tianjin University, Tianjin, 300072 China
| | - Mingkun Guo
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072 China; State Key Laboratory of Advanced Medical Materials and Devices, Tianjin University, Tianjin, 300072 China
| | - Jie Zhang
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072 China; State Key Laboratory of Advanced Medical Materials and Devices, Tianjin University, Tianjin, 300072 China
| | - Yanru Bai
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072 China; State Key Laboratory of Advanced Medical Materials and Devices, Tianjin University, Tianjin, 300072 China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392 China; Tianjin Key Laboratory of Brain Science and Neuroengineering, Tianjin, 300072 China
| | - Guangjian Ni
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072 China; State Key Laboratory of Advanced Medical Materials and Devices, Tianjin University, Tianjin, 300072 China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392 China; Tianjin Key Laboratory of Brain Science and Neuroengineering, Tianjin, 300072 China.
| |
Collapse
|
2
|
Pan F, Jia T, Sun J, Wang AY, Ji L, Li C. Enhancing Motor Learning Performance by Incorporating Brain-to-brain Coupling with Affective Interaction. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2024; 2024:1-6. [PMID: 40039203 DOI: 10.1109/embc53108.2024.10781569] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/06/2025]
Abstract
The widespread clinical application of rehabilitation robots is an inevitable trend. Nonetheless, the absence of affective interactions in robot-based training could significantly limit rehabilitation robots' clinical effects. While a favorable patient-clinician relationship during rehabilitation training may promote positive affectivity and result in improved clinical outcomes, the mechanisms underlying this phenomenon remain unclear. In this study, a motor learning experiment was designed to prospectively investigate the impact of interpersonal interactions on training outcomes. Ten pairs of friends engaged in a left-hand drawing task across two training sessions: one led by a close partner and the other by a robot. Pairs of participants' EEG was recorded during training sessions via hyperscaning and task performance was evaluated before and after the training. Results revealed significant motor learning progress and enhanced brain activation in the central, parieto-occipital, and frontal cortex, along with strengthened interbrain coupling in partner-led training compared with robot-based training. These findings suggest that partner-assisted training fosters stronger motivation, attention, and positive emotions, contributing to superior outcomes. This study may provide theoretical and technical support for the design of human-machine affective interaction in rehabilitation robots.
Collapse
|
3
|
Zhozhikashvili N, Protopova M, Shkurenko T, Arsalidou M, Zakharov I, Kotchoubey B, Malykh S, Pavlov YG. Working memory processes and intrinsic motivation: An EEG study. Int J Psychophysiol 2024; 201:112355. [PMID: 38718899 DOI: 10.1016/j.ijpsycho.2024.112355] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2024] [Revised: 04/21/2024] [Accepted: 04/30/2024] [Indexed: 06/11/2024]
Abstract
Processes typically encompassed by working memory (WM) include encoding, retention, and retrieval of information. Previous research has demonstrated that motivation can influence WM performance, although the specific WM processes affected by motivation are not yet fully understood. In this study, we investigated the effects of motivation on different WM processes, examining how task difficulty modulates these effects. We hypothesized that motivation level and personality traits of the participants (N = 48, 32 females; mean age = 21) would modulate the parietal alpha and frontal theta electroencephalography (EEG) correlates of WM encoding, retention, and retrieval phases of the Sternberg task. This effect was expected to be more pronounced under conditions of very high task difficulty. We found that increasing difficulty led to reduced accuracy and increased response time, but no significant relationship was found between motivation and accuracy. However, EEG data revealed that motivation influenced WM processes, as indicated by changes in alpha and theta oscillations. Specifically, higher levels of the Resilience trait-associated with mental toughness, hardiness, self-efficacy, achievement motivation, and low anxiety-were related to increased alpha desynchronization during encoding and retrieval. Increased scores of Subjective Motivation to perform well in the task were related to enhanced frontal midline theta during retention. Additionally, these effects were significantly stronger under conditions of high difficulty. These findings provide insights into the specific WM processes that are influenced by motivation, and underscore the importance of considering both task difficulty and intrinsic motivation in WM research.
Collapse
Affiliation(s)
- Natalia Zhozhikashvili
- Faculty of Social Sciences, HSE University, Moscow, Russia; Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Tübingen, Germany.
| | - Maria Protopova
- Center for Language and Brain, HSE University, Moscow, Russia
| | | | | | - Ilya Zakharov
- Ural Federal University named after the first President of Russia B.N. Yeltsin, Yekaterinburg, Russia
| | - Boris Kotchoubey
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Tübingen, Germany
| | - Sergey Malykh
- Developmental Behavioral Genetics Lab, Psychological Institute of Russian Academy of Education, Moscow, Russia
| | - Yuri G Pavlov
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Tübingen, Germany
| |
Collapse
|
4
|
Phukhachee T, Maneewongvatana S, Chaiyanan C, Iramina K, Kaewkamnerdpong B. Identifying the Effect of Cognitive Motivation with the Method Based on Temporal Association Rule Mining Concept. SENSORS (BASEL, SWITZERLAND) 2024; 24:2857. [PMID: 38732962 PMCID: PMC11086084 DOI: 10.3390/s24092857] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/14/2024] [Revised: 04/18/2024] [Accepted: 04/25/2024] [Indexed: 05/13/2024]
Abstract
Being motivated has positive influences on task performance. However, motivation could result from various motives that affect different parts of the brain. Analyzing the motivation effect from all affected areas requires a high number of EEG electrodes, resulting in high cost, inflexibility, and burden to users. In various real-world applications, only the motivation effect is required for performance evaluation regardless of the motive. Analyzing the relationships between the motivation-affected brain areas associated with the task's performance could limit the required electrodes. This study introduced a method to identify the cognitive motivation effect with a reduced number of EEG electrodes. The temporal association rule mining (TARM) concept was used to analyze the relationships between attention and memorization brain areas under the effect of motivation from the cognitive motivation task. For accuracy improvement, the artificial bee colony (ABC) algorithm was applied with the central limit theorem (CLT) concept to optimize the TARM parameters. From the results, our method can identify the motivation effect with only FCz and P3 electrodes, with 74.5% classification accuracy on average with individual tests.
Collapse
Affiliation(s)
- Tustanah Phukhachee
- Computer Engineering Department, Faculty of Engineering, King Mongkut’s University of Technology Thonburi, Bangkok 10140, Thailand; (T.P.); (S.M.); (C.C.)
| | - Suthathip Maneewongvatana
- Computer Engineering Department, Faculty of Engineering, King Mongkut’s University of Technology Thonburi, Bangkok 10140, Thailand; (T.P.); (S.M.); (C.C.)
| | - Chayapol Chaiyanan
- Computer Engineering Department, Faculty of Engineering, King Mongkut’s University of Technology Thonburi, Bangkok 10140, Thailand; (T.P.); (S.M.); (C.C.)
| | - Keiji Iramina
- Graduate School of Systems Life Sciences, Kyushu University, Fukuoka 819-0395, Japan;
| | - Boonserm Kaewkamnerdpong
- Biological Engineering Program, Faculty of Engineering, King Mongkut’s University of Technology Thonburi, Bangkok 10140, Thailand
| |
Collapse
|
5
|
Rhodes LJ, Borghetti L, Morris MB. Multiscale entropy in a 10-minute vigilance task. Int J Psychophysiol 2024; 198:112323. [PMID: 38428744 DOI: 10.1016/j.ijpsycho.2024.112323] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Revised: 02/20/2024] [Accepted: 02/26/2024] [Indexed: 03/03/2024]
Abstract
Research has shown multiscale entropy, brain signal behavior across time scales, to reliably increase at lower time scales with time-on-task fatigue. However, multiscale entropy has not been examined in short vigilance tasks (i.e., ≤ 10 min). Addressing this gap, we examine multiscale entropy during a 10-minute Psychomotor Vigilance Test (PVT). Thirty-four participants provided neural data while completing the PVT. We compared the first 2 min of the task to the 7th and 8th minutes to avoid end-spurt effects. Results suggested increased multiscale entropy at lower time scales later compared to earlier in the task, suggesting multiscale entropy is a strong marker of time-on-task fatigue onset during short vigils. Separate analyses for Fast and Slow performers reveal differential entropy patterns, particularly over visual cortices. Here, observed brain-behavior linkage between entropy and reaction time for slow performers suggests that entropy assays over sensory cortices might have predictive value for fatigue onset or shifts from on- to off-task states.
Collapse
Affiliation(s)
- L Jack Rhodes
- Ball Aerospace at Wright-Patterson Air Force Base, OH, United States of America.
| | - Lorraine Borghetti
- Air Force Research Laboratory, Wright-Patterson Air Force Base, OH, United States of America
| | - Megan B Morris
- Air Force Research Laboratory, Wright-Patterson Air Force Base, OH, United States of America
| |
Collapse
|
6
|
Phukhachee T, Angsuwatanakul T, Iramina K, Kaewkamnerdpong B. A simultaneous EEG-fNIRS dataset of the visual cognitive motivation study in healthy adults. Data Brief 2024; 53:110260. [PMID: 38533112 PMCID: PMC10964074 DOI: 10.1016/j.dib.2024.110260] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2024] [Revised: 02/20/2024] [Accepted: 02/21/2024] [Indexed: 03/28/2024] Open
Abstract
This article described a publicly available dataset of the visual cognitive motivation study in healthy adults. To gain an in-depth understanding and insights into motivation, Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were measured simultaneously at shared locations while participants performed a visual cognitive motivation task. The participants' choices in the cognitive motivation task were recorded. The effects of their motivation were identified in the recognition test afterward. This dataset comprised EEG and fNIRS data from sixteen healthy adults (age: 21- 37 years; 14 males and 2 females) during the cognitive motivation task with visual scenic stimuli. In addition, the motivation and the corresponding motivation effect were also provided. This dataset provides understanding and analyzing opportunities for the process of attention and decision while the brain undergoes an induced motivated state and its effect on the recognition performance.
Collapse
Affiliation(s)
- Tustanah Phukhachee
- Computer Engineering Department, Faculty of Engineering, King Mongkut's University of Technology Thonburi, Bangkok 10140, Thailand
| | | | - Keiji Iramina
- Graduate School of Systems Life Sciences, Kyushu University, Fukuoka 819-0395, Japan
| | - Boonserm Kaewkamnerdpong
- Biological Engineering Program, Faculty of Engineering, King Mongkut's University of Technology Thonburi, Bangkok 10140, Thailand
| |
Collapse
|
7
|
O'Reilly JA. Modelling mouse auditory response dynamics along a continuum of consciousness using a deep recurrent neural network. J Neural Eng 2022; 19. [DOI: 10.1088/1741-2552/ac9257] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2022] [Accepted: 09/15/2022] [Indexed: 11/12/2022]
Abstract
Abstract
Objective Understanding neurophysiological changes that accompany transitions between anaesthetized and conscious states is a key objective of anesthesiology and consciousness science. This study aimed to characterize the dynamics of auditory-evoked potential morphology in mice along a continuum of consciousness. Approach Epidural field potentials were recorded from above the primary auditory cortices of two groups of laboratory mice: urethane-anaesthetized (A, n = 14) and conscious (C, n = 17). Both groups received auditory stimulation in the form of a repeated pure-tone stimulus, before and after receiving 10 mg/kg i.p. ketamine (AK and CK). Evoked responses were then ordered by ascending sample entropy into AK, A, CK, and C, considered to reflect physiological correlates of awareness. These data were used to train a recurrent neural network (RNN) with an input parameter encoding state. Model outputs were compared with grand-average event-related potential (ERP) waveforms. Subsequently, the state parameter was varied to simulate changes in the ERP that occur during transitions between states, and relationships with dominant peak amplitudes were quantified. Main results The RNN synthesized output waveforms that were in close agreement with grand-average ERPs for each group (r2 > 0.9, p < 0.0001). Varying the input state parameter generated model outputs reflecting changes in ERP morphology predicted to occur between states. Positive peak amplitudes within 25 to 50 ms, and negative peak amplitudes within 50 to 75 ms post-stimulus-onset, were found to display a sigmoidal characteristic during the transition from anaesthetized to conscious states. In contrast, negative peak amplitudes within 0 to 25 ms displayed greater linearity. Significance This study demonstrates a method for modelling changes in ERP morphology that accompany transitions between states of consciousness using a RNN. In future studies, this approach may be applied to human data to support the clinical use of ERPs to predict transition to consciousness.
Collapse
|
8
|
O'Reilly JA, Angsuwatanakul T, Wehrman J. Decoding violated sensory expectations from the auditory cortex of anaesthetised mice: Hierarchical recurrent neural network depicts separate 'danger' and 'safety' units. Eur J Neurosci 2022; 56:4154-4175. [PMID: 35695993 PMCID: PMC9545291 DOI: 10.1111/ejn.15736] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2022] [Revised: 06/02/2022] [Accepted: 06/07/2022] [Indexed: 12/27/2022]
Abstract
The ability to respond appropriately to sensory information received from the external environment is among the most fundamental capabilities of central nervous systems. In the auditory domain, processes underlying this behaviour are studied by measuring auditory‐evoked electrophysiology during sequences of sounds with predetermined regularities. Identifying neural correlates of ensuing auditory novelty responses is supported by research in experimental animals. In the present study, we reanalysed epidural field potential recordings from the auditory cortex of anaesthetised mice during frequency and intensity oddball stimulation. Multivariate pattern analysis (MVPA) and hierarchical recurrent neural network (RNN) modelling were adopted to explore these data with greater resolution than previously considered using conventional methods. Time‐wise and generalised temporal decoding MVPA approaches revealed previously underestimated asymmetry between responses to sound‐level transitions in the intensity oddball paradigm, in contrast with tone frequency changes. After training, the cross‐validated RNN model architecture with four hidden layers produced output waveforms in response to simulated auditory inputs that were strongly correlated with grand‐average auditory‐evoked potential waveforms (r2 > .9). Units in hidden layers were classified based on their temporal response properties and characterised using principal component analysis and sample entropy. These demonstrated spontaneous alpha rhythms, sound onset and offset responses and putative ‘safety’ and ‘danger’ units activated by relatively inconspicuous and salient changes in auditory inputs, respectively. The hypothesised existence of corresponding biological neural sources is naturally derived from this model. If proven, this could have significant implications for prevailing theories of auditory processing.
Collapse
Affiliation(s)
- Jamie A O'Reilly
- College of Biomedical Engineering, Rangsit University, Lak Hok, Thailand
| | | | - Jordan Wehrman
- Brain and Mind Centre, University of Sydney, Camperdown, New South Wales, Australia
| |
Collapse
|
9
|
Gao Q, Tan Y. Impact of Different Styles of Online Course Videos on Students' Attention During the COVID-19 Pandemic. Front Public Health 2022; 10:858780. [PMID: 35462812 PMCID: PMC9024118 DOI: 10.3389/fpubh.2022.858780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2022] [Accepted: 03/16/2022] [Indexed: 11/22/2022] Open
Abstract
Background The COVID-19 pandemic interfered with normal campus life, resulting in the need for the course to be conducted in an ideal online format. The purpose of this study is to analyze the impact of different styles of online political course videos on students' attention during the COVID-19 pandemic. Methods Four college students participated in this small sample study. They were required to conduct two sessions of the experiment, in which they were required to watch three different styles of course videos in each session. While watching the videos, their EEG signals were acquired. For the acquired EEG signals, the sample entropy (SampEn) features were extracted. On the other hand, Mayer's theories of multimedia technology provide guidance for teachers' online courses to enhance students' attention levels. The results of EEG signals analysis and Mayer's theories of multimedia technology were combined to compare and analyze the effects of three styles of instructional videos. Results Based on comparisons of the SampEn and Mayer's theories of multimedia technology analysis, the results suggest that online instruction in a style where the instructor and content appear on the screen at the same time and the instructor points out the location of the content as it is explained is more likely to elicit higher levels of students' attention. Conclusions During the COVID-19 pandemic, online instructional methods have an impact on students' classroom attention. It is essential for teachers to design online instructional methods based on students' classroom attention levels and some multimedia instructional techniques to improve students' learning efficiency.
Collapse
Affiliation(s)
- Qi Gao
- School of Economics, School of Marxism, Nankai University, Tianjin, China
- *Correspondence: Qi Gao
| | - Ying Tan
- College of Artificial Intelligence, Nankai University, Tianjin, China
| |
Collapse
|
10
|
Investigation on Identifying Implicit Learning Event from EEG Signal Using Multiscale Entropy and Artificial Bee Colony. ENTROPY 2021; 23:e23050617. [PMID: 34065692 PMCID: PMC8155885 DOI: 10.3390/e23050617] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/09/2021] [Revised: 05/11/2021] [Accepted: 05/11/2021] [Indexed: 12/02/2022]
Abstract
The way people learn will play an essential role in the sustainable development of the educational system for the future. Utilizing technology in the age of information and incorporating it into how people learn can produce better learners. Implicit learning is a type of learning of the underlying rules without consciously seeking or understanding the rules; it is commonly seen in small children while learning how to speak their native language without learning grammar. This research aims to introduce a processing system that can systematically identify the relationship between implicit learning events and their Encephalogram (EEG) signal characteristics. This study converted the EEG signal from participants while performing cognitive task experiments into Multiscale Entropy (MSE) data. Using MSE data from different frequency bands and channels as features, the system explored a wide range of classifiers and observed their performance to see how they classified the features related to participants’ performance. The Artificial Bee Colony (ABC) method was used for feature selection to improve the process to make the system more efficient. The results showed that the system could correctly identify the differences between participants’ performance using MSE data and the ABC method with 95% confidence.
Collapse
|