Multi-Modal Integration of EEG-fNIRS for Characterization of Brain Activity Evoked by Preferred Music

A systematic review of the comparative effects of sound and music interventions for intensive care unit patients' outcomes

BACKGROUND

Despite syntheses of evidence showing efficacy of music intervention for improving psychological and physiological outcomes in critically ill patients, interventions that include nonmusic sounds have not been addressed in reviews of evidence. It is unclear if nonmusic sounds in the intensive care unit (ICU) can confer benefits similar to those of music.

OBJECTIVE

The aim of this study was to summarise and contrast available evidence on the effect of music and nonmusic sound interventions for the physiological and psychological outcomes of ICU patients based on the results of randomised controlled trials.

METHODS

This systematic review was directed by a protocol based on the Methodological Expectations of Cochrane Intervention Reviews. Quality of studies was assessed with the Cochrane risk of bias assessment tool. Searches were performed in the following databases: MEDLINE, Embase, APA PsycInfo, CINAHL Plus with Full Text, Academic Search Complete, RILM Abstracts of Music Literature, Web of Science, and Scopus.

RESULTS

We identified 59 articles meeting the inclusion criteria, 37 involving music and 22 involving nonmusic sound interventions, with one study comparing music and sound. The identified studies were representative of a general ICU population, regardless of patients' ability to communicate. Our review demonstrated that both slow-tempo music and sound interventions can significantly (i) decrease pain; (ii) improve sleep; (iii) regulate cortisol levels; (iv) reduce sedative and analgesic need; and (v) reduce stress/anxiety and improve relaxation when compared with standard care and noise reduction. Moreover, compared to nonmusic sound interventions, there is more evidence that music interventions have an effect on stress biomarkers, vital signs, and haemodynamic measures.

CONCLUSION

These results raise the possibility that different auditory interventions may have varying degrees of effectiveness for specific patient outcomes in the ICU. More investigation is needed to clarify if nonmusic sound interventions may be equivalent or not to music interventions for the management of discrete symptoms in ICU patients.

REGISTRATION OF REVIEWS

The protocol was registered on Open Science Framework in November 6 2023 (https://doi.org/10.17605/OSF.IO/45F6E).

The bidirectional role of music effect in epilepsy: Friend or foe?

Epilepsy is a prevalent neurological disease that impacts around 70 million individuals globally. Anti-seizure medications (ASMs) are the first choice for clinicians to control unprovoked epileptic seizures. Although more than 30 ASMs are available in the market, patients with epilepsy (PWEs) still show poor responses to adequate drug treatment. Meanwhile, long-term medications not only bring heavy financial burdens but also lead to undesirable side effects. Music, a ubiquitous art form throughout human history, has been confirmed as therapeutically effective in various neurological conditions, including epilepsy. This alternative therapy offers convenience and a relatively safe approach to alleviating epileptic symptoms. Paradoxically, besides anti-convulsant effect, some particular music would cause seizures inversely, indicating the pro-convulsant effect of it. Considering that investigating the impact of music on epilepsy emerges as a compelling subject. In this review, we tried to present the following sections of content on this topic. Initially, we overviewed the impact of music on the brain and the significant progress of music therapy in central neurological disorders. Afterward, we classified the anti-convulsant and pro-convulsant effects of music in epilepsy, relying on both clinical and laboratory evidences. Finally, possible mechanisms and neural basis of the music effect were concluded, and the translational potentials and some future outlooks about the music effect in epilepsy were proposed. PLAIN LANGUAGE SUMMARY: Epilepsy is an extremely severe neurological disorder. Although anti-seizure medications are preferred choice to control seizures, the efficacy is not satisfied due to the tolerance. Anecdotal music effect had been deemed functional diversity but not clarified on epilepsy, pro-convulsive, or anti-convulsive. Here, we reviewed this interesting but puzzling topic, as well as illustrating the potential mechanisms and its translational potential.

Effects of personalized music listening on post-stroke cognitive impairment: A randomized controlled trial

BACKGROUND AND PURPOSE

Previous studies have suggested that music listening has the potential to positively affect cognitive functions and mood in individuals with post-stroke cognitive impairment (PSCI), with a preference for self-selected music likely to yield better outcomes. However, there is insufficient clinical evidence to suggest the use of music listening in routine rehabilitation care to treat PSCI. This randomized control trial (RCT) aims to investigate the effects of personalized music listening on mood improvement, activities of daily living (ADLs), and cognitive functions in individuals with PSCI.

MATERIALS AND METHODS

A total of 34 patients with PSCI were randomly assigned to either the music group or the control group. Patients in the music group underwent a three-month personalized music-listening intervention. The intervention involved listening to a personalized playlist tailored to each individual's cultural, ethnic, and social background, life experiences, and personal music preferences. In contrast, the control group patients listened to white noise as a placebo. Cognitive function, neurological function, mood, and ADLs were assessed.

RESULTS

After three months of treatment, the music group showed significantly higher Montreal Cognitive Assessment (MoCA) scores compared to the control group (p=0.027), particularly in the domains of delayed recall (p=0.019) and orientation (p=0.023). Moreover, the music group demonstrated significantly better scores in National Institutes of Health Stroke Scale (NIHSS) (p=0.008), Barthel Index (BI) (p=0.019), and Zarit Caregiver Burden Interview (ZBI) (p=0.008) compared to the control group. No effects were found on mood as measured by the Hamilton Anxiety Rating Scale (HAMA) and the Hamilton Depression Rating Scale (HAMD).

CONCLUSION

Personalized music listening promotes the recovery of cognitive and neurological functions, improves ADLs, and reduces caregiver burden in patients with PSCI.

Strategic Integration: A Cross-Disciplinary Review of the fNIRS-EEG Dual-Modality Imaging System for Delivering Multimodal Neuroimaging to Applications

Background: Recent years have seen a surge of interest in dual-modality imaging systems that integrate functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to probe brain function. This review aims to explore the advancements and clinical applications of this technology, emphasizing the synergistic integration of fNIRS and EEG. Methods: The review begins with a detailed examination of the fundamental principles and distinctive features of fNIRS and EEG techniques. It includes critical technical specifications, data-processing methodologies, and analysis techniques, alongside an exhaustive evaluation of 30 seminal studies that highlight the strengths and weaknesses of the fNIRS-EEG bimodal system. Results: The paper presents multiple case studies across various clinical domains-such as attention-deficit hyperactivity disorder, infantile spasms, depth of anesthesia, intelligence quotient estimation, and epilepsy-demonstrating the fNIRS-EEG system's potential in uncovering disease mechanisms, evaluating treatment efficacy, and providing precise diagnostic options. Noteworthy research findings and pivotal breakthroughs further reinforce the developmental trajectory of this interdisciplinary field. Conclusions: The review addresses challenges and anticipates future directions for the fNIRS-EEG dual-modal imaging system, including improvements in hardware and software, enhanced system performance, cost reduction, real-time monitoring capabilities, and broader clinical applications. It offers researchers a comprehensive understanding of the field, highlighting the potential applications of fNIRS-EEG systems in neuroscience and clinical medicine.

Lights on music cognition: A systematic and critical review of fNIRS applications and future perspectives

Research investigating the neural processes related to music perception and production constitutes a well-established field within the cognitive neurosciences. While most neuroimaging tools have limitations in studying the complexity of musical experiences, functional Near-Infrared Spectroscopy (fNIRS) represents a promising, relatively new tool for studying music processes in both laboratory and ecological settings, which is also suitable for both typical and pathological populations across development. Here we systematically review fNIRS studies on music cognition, highlighting prospects and potentialities. We also include an overview of fNIRS basic theory, together with a brief comparison to characteristics of other neuroimaging tools. Fifty-nine studies meeting inclusion criteria (i.e., using fNIRS with music as the primary stimulus) are presented across five thematic sections. Critical discussion of methodology leads us to propose guidelines of good practices aiming for robust signal analyses and reproducibility. A continuously updated world map is proposed, including basic information from studies meeting the inclusion criteria. It provides an organized, accessible, and updatable reference database, which could serve as a catalyst for future collaborations within the community. In conclusion, fNIRS shows potential for investigating cognitive processes in music, particularly in ecological contexts and with special populations, aligning with current research priorities in music cognition.

The Impact of Listening to Background Music on Inhibition Control and Prefrontal Cortical Activation in Healthy Older Adults: A Study Using Functional Near-Infrared Spectroscopy

Introduction Aging declines executive functions, including attentional function and inhibitory control, which is the ability to inhibit inappropriate or irrelevant responses. Certain types of background music are negatively correlated with cognitive function. The prefrontal network is correlated with task performance related to executive function. This study aimed to assess the impact of listening to background music on inhibition control and prefrontal cortical (PFC) activation measured using functional near-infrared spectroscopy (fNIRS) in healthy older people. Methods In total, 59 healthy volunteers, including 32 healthy older and 27 younger individuals (mean age ± standard deviation: 69 ± 7 and 32 ± 8 years, respectively), participated in this study. The participants completed the inhibition control task (the go/no-go task) and a similar task while listening to certain melodies of children's songs that are popular in Japan. Changes in cerebral blood flow in the PFC during each task were evaluated using multichannel fNIRS. The relative changes in oxygenated hemoglobin (oxy-Hb) levels during the no-go and go tasks under the music and no-music conditions were compared using a paired t-test. Among the channels with a significant difference in oxy-Hb levels during the go/no-go task between the music and no-music conditions in the older group, the correlation between changes in accuracy response and oxy-Hb levels was validated using Pearson's correlation test. Results The task accuracy was significantly reduced under the music condition compared with that under the no-music condition in the older group but not in the younger group. The accuracy reduction was significantly greater in the older group than in the younger group. In older people, the oxy-Hb levels in 20 channels located in the bilateral Broadman area (BA) 9 and BA46 in the dorsolateral prefrontal cortex and the bilateral BA10 in the frontal pole cortex significantly increased during the no-go tasks under the music condition. During the go/no-go task under the music condition, the decline in task accuracy was significantly correlated with increased oxy-Hb levels in six channels located in the bilateral BA10 in older people. Conclusion Background music induced the decline of inhibition control and increase of PFC activity in healthy older adults.

Assessing Consciousness in Patients With Disorders of Consciousness Using a Musical Stimulation Paradigm and Verifiable Criteria

Numerous studies have shown that musical stimulation can activate corresponding functional brain areas. Electroencephalogram (EEG) activity during musical stimulation can be used to assess the consciousness states of patients with disorders of consciousness (DOC). In this study, a musical stimulation paradigm and verifiable criteria were used for consciousness assessment. Twenty-nine participants (13 healthy subjects, 6 patients in a minimally conscious state (MCS) and 10 patients in a vegetative state (VS)) were recruited, and EEG signals were collected while participants listened to preferred and relaxing music. Fusion features based on differential entropy (DE), common spatial pattern (CSP), and EEG-based network pattern (ENP) features were extracted from EEG signals, and a convolutional neural network-long short-term memory (CNN-LSTM) model was employed to classify preferred and relaxing music.The results showed that the average classification accuracy for healthy subjects reached 85.58%. For two of the patients in the MCS group, the classification accuracies reached 78.18% and 66.14%, and they were diagnosed with emergence from MCS (EMCS) two months later. The accuracies of three patients in the VS group were 58.18%, 64.32% and 62.05%, with two patients showing slight increases in scale scores. Our study suggests that musical stimulation could be an effective method for consciousness detection, with significant diagnostic implications for patients with DOC.

EEG-fNIRS-Based Emotion Recognition Using Graph Convolution and Capsule Attention Network

Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) can objectively reflect a person's emotional state and have been widely studied in emotion recognition. However, the effective feature fusion and discriminative feature learning from EEG-fNIRS data is challenging. In order to improve the accuracy of emotion recognition, a graph convolution and capsule attention network model (GCN-CA-CapsNet) is proposed. Firstly, EEG-fNIRS signals are collected from 50 subjects induced by emotional video clips. And then, the features of the EEG and fNIRS are extracted; the EEG-fNIRS features are fused to generate higher-quality primary capsules by graph convolution with the Pearson correlation adjacency matrix. Finally, the capsule attention module is introduced to assign different weights to the primary capsules, and higher-quality primary capsules are selected to generate better classification capsules in the dynamic routing mechanism. We validate the efficacy of the proposed method on our emotional EEG-fNIRS dataset with an ablation study. Extensive experiments demonstrate that the proposed GCN-CA-CapsNet method achieves a more satisfactory performance against the state-of-the-art methods, and the average accuracy can increase by 3-11%.

Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks

The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG-fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO2, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN-LSTM, CNN-GRU, LSTM-GRU, and CNN-LSTM-GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN-LSTM-GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN-LSTM. The experimental result shows that decision-fused EEG-HbO2-HbR and EEG-fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN-LSTM-GRU in Dataset 01 and CNN-LSTM in Dataset 02 yield the highest performance.

Understanding the Effect of Listening to Music, Playing Music, and Singing on Brain Function: A Scoping Review of fNIRS Studies

Music is integrated into daily life when listening to it, playing it, and singing, uniquely modulating brain activity. Functional near-infrared spectroscopy (fNIRS), celebrated for its ecological validity, has been used to elucidate this music-brain interaction. This scoping review synthesizes 22 empirical studies using fNIRS to explore the intricate relationship between music and brain function. This synthesis of existing evidence reveals that diverse musical activities, such as listening to music, singing, and playing instruments, evoke unique brain responses influenced by individual traits and musical attributes. A further analysis identifies five key themes, including the effect of passive and active music experiences on relevant human brain areas, lateralization in music perception, individual variations in neural responses, neural synchronization in musical performance, and new insights fNIRS has revealed in these lines of research. While this review highlights the limited focus on specific brain regions and the lack of comparative analyses between musicians and non-musicians, it emphasizes the need for future research to investigate the complex interplay between music and the human brain.

Music-evoked emotions classification using vision transformer in EEG signals

Introduction

The field of electroencephalogram (EEG)-based emotion identification has received significant attention and has been widely utilized in both human-computer interaction and therapeutic settings. The process of manually analyzing electroencephalogram signals is characterized by a significant investment of time and work. While machine learning methods have shown promising results in classifying emotions based on EEG data, the task of extracting distinct characteristics from these signals still poses a considerable difficulty.

Methods

In this study, we provide a unique deep learning model that incorporates an attention mechanism to effectively extract spatial and temporal information from emotion EEG recordings. The purpose of this model is to address the existing gap in the field. The implementation of emotion EEG classification involves the utilization of a global average pooling layer and a fully linked layer, which are employed to leverage the discernible characteristics. In order to assess the effectiveness of the suggested methodology, we initially gathered a dataset of EEG recordings related to music-induced emotions.

Experiments

Subsequently, we ran comparative tests between the state-of-the-art algorithms and the method given in this study, utilizing this proprietary dataset. Furthermore, a publicly accessible dataset was included in the subsequent comparative trials.

Discussion

The experimental findings provide evidence that the suggested methodology outperforms existing approaches in the categorization of emotion EEG signals, both in binary (positive and negative) and ternary (positive, negative, and neutral) scenarios.

Topical Cellular/Tissue and Molecular Aspects Regarding Nonpharmacological Interventions in Alzheimer's Disease-A Systematic Review

One of the most complex and challenging developments at the beginning of the third millennium is the alarming increase in demographic aging, mainly-but not exclusively-affecting developed countries. This reality results in one of the harsh medical, social, and economic consequences: the continuously increasing number of people with dementia, including Alzheimer's disease (AD), which accounts for up to 80% of all such types of pathology. Its large and progressive disabling potential, which eventually leads to death, therefore represents an important public health matter, especially because there is no known cure for this disease. Consequently, periodic reappraisals of different therapeutic possibilities are necessary. For this purpose, we conducted this systematic literature review investigating nonpharmacological interventions for AD, including their currently known cellular and molecular action bases. This endeavor was based on the PRISMA method, by which we selected 116 eligible articles published during the last year. Because of the unfortunate lack of effective treatments for AD, it is necessary to enhance efforts toward identifying and improving various therapeutic and rehabilitative approaches, as well as related prophylactic measures.

Effects of long-term COVID-19 confinement and music stimulation on mental state and brain activity of young people

The Corona Virus Disease 2019 (COVID-19) pandemic may have had a negative emotional impact on individuals. This study investigated the effect of long-term lockdown and music on young people’s mood and neurophysiological responses in the prefrontal cortex (PFC). Fifteen healthy young adults were recruited and PFC activation was acquired using functional near-infrared spectroscopy during the conditions of resting, Stroop and music stimulation. The Depression Anxiety Stress Scales mental scale scores were simultaneously recorded. Mixed effect models, paired t-tests, one-way ANOVAs and Spearman analyses were adopted to analyse the experimental parameters. Stress, anxiety and depression levels increased significantly from Day 30 to Day 40. In terms of reaction time, both Stroop1 and Stroop2 were faster on Day 40 than on Day 30 (P = 0.01, P = 0.003). The relative concentration changes of oxyhemoglobin were significantly higher during premusic conditions than music stimulation and postmusic Stroop. The intensity of functional connectivity shifted from inter- to intracerebral over time. In conclusion, the reduced hemodynamic response of the PFC in healthy young adults is associated with negative emotions, especially anxiety, during lockdown. Immediate music stimulation appears to improve efficiency by altering the pattern of connections in PFC.

Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have potentially complementary characteristics that reflect the electrical and hemodynamic characteristics of neural responses, so EEG-fNIRS-based hybrid brain-computer interface (BCI) is the research hotspots in recent years. However, current studies lack a comprehensive systematic approach to properly fuse EEG and fNIRS data and exploit their complementary potential, which is critical for improving BCI performance. To address this issue, this study proposes a novel multimodal fusion framework based on multi-level progressive learning with multi-domain features. The framework consists of a multi-domain feature extraction process for EEG and fNIRS, a feature selection process based on atomic search optimization, and a multi-domain feature fusion process based on multi-level progressive machine learning. The proposed method was validated on EEG-fNIRS-based motor imagery (MI) and mental arithmetic (MA) tasks involving 29 subjects, and the experimental results show that multi-domain features provide better classification performance than single-domain features, and multi-modality provides better classification performance than single-modality. Furthermore, the experimental results and comparison with other methods demonstrated the effectiveness and superiority of the proposed method in EEG and fNIRS information fusion, it can achieve an average classification accuracy of 96.74% in the MI task and 98.42% in the MA task. Our proposed method may provide a general framework for future fusion processing of multimodal brain signals based on EEG-fNIRS.