EEG/fNIRS Based Workload Classification Using Functional Brain Connectivity and Machine Learning

Monitoring nap deprivation-induced fatigue using fNIRS and deep learning

Fatigue-induced incidents in transportation, aerospace, military, and other areas have been on the rise, posing a threat to human life and safety. The determination of fatigue states holds significant importance, especially through reliable and conveniently available physiological indicators. Here, a portable custom-built fNIRS system was used to monitor the fatigue state caused by nap deprivation. fNIRS signals in ten channels at the prefrontal cortex were collected, changes in blood oxygen concentration were analyzed, followed by a deep learning model to classify fatigue states. For the high-dimensionality and multi-channel characteristics of the fNIRS signal data, a novel 1D revised CNN-ResNet network was proposed based on the double-layer channel attenuation residual block. The results showed a 97.78% accuracy in fatigue state classification, significantly superior than several conventional methods. Furthermore, a fatigue-arousal experiment was designed to explore the feasibility of forced arousal of fatigued subjects through exercise stimulation. The fNIRS results showed a significant increase in brain activity with the conduction of exercise. The proposed method serves as a reliable tool for the evaluation of fatigue states, potentially reducing fatigue-induced harms and risks.

Multi-level cognitive state classification of learners using complex brain networks and interpretable machine learning

Identifying the cognitive state can help educators understand the evolving thought processes of learners, and it is important in promoting the development of higher-order thinking skills (HOTS). Cognitive neuroscience research identifies cognitive states by designing experimental tasks and recording electroencephalography (EEG) signals during task performance. However, most of the previous studies primarily concentrated on extracting features from individual channels in single-type tasks, ignoring the interconnection across channels. In this study, three learning activities (i.e., video watching activity, keyword extracting activity, and essay creating activity) were designed based on a revised Bloom's taxonomy and the Interactive-Constructive-Active-Passive framework and used with 31 college students. The EEG signals were recorded when they were engaged in these activities. First, whole-brain network temporal dynamics were characterized by EEG microstate sequence analysis. Such dynamic changes rely on learning activity and corresponding functional brain systems. Subsequently, phase locking value was used to construct synchrony-based functional brain networks. The network characteristics were extracted to be inputted into different machine learning classifiers: Support Vector Machine, K-Nearest Neighbour, Random Forest, and eXtreme Gradient Boosting (XGBoost). XGBoost showed superior performance in the classification of cognitive states, with an accuracy of 88.07%. Furthermore, SHapley Additive exPlanations (SHAP) was adopted to reveal the connections between different brain regions that contributed to the classification of cognitive state. SHAP analysis reveals that the connections in the frontal, temporal, and central regions are most important for the high cognitive state. Collectively, this study may provide further evidence for educators to design cognitive-guided instructional activities to enhance learners' HOTS.

Enhanced EEG-based cognitive workload detection using RADWT and machine learning

Understanding cognitive workload improves learning performance and provides insights into human cognitive processes. Estimating cognitive workload finds practical applications in adaptive learning systems, brain-computer interfaces, and cognitive monitoring. In this work, different levels of cognitive workload are investigated, and a classification approach based on the Rational-Dilation Wavelet Transform (RADWT) is proposed. RADWT excels at capturing the oscillatory behavior of EEG signal sub-bands, offering high precision through its ability to adaptively analyze both temporal and spectral dynamics. Different classifications of machine learning and feature selection techniques were evaluated to get optimum classification accuracy and identify the most effective combination of features for the used dataset. The analysis shows that the most relevant brain region in differentiating cognitive workload levels is the frontal region, along with alpha and theta rhythm sub-bands. Integrating RADWT with a Linear Support Vector Machine (LSVM) and minimum Redundancy Maximum Relevance (mRMR) feature selection method yields notable classification accuracy. Concretely, the model yields accuracies of 96.6% for 0-back vs.3-back, 94.9% for 0-back vs 2-back, 92.3% for 2-back vs 3-back, and 81.7% for the three-class scenario. These results confirm the validity of the method proposed for estimating cognitive workload using the RADWT- and machine learning-based approach. The results also offer insights into neural mechanisms and a foundation for advanced applications in adaptive systems, brain-computer interfaces, and cognitive monitoring.

Movement Sensing Opportunities for Monitoring Dynamic Cognitive States

In occupational domains such as sports, healthcare, driving, and military, both individuals and small groups are expected to perform challenging tasks under adverse conditions that induce transient cognitive states such as stress, workload, and uncertainty. Wearable and standoff 6DOF sensing technologies are advancing rapidly, including increasingly miniaturized yet robust inertial measurement units (IMUs) and portable marker-less infrared optical motion tracking. These sensing technologies may offer opportunities to track overt physical behavior and classify cognitive states relevant to human performance in diverse human-machine domains. We describe progress in research attempting to distinguish cognitive states by tracking movement behavior in both individuals and small groups, examining potential applications in sports, healthcare, driving, and the military. In the context of military training and operations, there are no generally accepted methods for classifying transient mental states such as uncertainty from movement-related data, despite its importance for shaping decision-making and behavior. To fill this gap, an example data set is presented including optical motion capture of rifle trajectories during a dynamic marksmanship task that elicits variable uncertainty; using machine learning, we demonstrate that features of weapon trajectories capturing the complexity of motion are valuable for classifying low versus high uncertainty states. We argue that leveraging metrics of human movement behavior reveals opportunities to complement relatively costly and less portable neurophysiological sensing technologies and enables domain-specific human-machine interfaces to support a wide range of cognitive functions.

Temporal attention fusion network with custom loss function for EEG-fNIRS classification

Objective.Methods that can detect brain activities accurately are crucial owing to the increasing prevalence of neurological disorders. In this context, a combination of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) offers a powerful approach to understanding normal and pathological brain functions, thereby overcoming the limitations of each modality, such as susceptibility to artifacts of EEG and limited temporal resolution of fNIRS. However, challenges such as class imbalance and inter-class variability within multisubject data hinder their full potential.Approach.To address this issue, we propose a novel temporal attention fusion network (TAFN) with a custom loss function. The TAFN model incorporates attention mechanisms to its long short-term memory and temporal convolutional layers to accurately capture spatial and temporal dependencies in the EEG-fNIRS data. The custom loss function combines class weights and asymmetric loss terms to ensure the precise classification of cognitive and motor intentions, along with addressing class imbalance issues.Main results.Rigorous testing demonstrated the exceptional cross-subject accuracy of the TAFN, exceeding 99% for cognitive tasks and 97% for motor imagery (MI) tasks. Additionally, the ability of the model to detect subtle differences in epilepsy was analyzed using scalp topography in MI tasks.Significance.This study presents a technique that outperforms traditional methods for detecting high-precision brain activity with subtle differences in the associated patterns. This makes it a promising tool for applications such as epilepsy and seizure detection, in which discerning subtle pattern differences is of paramount importance.

Strategic Integration: A Cross-Disciplinary Review of the fNIRS-EEG Dual-Modality Imaging System for Delivering Multimodal Neuroimaging to Applications

Background: Recent years have seen a surge of interest in dual-modality imaging systems that integrate functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to probe brain function. This review aims to explore the advancements and clinical applications of this technology, emphasizing the synergistic integration of fNIRS and EEG. Methods: The review begins with a detailed examination of the fundamental principles and distinctive features of fNIRS and EEG techniques. It includes critical technical specifications, data-processing methodologies, and analysis techniques, alongside an exhaustive evaluation of 30 seminal studies that highlight the strengths and weaknesses of the fNIRS-EEG bimodal system. Results: The paper presents multiple case studies across various clinical domains-such as attention-deficit hyperactivity disorder, infantile spasms, depth of anesthesia, intelligence quotient estimation, and epilepsy-demonstrating the fNIRS-EEG system's potential in uncovering disease mechanisms, evaluating treatment efficacy, and providing precise diagnostic options. Noteworthy research findings and pivotal breakthroughs further reinforce the developmental trajectory of this interdisciplinary field. Conclusions: The review addresses challenges and anticipates future directions for the fNIRS-EEG dual-modal imaging system, including improvements in hardware and software, enhanced system performance, cost reduction, real-time monitoring capabilities, and broader clinical applications. It offers researchers a comprehensive understanding of the field, highlighting the potential applications of fNIRS-EEG systems in neuroscience and clinical medicine.

A Brain Network Analysis Model for Motion Sickness in Electric Vehicles Based on EEG and fNIRS Signal Fusion

Motion sickness is a common issue in electric vehicles, significantly impacting passenger comfort. This study aims to develop a functional brain network analysis model by integrating electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals to evaluate motion sickness symptoms. During real-world testing with the Feifan F7 series of new energy-electric vehicles from SAIC Motor Corp, data were collected from 32 participants. The EEG signals were divided into four frequency bands: delta-range, theta-range, alpha-range, and beta-range, and brain oxygenation variation was calculated from the fNIRS signals. Functional connectivity between brain regions was measured to construct functional brain network models for motion sickness analysis. A motion sickness detection model was developed using a graph convolutional network (GCN) to integrate EEG and fNIRS data. Our results show significant differences in brain functional connectivity between participants in motion and non-motion sickness states. The model that combined fNIRS data with high-frequency EEG signals achieved the best performance, improving the F1 score by 11.4% compared to using EEG data alone and by 8.2% compared to using fNIRS data alone. These results highlight the effectiveness of integrating EEG and fNIRS signals using GCN for motion sickness detection. They demonstrate the model's superiority over single-modality approaches, showcasing its potential for real-world applications in electric vehicles.

Simultaneous EEG-fNIRS Data Classification Through Selective Channel Representation and Spectrogram Imaging

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) can facilitate the advancement of brain-computer interfaces (BCIs). However, existing research in this domain has grappled with the challenge of the efficient selection of features, resulting in the underutilization of the temporal richness of EEG and the spatial specificity of fNIRS data.To effectively address this challenge, this study proposed a deep learning architecture called the multimodal DenseNet fusion (MDNF) model that was trained on two-dimensional (2D) EEG data images, leveraging advanced feature extraction techniques. The model transformed EEG data into 2D images using a short-time Fourier transform, applied transfer learning to extract discriminative features, and consequently integrated them with fNIRS-derived spectral entropy features. This approach aimed to bridge existing gaps in EEG-fNIRS-based BCI research by enhancing classification accuracy and versatility across various cognitive and motor imagery tasks.Experimental results on two public datasets demonstrated the superiority of our model over existing state-of-the-art methods.Thus, the high accuracy and precise feature utilization of the MDNF model demonstrates the potential in clinical applications for neurodiagnostics and rehabilitation, thereby paving the method for patient-specific therapeutic strategies.

Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks

The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG-fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO2, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN-LSTM, CNN-GRU, LSTM-GRU, and CNN-LSTM-GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN-LSTM-GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN-LSTM. The experimental result shows that decision-fused EEG-HbO2-HbR and EEG-fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN-LSTM-GRU in Dataset 01 and CNN-LSTM in Dataset 02 yield the highest performance.

Neurovascular Coupling Analysis Based on Multivariate Variational Gaussian Process Convergent Cross-Mapping

Neurovascular coupling (NVC) provides important insights into the intricate activity of brain functioning and may aid in the early diagnosis of brain diseases. Emerging evidences have shown that NVC could be assessed by the coupling between electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). However, this endeavor presents significant challenges due to the absence of standardized methodologies and reliable techniques for coupling analysis of these two modalities. In this study, we introduced a novel method, i.e., the collaborative multi-output variational Gaussian process convergent cross-mapping (CMVGP-CCM) approach to advance coupling analysis of EEG and fNIRS. To validate the robustness and reliability of the CMVGP-CCM method, we conducted extensive experiments using chaotic time series models with varying noise levels, sequence lengths, and causal driving strengths. In addition, we employed the CMVGP-CCM method to explore the NVC between EEG and fNIRS signals collected from 26 healthy participants using a working memory (WM) task. Results revealed a significant causal effect of EEG signals, particularly the delta, theta, and alpha frequency bands, on the fNIRS signals during WM. This influence was notably observed in the frontal lobe, and its strength exhibited a decline as cognitive demands increased. This study illuminates the complex connections between brain electrical activity and cerebral blood flow, offering new insights into the underlying NVC mechanisms of WM.

Novel Feature Generation for Classification of Motor Activity from Functional Near-Infrared Spectroscopy Signals Using Machine Learning

Recent research in the field of cognitive motor action decoding focuses on data acquired from Functional Near-Infrared Spectroscopy (fNIRS) and its analysis. This research aims to classify two different motor activities, namely, mental drawing (MD) and spatial navigation (SN), using fNIRS data from non-motor baseline data and other motor activities. Accurate activity detection in non-stationary signals like fNIRS is challenging and requires complex feature descriptors. As a novel framework, a new feature generation by fusion of wavelet feature, Hilbert, symlet, and Hjorth parameters is proposed for improving the accuracy of the classification. This new fused feature has statistical descriptor elements, time-localization in the frequency domain, edge feature, texture features, and phase information to detect and locate the activity accurately. Three types of independent component analysis, including FastICA, Picard, and Infomax were implemented for preprocessing which removes noises and motion artifacts. Two independent binary classifiers are designed to handle the complexity of classification in which one is responsible for mental drawing (MD) detection and the other one is spatial navigation (SN). Four different types of algorithms including nearest neighbors (KNN), Linear Discriminant Analysis (LDA), light gradient-boosting machine (LGBM), and Extreme Gradient Boosting (XGBOOST) were implemented. It has been identified that the LGBM classifier gives high accuracies-98% for mental drawing and 97% for spatial navigation. Comparison with existing research proves that the proposed method gives the highest classification accuracies. Statistical validation of the proposed new feature generation by the Kruskal-Wallis H-test and Mann-Whitney U non-parametric test proves the reliability of the proposed mechanism.

Advancements in non-invasive microwave brain stimulation: A comprehensive survey

This survey provides a comprehensive insight into the world of non-invasive brain stimulation and focuses on the evolving landscape of deep brain stimulation through microwave research. Non-invasive brain stimulation techniques provide new prospects for comprehending and treating neurological disorders. We investigate the methods shaping the future of deep brain stimulation, emphasizing the role of microwave technology in this transformative journey. Specifically, we explore antenna structures and optimization strategies to enhance the efficiency of high-frequency microwave stimulation. These advancements can potentially revolutionize the field by providing a safer and more precise means of modulating neural activity. Furthermore, we address the challenges that researchers currently face in the realm of microwave brain stimulation. From safety concerns to methodological intricacies, this survey outlines the barriers that must be overcome to fully unlock the potential of this technology. This survey serves as a roadmap for advancing research in microwave brain stimulation, pointing out potential directions and innovations that promise to reshape the field.

Neuroimaging Study of Brain Functional Differences in Generalized Anxiety Disorder and Depressive Disorder

Generalized anxiety disorder (GAD) and depressive disorder (DD) are distinct mental disorders, which are characterized by complex and unique neuroelectrophysiological mechanisms in psychiatric neurosciences. The understanding of the brain functional differences between GAD and DD is crucial for the accurate diagnosis and clinical efficacy evaluation. The aim of this study was to reveal the differences in functional brain imaging between GAD and DD based on multidimensional electroencephalogram (EEG) characteristics. To this end, 10 min resting-state EEG signals were recorded from 38 GAD and 34 DD individuals. Multidimensional EEG features were subsequently extracted, which include power spectrum density (PSD), fuzzy entropy (FE), and phase lag index (PLI). Then, a direct statistical analysis (i.e., ANOVA) and three ensemble learning models (i.e., Random Forest (RF), Light Gradient Boosting Machine (LightGBM), eXtreme Gradient Boosting (XGBoost)) were used on these EEG features for the differential recognitions. Our results showed that DD has significantly higher PSD values in the alpha1 and beta band, and a higher FE in the beta band, in comparison with GAD, along with the aberrant functional connections in all four bands between GAD and DD. Moreover, machine learning analysis further revealed that the distinct features predominantly occurred in the beta band and functional connections. Here, we show that DD has higher power and more complex brain activity patterns in the beta band and reorganized brain functional network structures in all bands compared to GAD. In sum, these findings move towards the practical identification of brain functional differences between GAD and DD.

From local counterfactuals to global feature importance: efficient, robust, and model-agnostic explanations for brain connectivity networks

BACKGROUND

Explainable artificial intelligence (XAI) is a technology that can enhance trust in mental state classifications by providing explanations for the reasoning behind artificial intelligence (AI) models outputs, especially for high-dimensional and highly-correlated brain signals. Feature importance and counterfactual explanations are two common approaches to generate these explanations, but both have drawbacks. While feature importance methods, such as shapley additive explanations (SHAP), can be computationally expensive and sensitive to feature correlation, counterfactual explanations only explain a single outcome instead of the entire model.

METHODS

To overcome these limitations, we propose a new procedure for computing global feature importance that involves aggregating local counterfactual explanations. This approach is specifically tailored to fMRI signals and is based on the hypothesis that instances close to the decision boundary and their counterfactuals mainly differ in the features identified as most important for the downstream classification task. We refer to this proposed feature importance measure as Boundary Crossing Solo Ratio (BoCSoR), since it quantifies the frequency with which a change in each feature in isolation leads to a change in classification outcome, i.e., the crossing of the model's decision boundary.

RESULTS AND CONCLUSIONS

Experimental results on synthetic data and real publicly available fMRI data from the Human Connect project show that the proposed BoCSoR measure is more robust to feature correlation and less computationally expensive than state-of-the-art methods. Additionally, it is equally effective in providing an explanation for the behavior of any AI model for brain signals. These properties are crucial for medical decision support systems, where many different features are often extracted from the same physiological measures and a gold standard is absent. Consequently, computing feature importance may become computationally expensive, and there may be a high probability of mutual correlation among features, leading to unreliable results from state-of-the-art XAI methods.

Altered Functional Brain Network Structure between Patients with High and Low Generalized Anxiety Disorder

To investigate the differences in functional brain network structures between patients with a high level of generalized anxiety disorder (HGAD) and those with a low level of generalized anxiety disorder (LGAD), a resting-state electroencephalogram (EEG) was recorded in 30 LGAD patients and 21 HGAD patients. Functional connectivity between all pairs of brain regions was determined by the Phase Lag Index (PLI) to construct a functional brain network. Then, the characteristic path length, clustering coefficient, and small world were calculated to estimate functional brain network structures. The results showed that the PLI values of HGAD were significantly increased in alpha2, and significantly decreased in the theta and alpha1 rhythms, and the small-world attributes for both HGAD patients and LGAD patients were less than one for all the rhythms. Moreover, the small-world values of HGAD were significantly lower than those of LGAD in the theta and alpha2 rhythms, which indicated that the brain functional network structure would deteriorate with the increase in generalized anxiety disorder (GAD) severity. Our findings may play a role in the development and understanding of LGAD and HGAD to determine whether interventions that target these brain changes may be effective in treating GAD.