EEG classification of driver mental states by deep learning

CubicPat: Investigations on the Mental Performance and Stress Detection Using EEG Signals

Background\Objectives: Solving the secrets of the brain is a significant challenge for researchers. This work aims to contribute to this area by presenting a new explainable feature engineering (XFE) architecture designed to obtain explainable results related to stress and mental performance using electroencephalography (EEG) signals. Materials and Methods: Two EEG datasets were collected to detect mental performance and stress. To achieve classification and explainable results, a new XFE model was developed, incorporating a novel feature extraction function called Cubic Pattern (CubicPat), which generates a three-dimensional feature vector by coding channels. Classification results were obtained using the cumulative weighted iterative neighborhood component analysis (CWINCA) feature selector and the t-algorithm-based k-nearest neighbors (tkNN) classifier. Additionally, explainable results were generated using the CWINCA selector and Directed Lobish (DLob). Results: The CubicPat-based model demonstrated both classification and interpretability. Using 10-fold cross-validation (CV) and leave-one-subject-out (LOSO) CV, the introduced CubicPat-driven model achieved over 95% and 75% classification accuracies, respectively, for both datasets. Conclusions: The interpretable results were obtained by deploying DLob and statistical analysis.

Causal brain network analysis of driving fatigue based on generalized orthogonalized partially directed coherence

Driving fatigue is a serious threat to driving safety. Therefore, it is of great significance to accurately detect driving fatigue. In this study, the generalized orthogonal partial directed coherence (gOPDC) algorithm, which measures the time-frequency domain interaction of electroencephalogram (EEG) signals, was used to accurately estimate the connectivity between cortical channels. The causal brain network of driver continuous driving is constructed. The results show that the clustering coefficient and global efficiency tend to decrease with the increase in driving time. Causal information flow in the left prefrontal, parietal, occipital regions and the right posterior frontal region increased significantly when subjects transitioned from awake to fatigued, while causal information flow in the right prefrontal, parietal, occipital regions and the left posterior frontal region decreased mutually significantly. Compared with the traditional driving fatigue algorithm, the accuracy of the method used in this paper is higher than the traditional methods.

Decoding of movement-related cortical potentials at different speeds

The decoding of electroencephalogram (EEG) signals, especially motion-related cortical potentials (MRCP), is vital for the early detection of motor intent before movement execution. To enhance the decoding accuracy of MRCP and promote the application of early motion intention in active rehabilitation training, we propose a method for decoding MRCP signals. Specifically, an experimental paradigm is designed for the efficient capture of MRCP signals. Moreover, a feature extraction method based on differentiation is proposed to effectively characterize action variability. Six subjects were recruited to validate the effectiveness of the decoding method. Experiments such as fixed-window classification, sliding-window detection, and asynchronous analysis demonstrate that the method can detect motion intention 316 milliseconds before action execution and is capable of continuously detecting both rapid and slow movements.

An improved CapsNet based on data augmentation for driver vigilance estimation with forehead single-channel EEG

Various studies have shown that it is necessary to estimate the drivers' vigilance to reduce the occurrence of traffic accidents. Most existing EEG-based vigilance estimation studies have been performed on intra-subject and multi-channel signals, and these methods are too costly and complicated to implement in practice. Hence, aiming at the problem of cross-subject vigilance estimation of single-channel EEG signals, an estimation algorithm based on capsule network (CapsNet) is proposed. Firstly, we propose a new construction method of the input feature maps to fit the characteristics of CapsNet to improve the algorithm accuracy. Meanwhile, the self-attention mechanism is incorporated in the algorithm to focus on the key information in feature maps. Secondly, we propose substituting the traditional multi-channel signals with the single-channel signals to improve the utility of algorithm. Thirdly, since the single-channel signals carry fewer dimensions of the information compared to the multi-channel signals, we use the conditional generative adversarial network to improve the accuracy of single-channel signals by increasing the amount of data. The proposed algorithm is verified on the SEED-VIG, and Root-mean-square-error (RMSE) and Pearson Correlation Coefficient (PCC) are used as the evaluation metrics. The results show that the proposed algorithm improves the computing speed while the RMSE is reduced by 3%, and the PCC is improved by 12% compared to the mainstream algorithm. Experiment results prove the feasibility of using forehead single-channel EEG signals for cross-subject vigilance estimation and offering the possibility of lightweight EEG vigilance estimation devices for practical applications.

Integrating Large Language Model, EEG, and Eye-Tracking for Word-Level Neural State Classification in Reading Comprehension

With the recent proliferation of large language models (LLMs), such as Generative Pre-trained Transformers (GPT), there has been a significant shift in exploring human and machine comprehension of semantic language meaning. This shift calls for interdisciplinary research that bridges cognitive science and natural language processing (NLP). This pilot study aims to provide insights into individuals' neural states during a semantic inference reading-comprehension task. We propose jointly analyzing LLMs, eye-gaze, and electroencephalographic (EEG) data to study how the brain processes words with varying degrees of relevance to a keyword during reading. We also use feature engineering to improve the fixation-related EEG data classification while participants read words with high versus low relevance to the keyword. The best validation accuracy in this word-level classification is over 60% across 12 subjects. Words highly relevant to the inference keyword received significantly more eye fixations per word: 1.0584 compared to 0.6576, including words with no fixations. This study represents the first attempt to classify brain states at a word level using LLM-generated labels. It provides valuable insights into human cognitive abilities and Artificial General Intelligence (AGI), and offers guidance for developing potential reading-assisted technologies.

Cognitive workload estimation using physiological measures: a review

Estimating cognitive workload levels is an emerging research topic in the cognitive neuroscience domain, as participants' performance is highly influenced by cognitive overload or underload results. Different physiological measures such as Electroencephalography (EEG), Functional Magnetic Resonance Imaging, Functional near-infrared spectroscopy, respiratory activity, and eye activity are efficiently used to estimate workload levels with the help of machine learning or deep learning techniques. Some reviews focus only on EEG-based workload estimation using machine learning classifiers or multimodal fusion of different physiological measures for workload estimation. However, a detailed analysis of all physiological measures for estimating cognitive workload levels still needs to be discovered. Thus, this survey highlights the in-depth analysis of all the physiological measures for assessing cognitive workload. This survey emphasizes the basics of cognitive workload, open-access datasets, the experimental paradigm of cognitive tasks, and different measures for estimating workload levels. Lastly, we emphasize the significant findings from this review and identify the open challenges. In addition, we also specify future scopes for researchers to overcome those challenges.

SFT-Net: A Network for Detecting Fatigue From EEG Signals by Combining 4D Feature Flow and Attention Mechanism

Fatigued driving is a leading cause of traffic accidents, and accurately predicting driver fatigue can significantly reduce their occurrence. However, modern fatigue detection models based on neural networks often face challenges such as poor interpretability and insufficient input feature dimensions. This article proposes a novel Spatial-Frequency-Temporal Network (SFT-Net) method for detecting driver fatigue using electroencephalogram (EEG) data. Our approach integrates EEG signals' spatial, frequency, and temporal information to improve recognition performance. We transform the differential entropy of five frequency bands of EEG signals into a 4D feature tensor to preserve these three types of information. An attention module is then used to recalibrate the spatial and frequency information of each input 4D feature tensor time slice. The output of this module is fed into a depthwise separable convolution (DSC) module, which extracts spatial and frequency features after attention fusion. Finally, long short-term memory (LSTM) is used to extract the temporal dependence of the sequence, and the final features are output through a linear layer. We validate the effectiveness of our model on the SEED-VIG dataset, and experimental results demonstrate that SFT-Net outperforms other popular models for EEG fatigue detection. Interpretability analysis supports the claim that our model has a certain level of interpretability. Our work addresses the challenge of detecting driver fatigue from EEG data and highlights the importance of integrating spatial, frequency, and temporal information.

A review on evaluating mental stress by deep learning using EEG signals

Mental stress is a common problem that affects individuals all over the world. Stress reduces human functionality during routine work and may lead to severe health defects. Early detection of stress is important for preventing diseases and other negative health-related consequences of stress. Several neuroimaging techniques have been utilized to assess mental stress, however, due to its ease of use, robustness, and non-invasiveness, electroencephalography (EEG) is commonly used. This paper aims to fill a knowledge gap by reviewing the different EEG-related deep learning algorithms with a focus on Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs) for the evaluation of mental stress. The review focuses on data representation, individual deep neural network model architectures, hybrid models, and results amongst others. The contributions of the paper address important issues such as data representation and model architectures. Out of all reviewed papers, 67% used CNN, 9% LSTM, and 24% hybrid models. Based on the reviewed literature, we found that dataset size and different representations contributed to the performance of the proposed networks. Raw EEG data produced classification accuracy around 62% while using spectral and topographical representation produced up to 88%. Nevertheless, the roles of generalizability across different deep learning models and individual differences remain key areas of inquiry. The review encourages the exploration of innovative avenues, such as EEG data image representations concurrently with graph convolutional neural networks (GCN), to mitigate the impact of inter-subject variability. This novel approach not only allows us to harmonize structural nuances within the data but also facilitates the integration of temporal dynamics, thereby enabling a more comprehensive assessment of mental stress levels.

Deep learning-based auditory attention decoding in listeners with hearing impairment. J Neural Eng 2024;21:036022. [PMID: 38729132 DOI: 10.1088/1741-2552/ad49d7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Objective.This study develops a deep learning (DL) method for fast auditory attention decoding (AAD) using electroencephalography (EEG) from listeners with hearing impairment (HI). It addresses three classification tasks: differentiating noise from speech-in-noise, classifying the direction of attended speech (left vs. right) and identifying the activation status of hearing aid noise reduction algorithms (OFF vs. ON). These tasks contribute to our understanding of how hearing technology influences auditory processing in the hearing-impaired population.Approach.Deep convolutional neural network (DCNN) models were designed for each task. Two training strategies were employed to clarify the impact of data splitting on AAD tasks: inter-trial, where the testing set used classification windows from trials that the training set had not seen, and intra-trial, where the testing set used unseen classification windows from trials where other segments were seen during training. The models were evaluated on EEG data from 31 participants with HI, listening to competing talkers amidst background noise.Main results.Using 1 s classification windows, DCNN models achieve accuracy (ACC) of 69.8%, 73.3% and 82.9% and area-under-curve (AUC) of 77.2%, 80.6% and 92.1% for the three tasks respectively on inter-trial strategy. In the intra-trial strategy, they achieved ACC of 87.9%, 80.1% and 97.5%, along with AUC of 94.6%, 89.1%, and 99.8%. Our DCNN models show good performance on short 1 s EEG samples, making them suitable for real-world applications. Conclusion: Our DCNN models successfully addressed three tasks with short 1 s EEG windows from participants with HI, showcasing their potential. While the inter-trial strategy demonstrated promise for assessing AAD, the intra-trial approach yielded inflated results, underscoring the important role of proper data splitting in EEG-based AAD tasks.Significance.Our findings showcase the promising potential of EEG-based tools for assessing auditory attention in clinical contexts and advancing hearing technology, while also promoting further exploration of alternative DL architectures and their potential constraints.

Research on Fatigue Driving Detection Technology Based on CA-ACGAN

Driver fatigue represents a significant peril to global traffic safety, necessitating the advancement of potent fatigue monitoring methodologies to bolster road safety. This research introduces a conditional generative adversarial network with a classification head that integrates convolutional and attention mechanisms (CA-ACGAN) designed for the precise identification of fatigue driving states through the analysis of electroencephalography (EEG) signals. First, this study constructed a 4D feature data model capable of mirroring drivers' fatigue state, meticulously analyzing the EEG signals' frequency, spatial, and temporal dimensions. Following this, we present the CA-ACGAN framework, a novel integration of attention schemes, the bottleneck residual block, and the Transformer element. This integration was designed to refine the processing of EEG signals significantly. In utilizing a conditional generative adversarial network equipped with a classification header, the framework aims to distinguish fatigue states effectively. Moreover, it addresses the scarcity of authentic data through the generation of superior-quality synthetic data. Empirical outcomes illustrate that the CA-ACGAN model surpasses various extant methods in the fatigue detection endeavor on the SEED-VIG public dataset. Moreover, juxtaposed with leading-edge GAN models, our model exhibits an efficacy in in producing high-quality data that is clearly superior. This investigation confirms the CA-ACGAN model's utility in fatigue driving identification and suggests fresh perspectives for deep learning applications in time series data generation and processing.

Driving fatigue detection based on brain source activity and ARMA model

Fatigue among drivers is a significant issue in society, and according to organizational reports, it substantially contributes to accidents. So accurate fatigue detection in drivers plays a crucial role in reducing the number of people fatalities or injured resulting from accidents. Several methods are proposed for fatigue driver recognition among which electroencephalography (EEG) is one. This paper proposed a method for fatigue recognition by EEG signals with extracted features from source and sensor spaces. The proposed method starts with preprocessing by applying filtering and artifact rejection. Then source localization methods are applied to EEG signals for active source extraction. A multivariate autoregressive (MVAR) model is fitted to selected sources, and a dual Kalman filter is applied to estimate the source activity and their relationships. Then multivariate autoregressive moving average (ARMA) is fitted between EEG and source activity signals. Features are extracted from model parameters, source relationship matrix, and wavelet transform of EEG and source activity signals. The novelty of this approach is the use of ARMA model between source activities (as input) and EEG signals (as output) and feature extraction from source relations. Relevant features are selected using a combination of RelifF and neighborhood component analysis (NCA) methods. Three classifiers, namely k-nearest neighbor (KNN), support vector machine (SVM), and naive Bayesian (NB) classifiers, are employed to classify drivers. To improve performance, the final label for fatigue detection is calculated by combining these classifiers using the voting method. The results demonstrate that the proposed method accurately recognizes and classifies fatigued drivers with the ensemble classifiers in comparison with other methods.

The applied principles of EEG analysis methods in neuroscience and clinical neurology

Electroencephalography (EEG) is a non-invasive measurement method for brain activity. Due to its safety, high resolution, and hypersensitivity to dynamic changes in brain neural signals, EEG has aroused much interest in scientific research and medical fields. This article reviews the types of EEG signals, multiple EEG signal analysis methods, and the application of relevant methods in the neuroscience field and for diagnosing neurological diseases. First, three types of EEG signals, including time-invariant EEG, accurate event-related EEG, and random event-related EEG, are introduced. Second, five main directions for the methods of EEG analysis, including power spectrum analysis, time-frequency analysis, connectivity analysis, source localization methods, and machine learning methods, are described in the main section, along with different sub-methods and effect evaluations for solving the same problem. Finally, the application scenarios of different EEG analysis methods are emphasized, and the advantages and disadvantages of similar methods are distinguished. This article is expected to assist researchers in selecting suitable EEG analysis methods based on their research objectives, provide references for subsequent research, and summarize current issues and prospects for the future.

EEG-fest: few-shot based attention network for driver's drowsiness estimation with EEG signals

The leading factor behind most vehicular accidents is the driver's inattentiveness. To accurately determine driver's drowsiness, Electroencephalography (EEG) has been proven to be a reliable and effective method. Even though previous studies have developed accurate driver's drowsiness detection algorithms, certain challenges still persist, such as (a) limited training sample sizes, (b) detecting anomalous signals, and (c) achieving subject-independent classification. In this paper we propose a novel solution, names as EEG-Fest, which is a generalized few-shot model aimed at addressing the aforementioned limitations. The EEG-Fest has the ability to (a) classify a query sample's level of drowsiness with only a few support sample inputs (b) identify whether a query sample is anomalous signals or not, and (c) perform subject-independent classification. During the evaluation, our proposed EEG-Fest algorithm demonstrates better performance compared to other two conventional EEG algorithms in cross-subject validation.

Real driving environment EEG-based detection of driving fatigue using the wavelet scattering network

BACKGROUND

Driving fatigue is one of the main factors leading to traffic accidents. So, it is necessary to detect driver fatigue accurately and quickly.

NEW METHOD

To precisely detect driving fatigue in a real driving environment, this paper adopts a classification method for driving fatigue based on the wavelet scattering network (WSN). Firstly, electroencephalogram (EEG) signals of 12 subjects in the real driving environment are collected and categorized into two states: fatigue and awake. Secondly, the WSN algorithm extracts wavelet scattering coefficients of EEG signals, and these coefficients are used as input in support vector machine (SVM) as feature vectors for classification.

RESULTS

The results showed that the average classification accuracy of 12 subjects reached 99.33%; the average precision rate reached 99.28%; the average recall rate reached 98.27%; the average F1 score reached 98.74%; and the average classification accuracy of the public data set SEED-VIG reached 99.39%. The average precision, recall rate and F1 score reached 99.27%, 98.41% and 98.83% respectively.

COMPARISON WITH EXISTING METHODS

In addition, the WSN algorithm is compared with traditional convolutional neural network (CNN), Sparse-deep belief networks (SDBN), Spatio-temporal convolutional neural networks (STCNN), Long short-term memory (LSTM), and other methods, and it is found that WSN has higher classification accuracy.

CONCLUSION

Furthermore, this method has good versatility, providing excellent recognition effect on small sample data sets, and fast running time, making it convenient for real-time online monitoring of driver fatigue. Therefore, the WSN algorithm is promising in efficiently detecting driving fatigue state of drivers in real environments, contributing to improved traffic safety.

EEG-Based Cross-Subject Driver Drowsiness Recognition With an Interpretable Convolutional Neural Network

In the context of electroencephalogram (EEG)-based driver drowsiness recognition, it is still challenging to design a calibration-free system, since EEG signals vary significantly among different subjects and recording sessions. Many efforts have been made to use deep learning methods for mental state recognition from EEG signals. However, existing work mostly treats deep learning models as black-box classifiers, while what have been learned by the models and to which extent they are affected by the noise in EEG data are still underexplored. In this article, we develop a novel convolutional neural network combined with an interpretation technique that allows sample-wise analysis of important features for classification. The network has a compact structure and takes advantage of separable convolutions to process the EEG signals in a spatial-temporal sequence. Results show that the model achieves an average accuracy of 78.35% on 11 subjects for leave-one-out cross-subject drowsiness recognition, which is higher than the conventional baseline methods of 53.40%-72.68% and state-of-the-art deep learning methods of 71.75%-75.19%. Interpretation results indicate the model has learned to recognize biologically meaningful features from EEG signals, e.g., alpha spindles, as strong indicators of drowsiness across different subjects. In addition, we also explore reasons behind some wrongly classified samples with the interpretation technique and discuss potential ways to improve the recognition accuracy. Our work illustrates a promising direction on using interpretable deep learning models to discover meaningful patterns related to different mental states from complex EEG signals.

Assigning channel weights using an attention mechanism: an EEG interpolation algorithm

During the acquisition of electroencephalographic (EEG) signals, various factors can influence the data and lead to the presence of one or multiple bad channels. Bad channel interpolation is the use of good channels data to reconstruct bad channel, thereby maintaining the original dimensions of the data for subsequent analysis tasks. The mainstream interpolation algorithm assigns weights to channels based on the physical distance of the electrodes and does not take into account the effect of physiological factors on the EEG signal. The algorithm proposed in this study utilizes an attention mechanism to allocate channel weights (AMACW). The model gets the correlation among channels by learning from good channel data. Interpolation assigns weights based on learned correlations without the need for electrode location information, solving the difficulty that traditional methods cannot interpolate bad channels at unknown locations. To avoid an overly concentrated weight distribution of the model when generating data, we designed the channel masking (CM). This method spreads attention and allows the model to utilize data from multiple channels. We evaluate the reconstruction performance of the model using EEG data with 1 to 5 bad channels. With EEGLAB's interpolation method as a performance reference, tests have shown that the AMACW models can effectively reconstruct bad channels.

A regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based cross-subject fatigue detection

Fatigue, one of the most important factors affecting road safety, has attracted many researchers' attention. Most existing fatigue detection methods are based on feature engineering and classification models. The feature engineering is greatly influenced by researchers' domain knowledge, which will lead to a poor performance in fatigue detection, especially in cross-subject experiment design. In addition, fatigue detection is often simplified as a classification problem of several discrete states. Models based on deep learning can realize automatic feature extraction without the limitation of researcher's domain knowledge. Therefore, this paper proposes a regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based (EEG-based) cross-subject fatigue detection. At the same time, a twofold random-offset zero-overlapping sampling method is proposed to train a bigger model and reduce overfitting. Compared with existing results, the proposed method achieves a much better result of 0.94 correlation coefficient (COR) and 0.09 root mean square error (RMSE) in a within-subject experiment design. What is more, there is no misclassification between awake and drowsy states. For cross-subject experiment design, the COR and RMSE are 0.79 and 0.15, respectively, which are close to the existing within-subject results and better than similar cross-subject results. The cross-subject regression model is very important for fatigue detection application since the fatigue indication is more precise than several discrete states and no model calibration is required for a new user. The twofold random-offset zero-overlapping sampling method can also be used as a reference by other EEG-based deep learning research.

Drivers' Mental Engagement Analysis Using Multi-Sensor Fusion Approaches Based on Deep Convolutional Neural Networks

In this paper, we present a comprehensive assessment of individuals' mental engagement states during manual and autonomous driving scenarios using a driving simulator. Our study employed two sensor fusion approaches, combining the data and features of multimodal signals. Participants in our experiment were equipped with Electroencephalogram (EEG), Skin Potential Response (SPR), and Electrocardiogram (ECG) sensors, allowing us to collect their corresponding physiological signals. To facilitate the real-time recording and synchronization of these signals, we developed a custom-designed Graphical User Interface (GUI). The recorded signals were pre-processed to eliminate noise and artifacts. Subsequently, the cleaned data were segmented into 3 s windows and labeled according to the drivers' high or low mental engagement states during manual and autonomous driving. To implement sensor fusion approaches, we utilized two different architectures based on deep Convolutional Neural Networks (ConvNets), specifically utilizing the Braindecode Deep4 ConvNet model. The first architecture consisted of four convolutional layers followed by a dense layer. This model processed the synchronized experimental data as a 2D array input. We also proposed a novel second architecture comprising three branches of the same ConvNet model, each with four convolutional layers, followed by a concatenation layer for integrating the ConvNet branches, and finally, two dense layers. This model received the experimental data from each sensor as a separate 2D array input for each ConvNet branch. Both architectures were evaluated using a Leave-One-Subject-Out (LOSO) cross-validation approach. For both cases, we compared the results obtained when using only EEG signals with the results obtained by adding SPR and ECG signals. In particular, the second fusion approach, using all sensor signals, achieved the highest accuracy score, reaching 82.0%. This outcome demonstrates that our proposed architecture, particularly when integrating EEG, SPR, and ECG signals at the feature level, can effectively discern the mental engagement of drivers.

Self-Attentive Channel-Connectivity Capsule Network for EEG-Based Driving Fatigue Detection

Deep neural networks have recently been successfully extended to EEG-based driving fatigue detection. Nevertheless, most existing models fail to reveal the intrinsic inter-channel relations that are known to be beneficial for EEG-based classification. Additionally, these models require substantial data for training, which is often impractical due to the high cost of data collection. To simultaneously address these two issues, we propose a Self-Attentive Channel-Connectivity Capsule Network (SACC-CapsNet) for EEG-based driving fatigue detection in this paper. SACC-CapsNet starts with a temporal-channel attention module to investigate the critical temporal information and important channels for driving fatigue detection, refining the input EEG signals. Subsequently, the refined EEG data are transformed into a channel covariance matrix to capture the inter-channel relations, followed by selective kernel attention to extract the highly discriminative channel-connectivity features. Finally, a capsule neural network is employed to effectively learn the relationships between connectivity features, which is more suitable for limited data. To confirm the effectiveness of SACC-CapsNet, we collected 24-channel EEG data from 31 subjects (mean age=23.13±2.68 years, male/female=18/13) in a simulated fatigue driving environment. Extensive experiments were conducted with the acquired data, and the comparison results show that our proposed model outperforms state-of-the-art methods. Additionally, the channel covariance matrix learned from SACC-CapsNet reveals that the frontal pole is most informative for detecting driving fatigue, followed by the parietal and central regions. Intriguingly, the temporal-channel attention module can enhance the significance of these critical regions, and the reconstructed channel covariance matrix generated by the decoder network of SACC-CapsNet can effectively preserve valuable information about them.

T-A-MFFNet: Multi-feature fusion network for EEG analysis and driving fatigue detection based on time domain network and attention network

Driving fatigue detection based on EEG signals is a research hotspot in applying brain-computer interfaces. EEG signal is complex, unstable, and nonlinear. Most existing methods rarely analyze the data characteristics from multiple dimensions, so it takes work to analyze the data comprehensively. To analyze EEG signals more comprehensively, this paper evaluates a feature extraction strategy of EEG data based on differential entropy (DE). This method combines the characteristics of different frequency bands, extracts the frequency domain characteristics of EEG, and retains the spatial information between channels. This paper proposes a multi-feature fusion network (T-A-MFFNet) based on the time domain and attention network. The model is composed of a time domain network (TNet), channel attention network (CANet), spatial attention network (SANet), and multi-feature fusion network(MFFNet) based on a squeeze network. T-A-MFFNet aims to learn more valuable features from the input data to achieve good classification results. Specifically, the TNet network extracts high-level time series information from EEG data. CANet and SANet are used to fuse channel and spatial features. They use MFFNet to merge multi-dimensional features and realize classification. The validity of the model is verified on the SEED-VIG dataset. The experimental results show that the accuracy of the proposed method reaches 85.65 %, which is superior to the current popular model. The proposed method can learn more valuable information from EEG signals to improve the ability to identify fatigue status and promote the development of the research field of driving fatigue detection based on EEG signals.

Mental workload level assessment based on compounded hysteresis effect

In the domain of neuroergonomics, cognitive workload estimation has taken a significant concern among the researchers. This is because the knowledge gathered from its estimation is useful for distributing tasks among the operators, understanding human capability and intervening operators at times of havoc. Brain signals give a promising prospective for understanding cognitive workload. For this, electroencephalography (EEG) is by far the most efficient modality in interpreting the covert information arising in the brain. The present work explores the feasibility of EEG rhythms for monitoring continuous change occurring in a person's cognitive workload. This continuous monitoring is achieved by graphicallyinterpreting the cumulative effect of changes in EEG rhythms observed in the current instance and the former instance based on the hysteresis effect. In this work, classification is done to predict the data class label using an artificial neural network (ANN) architecture. The proposed model gives a classification accuracy of 98.66%.

Exploring deep residual network based features for automatic schizophrenia detection from EEG

Schizophrenia is a severe mental illness which can cause lifelong disability. Most recent studies on the Electroencephalogram (EEG)-based diagnosis of schizophrenia rely on bespoke/hand-crafted feature extraction techniques. Traditional manual feature extraction methods are time-consuming, imprecise, and have a limited ability to balance accuracy and efficiency. Addressing this issue, this study introduces a deep residual network (deep ResNet) based feature extraction design that can automatically extract representative features from EEG signal data for identifying schizophrenia. This proposed method consists of three stages: signal pre-processing by average filtering method, extraction of hidden patterns of EEG signals by deep ResNet, and classification of schizophrenia by softmax layer. To assess the performance of the obtained deep features, ResNet softmax classifier and also several machine learning (ML) techniques are applied on the same feature set. The experimental results for a Kaggle schizophrenia EEG dataset show that the deep features with support vector machine classifier could achieve the highest performances (99.23% accuracy) compared to the ResNet classifier. Furthermore, the proposed model performs better than the existing approaches. The findings suggest that our proposed strategy has capability to discover important biomarkers for automatic diagnosis of schizophrenia from EEG, which will aid in the development of a computer assisted diagnostic system by specialists.

EEG driving fatigue detection based on log-Mel spectrogram and convolutional recurrent neural networks

Driver fatigue detection is one of the essential tools to reduce accidents and improve traffic safety. Its main challenge lies in the problem of how to identify the driver's fatigue state accurately. Existing detection methods include yawning and blinking based on facial expressions and physiological signals. Still, lighting and the environment affect the detection results based on facial expressions. In contrast, the electroencephalographic (EEG) signal is a physiological signal that directly responds to the human mental state, thus reducing the impact on the detection results. This paper proposes a log-Mel spectrogram and Convolution Recurrent Neural Network (CRNN) model based on EEG to implement driver fatigue detection. This structure allows the advantages of the different networks to be exploited to overcome the disadvantages of using them individually. The process is as follows: first, the original EEG signal is subjected to a one-dimensional convolution method to achieve a Short Time Fourier Transform (STFT) and passed through a Mel filter bank to obtain a logarithmic Mel spectrogram, and then the resulting logarithmic Mel spectrogram is fed into a fatigue detection model to complete the fatigue detection task for the EEG signals. The fatigue detection model consists of a 6-layer convolutional neural network (CNN), bi-directional recurrent neural networks (Bi-RNNs), and a classifier. In the modeling phase, spectrogram features are transported to the 6-layer CNN to automatically learn high-level features, thereby extracting temporal features in the bi-directional RNN to obtain spectrogram-temporal information. Finally, the alert or fatigue state is obtained by a classifier consisting of a fully connected layer, a ReLU activation function, and a softmax function. Experiments were conducted on publicly available datasets in this study. The results show that the method can accurately distinguish between alert and fatigue states with high stability. In addition, the performance of four existing methods was compared with the results of the proposed method, all of which showed that the proposed method could achieve the best results so far.

Predicting collision cases at unsignalized intersections using EEG metrics and driving simulator platform

Unsignalized intersection collision has been one of the most dangerous accidents in the world. How to identify road hazards and predict the potential intersection collision ahead are challenging problems in traffic safety. This paper studies the feasibility of EEG metrics to forecast road hazards and presents an improved neural network model to predict intersection collision based on EEG metrics and driving behavior. It is demonstrated that EEG metrics show significant differences between collision and non-collision cases. It indicates that EEG metrics can serve as effective indicators to predict the collision probability. The drivers with higher relative power in fast frequency band (alpha and beta), lower relative power in slow frequency band (delta and theta) are more likely to have conflicts. The prediction using three machine learning models (Multi-layer perceptron (MLP), Logistic regression (LR) and Random forest (RF)) based on three input datasets (only EEG metrics, only driving behavior and combined EEG metrics with driving behavior) are compared. The results show that for single time point prediction, MLP model has the highest accuracy among three machine learning models. The model solely based on EEG metrics datasets has higher accuracy than driving behavior as well as combined datasets. However, for multi-time point prediction, the accuracy of MLP is only 73.9%, worse than LR and RF. We improved the MLP model by adding attention mechanism layer and using random forest model to select important features. As a consequence, the accuracy is greatly improved and reaches 88%. This study demonstrates the importance and feasibility of EEG signals to identify unsafe drivers ahead. The improved neural network model can be helpful to reduce intersection accidents and improve traffic safety.

Status of deep learning for EEG-based brain-computer interface applications

In the previous decade, breakthroughs in the central nervous system bioinformatics and computational innovation have prompted significant developments in brain-computer interface (BCI), elevating it to the forefront of applied science and research. BCI revitalization enables neurorehabilitation strategies for physically disabled patients (e.g., disabled patients and hemiplegia) and patients with brain injury (e.g., patients with stroke). Different methods have been developed for electroencephalogram (EEG)-based BCI applications. Due to the lack of a large set of EEG data, methods using matrix factorization and machine learning were the most popular. However, things have changed recently because a number of large, high-quality EEG datasets are now being made public and used in deep learning-based BCI applications. On the other hand, deep learning is demonstrating great prospects for solving complex relevant tasks such as motor imagery classification, epileptic seizure detection, and driver attention recognition using EEG data. Researchers are doing a lot of work on deep learning-based approaches in the BCI field right now. Moreover, there is a great demand for a study that emphasizes only deep learning models for EEG-based BCI applications. Therefore, we introduce this study to the recent proposed deep learning-based approaches in BCI using EEG data (from 2017 to 2022). The main differences, such as merits, drawbacks, and applications are introduced. Furthermore, we point out current challenges and the directions for future studies. We argue that this review study will help the EEG research community in their future research.

A Wide Dynamic Range Sigma-Delta Modulator for EEG Acquisition Using Randomized DWA and Dynamic-Modulated Scaling-Down Techniques

This paper proposes a wide dynamic range (DR) and high-resolution discrete-time (DT) 2-order 4-bit sigma-delta modulator with a novel dynamic-modulated scaling-down (DM-SD) technology for non-invasive electroencephalogram (EEG) acquisition. The DM-SD technology can expand the input dynamic range and suppress large input offsets at the same time. The modulator was designed with 180nm CMOS technology with an area of 0.49 mm2. We achieve a 118.1 dB SNDR when the input signal is 437.5 Hz and the signal bandwidth is 1500 Hz. Due to the proposed DM-SD technology, the DR is expanded to 126 dB. The power consumption of the whole modulator is 1.6 mW and a 177.8 dB Schreier figure-of-merit (FoMs) is realized.

EEG-Based Mental Tasks Recognition via a Deep Learning-Driven Anomaly Detector

This paper introduces an unsupervised deep learning-driven scheme for mental tasks' recognition using EEG signals. To this end, the Multichannel Wiener filter was first applied to EEG signals as an artifact removal algorithm to achieve robust recognition. Then, a quadratic time-frequency distribution (QTFD) was applied to extract effective time-frequency signal representation of the EEG signals and catch the EEG signals' spectral variations over time to improve the recognition of mental tasks. The QTFD time-frequency features are employed as input for the proposed deep belief network (DBN)-driven Isolation Forest (iF) scheme to classify the EEG signals. Indeed, a single DBN-based iF detector is constructed based on each class's training data, with the class's samples as inliers and all other samples as anomalies (i.e., one-vs.-rest). The DBN is considered to learn pertinent information without assumptions on the data distribution, and the iF scheme is used for data discrimination. This approach is assessed using experimental data comprising five mental tasks from a publicly available database from the Graz University of Technology. Compared to the DBN-based Elliptical Envelope, Local Outlier Factor, and state-of-the-art EEG-based classification methods, the proposed DBN-based iF detector offers superior discrimination performance of mental tasks.

Smart Steering Sleeve (S3): A Non-Intrusive and Integrative Sensing Platform for Driver Physiological Monitoring

Driving is a ubiquitous activity that requires both motor skills and cognitive focus. These aspects become more problematic for some seniors, who have underlining medical conditions and tend to lose some of these capabilities. Therefore, driving can be used as a controlled environment for the frequent, non-intrusive monitoring of bio-physical and cognitive status within drivers. Such information can then be utilized for enhanced assistive vehicle controls and/or driver health monitoring. In this paper, we present a novel multi-modal smart steering sleeve (S3) system with an integrated sensing platform that can non-intrusively and continuously measure a driver's physiological signals, including electrodermal activity (EDA), electromyography (EMG), and hand pressure. The sensor suite was developed by combining low-cost interdigitated electrodes with a piezoresistive force sensor on a single, flexible polymer substrate. Comprehensive characterizations on the sensing modalities were performed with promising results demonstrated. The sweat-sensing unit (SSU) for EDA monitoring works under a 100 Hz alternative current (AC) source. The EMG signal acquired by the EMG-sensing unit (EMGSU) was amplified to within 5 V. The force-sensing unit (FSU) for hand pressure detection has a range of 25 N. This flexible sensor was mounted on an off-the-shelf steering wheel sleeve, making it an add-on system that can be installed on any existing vehicles for convenient and wide-coverage driver monitoring. A cloud-based communication scheme was developed for the ease of data collection and analysis. Sensing platform development, performance, and limitations, as well as other potential applications, are discussed in detail in this paper.

EEG brain signals to detect the sleep health of a driver: An automated framework system based on deep learning

Mental fatigue is complex disorganization that affects the human being's efficiency in work and daily activities (e.g., driving, exercising). Encephalography is routinely used to discern this fatigue. Several automatic procedures have deployed conventional approaches to support neurologists in mental fatigue detection episodes (e.g., sleepy vs. normal). In all of the traditional procedures (e.g., support vector machine, discrimination fisher, K-nearest neighbor, and Bayesian classification), only a low accuracy is achieved when a binary classification task (e.g., tired vs. normal) is applied. The convolutional neural network model identifies the correct mathematical manipulation to turn the input into the output. In this study, a convolutional neural network is trained to recognize brain signals recorded by a wearable encephalographic cap. Unfortunately, the convolutional neural network works with large datasets. To overcome this problem, an augmentation scheme for a convolutional neural network model is essential because it can achieve higher accuracy than the traditional classifiers. The results show that our model achieved 97.3% compared to the state-of-the-art traditional methods (e.g., SVM and LDA).

Classifying Driving Fatigue by Using EEG Signals

Fatigue driving is one of the main reasons for the occurrence of traffic accidents. Brain-computer interface, as a human-computer interaction method based on EEG signals, can communicate with the outside world and move freely through brain signals without relying on the peripheral neuromuscular system. In this paper, a simulation driving platform composed of driving simulation equipment and driving simulation software is used to simulate the real driving process. The EEG signals of the subjects are collected through simulated driving, and the EEG of five subjects is selected as the training sample, and the remaining one is the subject. As a test sample, perform feature extraction and classification experiments, select any set of normal signals and fatigue signals recorded in the driving fatigue experiment for data analysis, and then study the classification of driver fatigue levels. Experiments have proved that the PSO-H-ELM algorithm has only about 4% advantage compared with the average accuracy of the KNN algorithm and the SVM algorithm. The gap is not as big as expected, but as a new algorithm, it is applied to the detection of fatigue EEG. The two traditional algorithms are indeed more suitable. It shows that the driver fatigue level can be judged by detecting EEG, which will provide a basis for the development of on-board, real-time driving fatigue alarm devices. It will lay the foundation for traffic management departments to intervene in driving fatigue reasonably and provide a reliable basis for minimizing traffic accidents.

Driver drowsiness estimation using EEG signals with a dynamical encoder-decoder modeling framework

Drowsiness is a leading cause of accidents on the road as it negatively affects the driver’s ability to safely operate a vehicle. Neural activity recorded by EEG electrodes is a widely used physiological correlate of driver drowsiness. This paper presents a novel dynamical modeling solution to estimate the instantaneous level of the driver drowsiness using EEG signals, where the PERcentage of eyelid CLOSure (PERCLOS) is employed as the ground truth of driver drowsiness. Applying our proposed modeling framework, we find neural features present in EEG data that encode PERCLOS. In the decoding phase, we use a Bayesian filtering solution to estimate the PERCLOS level over time. A data set that comprises 18 driving tests, conducted by 13 drivers, has been used to investigate the performance of the proposed framework. The modeling performance in estimation of PERCLOS provides robust and repeatable results in tests with manual and automated driving modes by an average RMSE of 0.117 (at a PERCLOS range of 0 to 1) and average High Probability Density percentage of 62.5%. We further hypothesized that there are biomarkers that encode the PERCLOS across different driving tests and participants. Using this solution, we identified possible biomarkers such as Theta and Delta powers. Results show that about 73% and 66% of the Theta and Delta powers which are selected as biomarkers are increasing as PERCLOS grows during the driving test. We argue that the proposed method is a robust and reliable solution to estimate drowsiness in real-time which opens the door in utilizing EEG-based measures in driver drowsiness detection systems.

A survey of brain network analysis by electroencephalographic signals

Brain network analysis is one efficient tool in exploring human brain diseases and can differentiate the alterations from comparative networks. The alterations account for time, mental states, tasks, individuals, and so forth. Furthermore, the changes determine the segregation and integration of functional networks that lead to network reorganization (or reconfiguration) to extend the neuroplasticity of the brain. Exploring related brain networks should be of interest that may provide roadmaps for brain research and clinical diagnosis. Recent electroencephalogram (EEG) studies have revealed the secrets of the brain networks and diseases (or disorders) within and between subjects and have provided instructive and promising suggestions and methods. This review summarized the corresponding algorithms that had been used to construct functional or effective networks on the scalp and cerebral cortex. We reviewed EEG network analysis that unveils more cognitive functions and neural disorders of the human and then explored the relationship between brain science and artificial intelligence which may fuel each other to accelerate their advances, and also discussed some innovations and future challenges in the end.

Neurological state changes indicative of ADHD in children learned via EEG-based LSTM networks

Attention-deficit/hyperactivity disorder (ADHD) is a neurodevelopmental disorder that pervasively interferes with the lives of individuals starting in childhood.

OBJECTIVE

To address the subjectivity of current diagnostic approaches, many studies have been dedicated to efforts to identify the differences between ADHD and neurotypical (NT) individuals using EEG and continuous performance tests (CPT).

APPROACH

In this study, we proposed EEG-based long short-term memory (LSTM) networks that utilize deep learning techniques with learning the cognitive state transition to discriminate between ADHD and NT children via EEG signal processing. A total of thirty neurotypical children and thirty ADHD children participated in CPT tests while being monitored with EEG. Several architectures of deep and machine learning were applied to three EEG data segments including resting state, cognitive execution, and a period containing a fusion of those.

MAIN RESULTS

The experimental results indicated that EEG-based LSTM networks produced the best performance with an average accuracy of 90.50 ± 0.81 % in comparison with the deep neural networks, the convolutional neural networks, and the support vector machines with learning the cognitive state transition of EEG data. Novel observations of individual neural markers showed that the beta power activity of the O1 and O2 sites contributed the most to the classifications, subjects exhibited decreased beta power in the ADHD group, and had larger decreases during cognitive execution.

SIGNIFICANCE

These findings showed that the proposed EEG-based LSTM networks are capable of extracting the varied temporal characteristics of high-resolution electrophysiological signals to differentiate between ADHD and NT children, and brought a new insight to facilitate the diagnosis of ADHD. The registration numbers of the institutional review boards are 16MMHIS021 and EC1070401-F.

EEG-FCV: An EEG-Based Functional Connectivity Visualization Framework for Cognitive State Evaluation

Electroencephalogram (EEG)-based tools for brain functional connectivity (FC) analysis and visualization play an important role in evaluating brain cognitive function. However, existing similar FC analysis tools are not only visualized in 2 dimensions (2D) but also are highly prone to cause visual clutter and unable to dynamically reflect brain connectivity changes over time. Therefore, we design and implement an EEG-based FC visualization framework in this study, named EEG-FCV, for brain cognitive state evaluation. EEG-FCV is composed of three parts: the Data Processing module, Connectivity Analysis module, and Visualization module. Specially, FC is visualized in 3 dimensions (3D) by introducing three existing metrics: Pearson Correlation Coefficient (PCC), Coherence, and PLV. Furthermore, a novel metric named Comprehensive is proposed to solve the problem of visual clutter. EEG-FCV can also visualize dynamically brain FC changes over time. Experimental results on two available datasets show that EEG-FCV has not only results consistent with existing related studies on brain FC but also can reflect dynamically brain FC changes over time. We believe EEG-FCV could prompt further progress in brain cognitive function evaluation.

Multi-Parameter Physiological State Monitoring in Target Detection Under Real-World Settings

Mental state changes induced by stimuli under experimental settings or by daily events in real life affect task performance and are entwined with physical and mental health. In this study, we developed a physiological state indicator with five parameters that reflect the subject's real-time physiological states based on online EEG signal processing. These five parameters are attention, fatigue, stress, and the brain activity shifts of the left and right hemispheres. We designed a target detection experiment modified by a cognitive attention network test for validating the effectiveness of the proposed indicator, as such conditions would better approximate a real chaotic environment. Results demonstrated that attention levels while performing the target detection task were significantly higher than during rest periods, but also exhibited a decay over time. In contrast, the fatigue level increased gradually and plateaued by the third rest period. Similar to attention levels, the stress level decreased as the experiment proceeded. These parameters are therefore shown to be highly correlated to different stages of the experiment, suggesting their usage as primary factors in passive brain-computer interfaces (BCI). In addition, the left and right brain activity indexes reveal the EEG neural modulations of the corresponding hemispheres, which set a feasible reference of activation for an active BCI control system, such as one executing motor imagery tasks. The proposed indicator is applicable to potential passive and active BCI applications for monitoring the subject's physiological state change in real-time, along with providing a means of evaluating the associated signal quality to enhance the BCI performance.

Asymmetry of Regional Phase Synchrony Cortical Networks Under Cognitive Alertness and Vigilance Decrement States

This study investigates intra-regional connectivity and regional hemispheric asymmetry under two vigilance states: alertness and vigilance decrement. The vigilance states were induced on nine healthy subjects while performing 30 min in-congruent Stroop color-word task (I-SCWT). We measured brain activity using Electroencephalography (EEG) signals with 64-channels. We quantified the regional network connectivity using the phase-locking value (PLV) with graph theory analysis (GTA) and Support Vector Machines (SVM). Results showed that the vigilance decrement state was associated with impaired information processing within the frontal and central regions in delta and theta frequency bands. Meanwhile, the hemispheric asymmetry results showed that the laterality shifted to the right-temporal in delta, right-central, parietal, and left frontal in theta, right-frontal and left-central, temporal and parietal in alpha, and right-parietal and left temporal in beta frequency bands. These findings represent the first demonstration of intra-regional connectivity and hemispheric asymmetry changes as a function of cognitive vigilance states. The overall results showed that vigilance decrement is region and frequency band-specific. Our SVM model achieved the highest classification accuracy of 99.73% in differentiating between the two vigilance states based on the frontal and central connectivity networks measures.

Evaluation of Mental Load of Drivers in Long Highway Tunnel Based on Electroencephalograph

In recent years, the mileage of the tunnels has substantially increased with the rapid highway construction that led to increasing highway tunnels. Most studies on tunnel accidents have mainly focused on the external environments, such as tunnel structure, traffic volume, and lighting. In addition, although many studies on mental load of drivers have been conducted for public roads, such studies for highway tunnels have been limited. In this study, three scenarios with different front vehicle speeds (60, 45, and 30 km/h) in a two-lane long tunnel (one lane in each travel direction) were evaluated using a driving simulator. The experiment involved 24 participants (14 men and 10 women) with an average age of 25.8 years and an average experience of 3.2 years. The electroencephalogram (EEG) technology was used to collect the leading EEG indicators during the driving simulation of the scenarios: α, β, and θ waves and the wave ratio, (α + θ)/β. According to the β-wave energy measurements, the alertness of drivers was the lowest at 45 km/h after adapting to the tunnel environment, indicating that the drivers were more comfortable at this speed. This preliminary finding should help in determining the speed limit in this type of tunnel.

EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications

Brain-Computer interfaces (BCIs) enhance the capability of human brain activities to interact with the environment. Recent advancements in technology and machine learning algorithms have increased interest in electroencephalographic (EEG)-based BCI applications. EEG-based intelligent BCI systems can facilitate continuous monitoring of fluctuations in human cognitive states under monotonous tasks, which is both beneficial for people in need of healthcare support and general researchers in different domain areas. In this review, we survey the recent literature on EEG signal sensing technologies and computational intelligence approaches in BCI applications, compensating for the gaps in the systematic summary of the past five years. Specifically, we first review the current status of BCI and signal sensing technologies for collecting reliable EEG signals. Then, we demonstrate state-of-the-art computational intelligence techniques, including fuzzy models and transfer learning in machine learning and deep learning algorithms, to detect, monitor, and maintain human cognitive states and task performance in prevalent applications. Finally, we present a couple of innovative BCI-inspired healthcare applications and discuss future research directions in EEG-based BCI research.

A Study on Seizure Detection of EEG Signals Represented in 2D

A seizure is a neurological disorder caused by abnormal neuronal discharges in the brain, which severely reduces the quality of life of patients and often endangers their lives. Automatic seizure detection is an important research area in the treatment of seizure and is a prerequisite for seizure intervention. Deep learning has been widely used for automatic detection of seizures, and many related research works decomposed the electroencephalogram (EEG) raw signal with a time window to obtain EEG signal slices, then performed feature extraction on the slices, and represented the obtained features as input data for neural networks. There are various methods for EEG signal decomposition, feature extraction, and representation, and most of the studies have been based on fixed hardware resources for the design of the scheme, which reduces the adaptability of the scheme in different application scenarios and makes it difficult to optimize the algorithms in the scheme. To address the above issues, this paper proposes a deep learning-based model for seizure detection, mainly characterized by the two-dimensional representation of EEG features and the scalability of neural networks. The model modularizes the main steps of seizure detection and improves the adaptability of the model to different hardware resource constraints, in order to increase the convenience of the algorithm optimization or the replacement of each module. The proposed model consists of five parts, and the model was tested using two epilepsy datasets separately. The experimental results showed that the proposed model has strong generality and good classification accuracy for seizure detection.

A High Accuracy Electrographic Seizure Classifier Trained Using Semi-Supervised Labeling Applied to a Large Spectrogram Dataset

The objective of this study was to explore using ECoG spectrogram images for training reliable cross-patient electrographic seizure classifiers, and to characterize the classifiers’ test accuracy as a function of amount of training data. ECoG channels in ∼138,000 time-series ECoG records from 113 patients were converted to RGB spectrogram images. Using an unsupervised spectrogram image clustering technique, manual labeling of 138,000 ECoG records (each with up to 4 ECoG channels) was completed in 320 h, which is an estimated 5 times faster than manual labeling without ECoG clustering. For training supervised classifier models, five random folds of data were created; with each fold containing 72, 18, and 23 patients’ data for model training, validation and testing respectively. Five convolutional neural network (CNN) architectures, including two with residual connections, were trained. Cross-patient classification accuracies and F₁ scores improved with model complexity, with the shallowest 6-layer model (with ∼1.5 million trainable parameters) producing a class-balanced seizure/non-seizure classification accuracy of 87.9% on ECoG channels and the deepest ResNet50-based model (with ∼23.5 million trainable parameters) producing a classification accuracy of 95.7%. The trained ResNet50-based model additionally had 93.5% agreement in scores with an independent expert labeller. Visual inspection of gradient-based saliency maps confirmed that the models’ classifications were based on relevant portions of the spectrogram images. Further, by repeating training experiments with data from varying number of patients, it was found that ECoG spectrogram images from just 10 patients were sufficient to train ResNet50-based models with 88% cross-patient accuracy, while at least 30 patients’ data was required to produce cross-patient classification accuracies of >90%.

Imaging Time Series of Eye Tracking Data to Classify Attentional States

It has been shown that conclusions about the human mental state can be drawn from eye gaze behavior by several previous studies. For this reason, eye tracking recordings are suitable as input data for attentional state classifiers. In current state-of-the-art studies, the extracted eye tracking feature set usually consists of descriptive statistics about specific eye movement characteristics (i.e., fixations, saccades, blinks, vergence, and pupil dilation). We suggest an Imaging Time Series approach for eye tracking data followed by classification using a convolutional neural net to improve the classification accuracy. We compared multiple algorithms that used the one-dimensional statistical summary feature set as input with two different implementations of the newly suggested method for three different data sets that target different aspects of attention. The results show that our two-dimensional image features with the convolutional neural net outperform the classical classifiers for most analyses, especially regarding generalization over participants and tasks. We conclude that current attentional state classifiers that are based on eye tracking can be optimized by adjusting the feature set while requiring less feature engineering and our future work will focus on a more detailed and suited investigation of this approach for other scenarios and data sets.

Cross-Subject Zero Calibration Driver's Drowsiness Detection: Exploring Spatiotemporal Image Encoding of EEG Signals for Convolutional Neural Network Classification

This paper explores two methodologies for drowsiness detection using EEG signals in a sustained-attention driving task considering pre-event time windows, and focusing on cross-subject zero calibration. Driving accidents are a major cause of injuries and deaths on the road. A considerable portion of those are due to fatigue and drowsiness. Advanced driver assistance systems that could detect mental states which are associated with hazardous situations, such as drowsiness, are of critical importance. EEG signals are used widely for brain-computer interfaces, as well as mental state recognition. However, these systems are still difficult to design due to very low signal-to-noise ratios and cross-subject disparities, requiring individual calibration cycles. To tackle this research domain, here, we explore drowsiness detection based on EEG signals' spatiotemporal image encoding representations in the form of either recurrence plots or gramian angular fields for deep convolutional neural network (CNN) classification. Results comparing both techniques using a public dataset of 27 subjects show a superior balanced accuracy of up to 75.87% for leave-one-out cross-validation, using both techniques, against works in the literature, demonstrating the possibility to pursue cross-subject zero calibration design.

Vehicle driver drowsiness detection method using wearable EEG based on convolution neural network

Vehicle drivers driving cars under the situation of drowsiness can cause serious traffic accidents. In this paper, a vehicle driver drowsiness detection method using wearable electroencephalographic (EEG) based on convolution neural network (CNN) is proposed. The presented method consists of three parts: data collection using wearable EEG, vehicle driver drowsiness detection and the early warning strategy. Firstly, a wearable brain computer interface (BCI) is used to monitor and collect the EEG signals in the simulation environment of drowsy driving and awake driving. Secondly, the neural networks with Inception module and modified AlexNet module are trained to classify the EEG signals. Finally, the early warning strategy module will function and it will sound an alarm if the vehicle driver is judged as drowsy. The method was tested on driving EEG data from simulated drowsy driving. The results show that using neural network with Inception module reached 95.59% classification accuracy based on one second time window samples and using modified AlexNet module reached 94.68%. The simulation and test results demonstrate the feasibility of the proposed drowsiness detection method for vehicle driving safety.

Detection of Negative Stress through Spectral Features of Electroencephalographic Recordings and a Convolutional Neural Network

In recent years, electroencephalographic (EEG) signals have been intensively used in the area of emotion recognition, partcularly in distress identification due to its negative impact on physical and mental health. Traditionally, brain activity has been studied from a frequency perspective by computing the power spectral density of the EEG recordings and extracting features from different frequency sub-bands. However, these features are often individually extracted from single EEG channels, such that each brain region is separately evaluated, even when it has been corroborated that mental processes are based on the coordination of different brain areas working simultaneously. To take advantage of the brain’s behaviour as a synchronized network, in the present work, 2-D and 3-D spectral images constructed from common 32 channel EEG signals are evaluated for the first time to discern between emotional states of calm and distress using a well-known deep-learning algorithm, such as AlexNet. The obtained results revealed a significant improvement in the classification performance regarding previous works, reaching an accuracy about 84%. Moreover, no significant differences between the results provided by the diverse approaches considered to reconstruct 2-D and 3-D spectral maps from the original location of the EEG channels over the scalp were noticed, thus suggesting that these kinds of images preserve original spatial brain information.

A dynamic center and multi threshold point based stable feature extraction network for driver fatigue detection utilizing EEG signals

Driver fatigue is the one of the main reasons of the traffic accidents. The human brain is a complex structure, whose function can be evaluated with electroencephalogram (EEG). Automated driver fatigue detection utilizing EEG decreases the incidence probability of related traffic accidents. Therefore, devising an appropriate feature extraction technique and selecting a competent classification method can be considered as the crucial part of the effective driver fatigue detection. Therefore, in this study, an EEG-based intelligent system was devised for driver fatigue detection. The proposed framework includes a new feature generation network, which is implemented by using texture descriptors, for fatigue detection. The proposed scheme contains pre-processing, feature generation, informative features selection and classification with shallow classifiers phases. In the pre-processing, discrete cosine transform and fast Fourier transform are used together. Moreover, dynamic center based binary pattern and multi threshold ternary pattern are utilized together to create a new feature generation network. To improve the detection performance, we utilized discrete wavelet transform as a pooling method, in which the functional brain network-based feature describing the relationship between fatigue and brain network organization. In the feature selection phase, a hybrid three layered feature selection method is presented, and benchmark classifiers are used in the classification phase to demonstrate the strength of the proposed method. In the experiments, the proposed framework achieved 97.29% classification accuracy for fatigue detection using EEG signals. This result reveals that the proposed framework can be utilized effectively for driver fatigue detection.