A dynamic center and multi threshold point based stable feature extraction network for driver fatigue detection utilizing EEG signals

EEG-based fatigue state evaluation by combining complex network and frequency-spatial features

BACKGROUND

The proportion of traffic accidents caused by fatigue driving is increasing year by year, which has aroused wide concerns for researchers. In order to rapidly and accurately detect drivers' fatigue, this paper proposed an electroencephalogram (EEG)-based fatigue state evaluation method by combining complex network and frequency-spatial features.

NEW METHOD

First, this paper constructed a complex network model based on the relative wavelet entropy to characterize the correlation strength information between channels. Then, the differential entropy and symmetry quotient were respectively calculated to extract frequency and spatial features. Then, the brain heat map combined the complex network and frequency-spatial features with different dimensions together as the fusion features. Finally, a convolutional neural network-long short-term memory (CNN-LSTM) neural network was used to evaluate the three-class fatigue states of the EEG data in the Shanghai Jiao Tong University (SJTU) Emotion EEG Dataset (SEED)-VIG dataset, and it was validated on the dataset on the Mendeley Data website.

RESULTS

The experimental results of SEED-VIG dataset show that the average classification accuracy of three-class fatigue states, namely, awake, tired and drowsy, reaches 96.57 %. The average classification accuracy on the dataset on the Mendeley Data website reaches 99.23 %.

COMPARISON WITH EXISTING METHODS

This method has a best evaluation performance compared with the state-of-the-art methods for the three-class fatigue states recognition.

CONCLUSIONS

The experiment results validated the feasibility of the fatigue state evaluation method based on the correlations between channels and the frequency-spatial features, which is of great significance for developing a driver fatigue detection system.

An improved CapsNet based on data augmentation for driver vigilance estimation with forehead single-channel EEG

Various studies have shown that it is necessary to estimate the drivers' vigilance to reduce the occurrence of traffic accidents. Most existing EEG-based vigilance estimation studies have been performed on intra-subject and multi-channel signals, and these methods are too costly and complicated to implement in practice. Hence, aiming at the problem of cross-subject vigilance estimation of single-channel EEG signals, an estimation algorithm based on capsule network (CapsNet) is proposed. Firstly, we propose a new construction method of the input feature maps to fit the characteristics of CapsNet to improve the algorithm accuracy. Meanwhile, the self-attention mechanism is incorporated in the algorithm to focus on the key information in feature maps. Secondly, we propose substituting the traditional multi-channel signals with the single-channel signals to improve the utility of algorithm. Thirdly, since the single-channel signals carry fewer dimensions of the information compared to the multi-channel signals, we use the conditional generative adversarial network to improve the accuracy of single-channel signals by increasing the amount of data. The proposed algorithm is verified on the SEED-VIG, and Root-mean-square-error (RMSE) and Pearson Correlation Coefficient (PCC) are used as the evaluation metrics. The results show that the proposed algorithm improves the computing speed while the RMSE is reduced by 3%, and the PCC is improved by 12% compared to the mainstream algorithm. Experiment results prove the feasibility of using forehead single-channel EEG signals for cross-subject vigilance estimation and offering the possibility of lightweight EEG vigilance estimation devices for practical applications.

PCG-based exercise fatigue detection method using multi-scale feature fusion model

Accurate detection of exercise fatigue based on physiological signals is vital for reasonable physical activity. Existing studies utilize widely Electrocardiogram (ECG) signals to achieve exercise monitoring. Nevertheless, ECG signals may be corrupted because of sweat or loose connection. As a non-invasive technique, Phonocardiogram (PCG) signals have a strong ability to reflect the Cardiovascular information, which is closely related to physical state. Therefore, a novel PCG-based detection method is proposed, where the feature fusion of deep learning features and linear features is the key technology of improving fatigue detection performance. Specifically, Short-Time Fourier Transform (STFT) is employed to convert 1D PCG signals into 2D images, and images are fed into the pre-trained convolutional neural network (VGG-16) for learning. Then, the fusion features are constructed by concatenating the VGG-16 output features and PCG linear features. Finally, the concatenated features are sent to Support Vector Machines (SVM) and Linear Discriminant Analysis (LDA) to distinguish six levels of exercise fatigue. The experimental results of two datasets show that the best performance of the proposed method achieves 91.47% and 99.00% accuracy, 91.49% and 99.09% F1-score, 90.99% and 99.07% sensitivity, which has comparable performance to an ECG-based system which is as gold standard (94.32% accuracy, 94.33% F1-score, 94.52% sensitivity).

Driver drowsiness detection methods using EEG signals: a systematic review

Electroencephalography (EEG) is a complex signal that may require several years of training, advanced signal processing, and feature extraction methodologies to interpret correctly. Recently, many methods have been used to extract and classify EEG data. This study reviews 62 papers that used EEG signals to detect driver drowsiness, published between January 2018 and 2022. We extract trends and highlight interesting approaches from this large body of literature to inform future research and formulate recommendations. To find relevant papers published in scientific journals, conferences, and electronic preprint repositories, researchers searched major databases covering the domains of science and engineering. For each investigation, many data items about (1) the data, (2) the channels used, (3) the extraction and classification procedure, and (4) the outcomes were extracted. These items were then analyzed one by one to uncover trends. Our analysis reveals that the amount of EEG data used across studies varies. We saw that more than half the studies used simulation driving experimental. About 21% of the studies used support vector machine (SVM), while 19% used convolutional neural networks (CNN). Overall, we can conclude that drowsiness and fatigue impair driving performance, resulting in drivers who are more exposed to risky situations.

A regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based cross-subject fatigue detection

Fatigue, one of the most important factors affecting road safety, has attracted many researchers' attention. Most existing fatigue detection methods are based on feature engineering and classification models. The feature engineering is greatly influenced by researchers' domain knowledge, which will lead to a poor performance in fatigue detection, especially in cross-subject experiment design. In addition, fatigue detection is often simplified as a classification problem of several discrete states. Models based on deep learning can realize automatic feature extraction without the limitation of researcher's domain knowledge. Therefore, this paper proposes a regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based (EEG-based) cross-subject fatigue detection. At the same time, a twofold random-offset zero-overlapping sampling method is proposed to train a bigger model and reduce overfitting. Compared with existing results, the proposed method achieves a much better result of 0.94 correlation coefficient (COR) and 0.09 root mean square error (RMSE) in a within-subject experiment design. What is more, there is no misclassification between awake and drowsy states. For cross-subject experiment design, the COR and RMSE are 0.79 and 0.15, respectively, which are close to the existing within-subject results and better than similar cross-subject results. The cross-subject regression model is very important for fatigue detection application since the fatigue indication is more precise than several discrete states and no model calibration is required for a new user. The twofold random-offset zero-overlapping sampling method can also be used as a reference by other EEG-based deep learning research.

A Multimodal Feature Fusion Framework for Sleep-Deprived Fatigue Detection to Prevent Accidents

Sleep-deprived fatigued person is likely to commit more errors that may even prove to be fatal. Thus, it is necessary to recognize this fatigue. The novelty of the proposed research work for the detection of this fatigue is that it is nonintrusive and based on multimodal feature fusion. In the proposed methodology, fatigue is detected by obtaining features from four domains: visual images, thermal images, keystroke dynamics, and voice features. In the proposed methodology, the samples of a volunteer (subject) are obtained from all four domains for feature extraction, and empirical weights are assigned to the four different domains. Young, healthy volunteers (n = 60) between the age group of 20 to 30 years participated in the experimental study. Further, they abstained from the consumption of alcohol, caffeine, or other drugs impacting their sleep pattern during the study. Through this multimodal technique, appropriate weights are given to the features obtained from the four domains. The results are compared with k-nearest neighbors (kNN), support vector machines (SVM), random tree, random forest, and multilayer perceptron classifiers. The proposed nonintrusive technique has obtained an average detection accuracy of 93.33% in 3-fold cross-validation.

An improved multi-source domain adaptation network for inter-subject mental fatigue detection based on DANN

OBJECTIVES

Electroencephalogram (EEG) is often used to detect mental fatigue because of its real-time characteristic and objective nature. However, because of the individual variability of EEG among different individuals, tedious and time-consuming calibration sessions are needed.

METHODS

Therefore, we propose a multi-source domain adaptation network for inter-subject mental fatigue detection named FLDANN, which is short for focal loss based domain-adversarial training of neural network. As for mental state feature extraction, power spectrum density is extracted based on the Welch method from four sub-bands of EEG signals. The features of the source domain and target domain are fed into the FLDANN network. The contributions of FLDANN include: (1) It uses the idea of adversarial to reduce feature differences between the source and target domain. (2) A loss function named focal loss is used to assign weights to source and target domain samples.

RESULTS

The experiment result shows that when the number of the source domains increases, the classification accuracy of domain-adversarial training of neural network (DANN) gradually decreases and finally tends to be stable. The proposed method achieves an accuracy of 84.10% ± 8.75% on the SEED-VIG dataset and 65.42% ± 7.47% on the self-designed dataset. In addition, the proposed method is compared with other domain adaptation methods and the results show that the proposed method outperforms those state-of-the-art methods.

CONCLUSIONS

The result proves that the proposed method is able to solve the problem of individual differences across subjects and to solve the problem of low classification performance of multi-source domain transfer learning.

Visual preference of plant features in different living environments using eye tracking and EEG

Plants play a very important role in landscape construction. In order to explore whether different living environment will affect people's preference for the structural features of plant organs, this study examined 26 villagers and 33 college students as the participants, and pictures of leaves, flowers and fruits of plants as the stimulus to conduct eye-tracking and EEG detection experiments. We found that eye movement indicators can explain people's visual preferences, but they are unable to find differences in preferences between groups. EEG indicators can make up for this deficiency, which further reveals the difference in psychological and physiological responses between the two groups when viewing stimuli. The final results show that the villagers and the students liked leaves best, preferring aciculiform and leathery leaves; solitary, purple and capitulum flowers; and medium-sized, spathulate, black and pear fruits. In addition, it was found that the overall attention of the villagers when watching stimuli was far lower than that of the students, but the degree of meditation was higher. With regard to eye movement and EEG, the total duration of fixations is highly positively correlated with the number of fixations, and the average pupil size has a weak negative correlation with attention. On the contrary, the average duration of fixations has a weak positive correlation with meditation. Generally speaking, we believe that Photinia×fraseri, Metasequoia glyptostroboides, Photinia serratifolia, Koelreuteria bipinnata and Cunninghamia lanceolata are superior landscape building plants in rural areas and on campuses; Pinus thunbergii, Myrica rubra, Camellia japonica and other plants with obvious features and bright colours are also the first choice in rural landscapes; and Yulania biondii, Cercis chinensis, Hibiscus mutabilis and other plants with simple structures are the first choice in campus landscapes. This study is of great significance for selecting plants for landscape construction and management according to different environments and local conditions.

Extraction of discriminative features from EEG signals of dyslexic children; before and after the treatment

Dyslexia is a neurological disorder manifested as difficulty reading and writing. It can occur despite adequate instruction, intelligence, and intact sensory abilities. Different electroencephalogram (EEG) patterns have been demonstrated between dyslexic and healthy subjects in previous studies. This study focuses on the difference between patients before and after treatment. The main goal is to identify the subset of features that adequately discriminate subjects before and after a specific treatment plan. The treatment consists of Transcranial Direct Current Stimulation (tDCS) and occupational therapy using the BrainWare SAFARI software. The EEG signals of sixteen dyslexic children were recorded during the eyes-closed resting state before and after treatment. The preprocessing step was followed by the extraction of a wide range of features to investigate the differences related to the treatment. An optimal subset of features extracted from recorded EEG signals was determined using Principal Component Analysis (PCA) in conjunction with the Sequential Floating Forward Selection (SFFS) algorithm. The results showed that treatment leads to significant changes in EEG features like spectral and phase-related EEG features, in various regions. It has been demonstrated that the extracted subset of discriminative features can be useful for classification applications in treatment assessment. The most discriminative subset of features could classify the data with an accuracy of 92% with SVM classifier. The above result confirms the efficacy of the treatment plans in improving dyslexic children's cognitive skills.

Driving Fatigue Detection with Three Non-Hair-Bearing EEG Channels and Modified Transformer Model

Driving fatigue is the main cause of traffic accidents, which seriously affects people's life and property safety. Many researchers have applied electroencephalogram (EEG) signals for driving fatigue detection to reduce negative effects. The main challenges are the practicality and accuracy of the EEG-based driving fatigue detection method when it is applied on the real road. In our previous study, we attempted to improve the practicality of fatigue detection based on the proposed non-hair-bearing (NHB) montage with fewer EEG channels, but the recognition accuracy was only 76.47% with the random forest (RF) model. In order to improve the accuracy with NHB montage, this study proposed an improved transformer architecture for one-dimensional feature vector classification based on introducing the Gated Linear Unit (GLU) in the Attention sub-block and Feed-Forward Networks (FFN) sub-block of a transformer, called GLU-Oneformer. Moreover, we constructed an NHB-EEG-based feature set, including the same EEG features (power ratio, approximate entropy, and mutual information (MI)) in our previous study, and the lateralization features of the power ratio and approximate entropy based on the strategy of brain lateralization. The results indicated that our GLU-Oneformer method significantly improved the recognition performance and achieved an accuracy of 86.97%. Our framework demonstrated that the combination of the NHB montage and the proposed GLU-Oneformer model could well support driving fatigue detection.

1D Multi-Point Local Ternary Pattern: A Novel Feature Extraction Method for Analyzing Cognitive Engagement of students in Flipped Learning Pedagogy

Flipped learning is a blended learning method based on academic engagement of students online (outside class) and offline (inside class). In this learning pedagogy, students receive lesson any time from lecture videos pre-loaded on digital platform at their convenience places and it is followed by in-classroom activities such as doubt clearing, problem solving, etc. However, students are constantly exposed to high levels of distraction in this age of the Internet. Therefore, it is hard for an instructor to know whether a student has paid attention while watching pre-loaded lecture video. In order to analyze attention level of individual students, captured brain signal or electroencephalogram (EEG) of students can be utilized. In this study, we utilize a popular feature extraction technique called Local Binary Pattern (LBP) and improvise it to develop an enhanced feature selection method. The adapted feature selection method termed as 1D Multi-Point Local Ternary Pattern (1D MP-LTP) is used to extract unique features from collected electroencephalogram (EEG) signals. Standard classification techniques are exploited to classify the attention level of students. Experiments are conducted with the data captured at Intelligent Data Analysis Lab, NIT Rourkela, to show effectiveness of the proposed feature extraction technique. The proposed 1D Multi-Point Local Ternary Pattern (1D MP-LTP)-based classification techniques outperform traditional and state-of-the-art classification techniques using LBP. This research can be helpful for instructors to identify students who need special care for improving their learning ability. Researchers in educational technology can extend this work by adopting this methodology in other online teaching pedagogy such as Massive Open Online Courses (MOOC).

Research on a Real-Time Driver Fatigue Detection Algorithm Based on Facial Video Sequences

The research on driver fatigue detection is of great significance to improve driving safety. This paper proposes a real-time comprehensive driver fatigue detection algorithm based on facial landmarks to improve the detection accuracy, which detects the driver’s fatigue status by using facial video sequences without equipping their bodies with other intelligent devices. A tasks-constrained deep convolutional network is constructed to detect the face region based on 68 key points, which can solve the optimization problem caused by the different convergence speeds of each task. According to the real-time facial video images, the eye feature of the eye aspect ratio (EAR), mouth aspect ratio (MAR) and percentage of eye closure time (PERCLOS) are calculated based on facial landmarks. A comprehensive driver fatigue assessment model is established to assess the fatigue status of drivers through eye/mouth feature selection. After a series of comparative experiments, the results show that this proposed algorithm achieves good performance in both accuracy and speed for driver fatigue detection.

A Hybrid Driver Fatigue and Distraction Detection Model Using AlexNet Based on Facial Features

Modern cities have imposed a fast-paced lifestyle where more drivers on the road suffer from fatigue and sleep deprivation. Consequently, road accidents have increased, becoming one of the leading causes of injuries and death among young adults and children. These accidents can be prevented if fatigue symptoms are diagnosed and detected sufficiently early. For this reason, we propose and compare two AlexNet CNN-based models to detect drivers’ fatigue behaviors, relying on head position and mouth movements as behavioral measures. We used two different approaches. The first approach is transfer learning, specifically, fine-tuning AlexNet, which allowed us to take advantage of what the model had already learned without developing it from scratch. The newly trained model was able to predict drivers’ drowsiness behaviors. The second approach is the use of AlexNet to extract features by training the top layers of the network. These features were reduced using non-negative matrix factorization (NMF) and classified with a support vector machine (SVM) classifier. The experiments showed that our proposed transfer learning model achieved an accuracy of 95.7%, while the feature extraction SVM-based model performed better, with an accuracy of 99.65%. Both models were trained on a simulated NTHU Driver Drowsiness Detection dataset.