EEG-Based Spatio-Temporal Convolutional Neural Network for Driver Fatigue Evaluation

Amplitude-Time Dual-View Fused EEG Temporal Feature Learning for Automatic Sleep Staging

Electroencephalogram (EEG) plays an important role in studying brain function and human cognitive performance, and the recognition of EEG signals is vital to develop an automatic sleep staging system. However, due to the complex nonstationary characteristics and the individual difference between subjects, how to obtain the effective signal features of the EEG for practical application is still a challenging task. In this article, we investigate the EEG feature learning problem and propose a novel temporal feature learning method based on amplitude-time dual-view fusion for automatic sleep staging. First, we explore the feature extraction ability of convolutional neural networks for the EEG signal from the perspective of interpretability and construct two new representation signals for the raw EEG from the views of amplitude and time. Then, we extract the amplitude-time signal features that reflect the transformation between different sleep stages from the obtained representation signals by using conventional 1-D CNNs. Furthermore, a hybrid dilation convolution module is used to learn the long-term temporal dependency features of EEG signals, which can overcome the shortcoming that the small-scale convolution kernel can only learn the local signal variation information. Finally, we conduct attention-based feature fusion for the learned dual-view signal features to further improve sleep staging performance. To evaluate the performance of the proposed method, we test 30-s-epoch EEG signal samples for healthy subjects and subjects with mild sleep disorders. The experimental results from the most commonly used datasets show that the proposed method has better sleep staging performance and has the potential for the development and application of an EEG-based automatic sleep staging system.

A generative adaptive convolutional neural network with attention mechanism for driver fatigue detection with class-imbalanced and insufficient data

Over the past few years, fatigue driving has emerged as one of the main causes of traffic accidents, necessitating the development of driver fatigue detection systems. However, many existing methods involves tedious manual parameter tunings, a process that is both time-consuming and results in task-specific models. On the other hand, most of the researches on fatigue recognition are based on class-balanced and sufficient data, and effectively "mine" meaningful information from class-imbalanced and insufficient data for fatigue recognition is still a challenge. In this paper, we proposed two novel models, the attention-based residual adaptive multiscale fully convolutional network-long short term memory network (ARMFCN-LSTM), and the Generative ARMFCN-LSTM (GARMFCN-LSTM) aiming to address this issue. ARMFCN-LSTM excels at automatically extracting multiscale representations through adaptive multiscale temporal convolutions, while capturing temporal dependency features through LSTM. GARMFCN-LSTM integrates Wasserstein GAN with gradient penalty (WGAN-GP) into ARMFCN-LSTM to improve driver fatigue detection performance by alleviating data scarcity and addressing class imbalances. Experimental results show that ARMFCN-LSTM achieves the highest classification accuracy of 95.84% in driver fatigue detection on the class-balanced EEG dataset (binary classification), and GARMFCN-LSTM attained an improved classification accuracy of 84.70% on the class-imbalanced EOG dataset (triple classification), surpassing the competing methods. Therefore, the proposed models are promising for further implementations in online driver fatigue detection systems.

Detection of atypical attentional behaviors in young subjects

BACKGROUND

Vigilance ability refers to the accuracy and speed with which a person performs a cognitive-motor task, either voluntarily (endogenous mode) or following a warning stimulus (exogenous mode). In the context of a force production task, our study focuses on the impact of the states of vigilance by proposing an original approach that allows distinguishing between good (inlier) and poor (outlier) participants. We assume that the use of an external signal and duration of the temporal preparation (foreperiod) increase the speed and the precision of motor responses. Our objective is particularly challenging in the context of a limited dataset with a high level of noise.

NEW METHOD

Our original methodological approach consists of coupling the RANSAC (RANdom SAmple Consensus) algorithm with a statistical machine learning algorithm to handle noise.

COMPARISON WITH EXISTING METHODS

Our clustering approach, based on the coupling of RANSAC methodology with ensemble classifiers, overcomes the limitations of conventional supervised algorithms that are either not robust to outliers (such as K-Nearest Neighbors) and/or not adapted to few-shot learning (such as Support Vector Machines and Artificial Neural Networks).

RESULTS

The clustering results were validated in terms of reaction time distributions and force error distributions with respect to participant groups. We show that the use of an external signal and duration of the temporal preparation (foreperiod) increase the speed and the precision of motor responses.

CONCLUSION

Our study has allowed us to detect atypical attentional patterns and succeeds in separating the inliers from the outliers.

Decoding emotion with phase-amplitude fusion features of EEG functional connectivity network

Decoding emotional neural representations from the electroencephalographic (EEG)-based functional connectivity network (FCN) is of great scientific importance for uncovering emotional cognition mechanisms and developing harmonious human-computer interactions. However, existing methods mainly rely on phase-based FCN measures (e.g., phase locking value [PLV]) to capture dynamic interactions between brain oscillations in emotional states, which fail to reflect the energy fluctuation of cortical oscillations over time. In this study, we initially examined the efficacy of amplitude-based functional networks (e.g., amplitude envelope correlation [AEC]) in representing emotional states. Subsequently, we proposed an efficient phase-amplitude fusion framework (PAF) to fuse PLV and AEC and used common spatial pattern (CSP) to extract fused spatial topological features from PAF for multi-class emotion recognition. We conducted extensive experiments on the DEAP and MAHNOB-HCI datasets. The results showed that: (1) AEC-derived discriminative spatial network topological features possess the ability to characterize emotional states, and the differential network patterns of AEC reflect dynamic interactions in brain regions associated with emotional cognition. (2) The proposed fusion features outperformed other state-of-the-art methods in terms of classification accuracy for both datasets. Moreover, the spatial filter learned from PAF is separable and interpretable, enabling a description of affective activation patterns from both phase and amplitude perspectives.

Deep-Learning-Assisted Sensor with Multiple Perception Capabilities for an Intelligent Driver Assistance Monitoring System

Driver assistance systems can help drivers achieve better control of their vehicles while driving and reduce driver fatigue and errors. However, the current driver assistance devices have a complex structure and severely violate the privacy of drivers, hindering the development of driver assistance technology. To address these limitations, this article proposes an intelligent driver assistance monitoring system (IDAMS), which combines a Kresling origami structure-based triboelectric sensor (KOS-TS) and a convolutional neural network (CNN)-based data analysis. For different driving behaviors, the output signals of the KOS-TSs contain various features, such as a driver's pressing force, pressing time, and sensor triggering sequence. This study develops a multiscale CNN that employs different pooling methods to process KOS-TS data and analyze temporal information. The proposed IDAMS is verified by driver identification experiments, and the results show that the accuracy of the IDAMS in discriminating eight different users is improved from 96.25% to 99.38%. In addition, the results indicate that IDAMS can successfully monitor driving behaviors and can accurately distinguish between different driving behaviors. Finally, the proposed IDAMS has excellent hands-off detection (HOD), identification, and driving behavior monitoring capabilities and shows broad potential for application in the fields of safety warning, personalization, and human-computer interaction.

A novel deep-learning model based on τ-shaped convolutional network (τNet) with long short-term memory (LSTM) for physiological fatigue detection from EEG and EOG signals

In recent years, fatigue driving has become the main cause of traffic accidents, leading to increased attention towards fatigue detection systems. However, the pooling and strided convolutional operations in fatigue detection algorithm based on traditional deep learning methods may led to the loss of some useful information. This paper proposed a novel [Formula: see text]-shaped convolutional network ([Formula: see text]) aiming to address this issue. Unlike traditional network structures, [Formula: see text] incorporates the operations of upsampling features and concatenating high- and low-level features, enabling full utilization of useful information. Moreover, considering that the fatigue state is a mental state involving temporal evolution, we proposed the novel long short-term memory (LSTM)-[Formula: see text]-shaped convolutional network (LSTM-[Formula: see text]), a parallel structure composed of LSTM and [Formula: see text] for fatigue detection, where [Formula: see text] extracts time-invariant features with location information, and LSTM extracts long temporal dependencies. We compared LSTM-[Formula: see text] with six competing methods based on two datasets. Results showed that the proposed algorithm achieved higher classification accuracy than the other methods, with 94.25% on EEG data (binary classification) and 82.19% on EOG data (triple classification). Additionally, the proposed algorithm exhibits low computational cost, good training stability, and robustness against insufficient training. Therefore, it is promising for further implementation of fatigue online detection systems.

Development of performance and learning rate evaluation models in robot-assisted surgery using electroencephalography and eye-tracking

The existing performance evaluation methods in robot-assisted surgery (RAS) are mainly subjective, costly, and affected by shortcomings such as the inconsistency of results and dependency on the raters' opinions. The aim of this study was to develop models for an objective evaluation of performance and rate of learning RAS skills while practicing surgical simulator tasks. The electroencephalogram (EEG) and eye-tracking data were recorded from 26 subjects while performing Tubes, Suture Sponge, and Dots and Needles tasks. Performance scores were generated by the simulator program. The functional brain networks were extracted using EEG data and coherence analysis. Then these networks, along with community detection analysis, facilitated the extraction of average search information and average temporal flexibility features at 21 Brodmann areas (BA) and four band frequencies. Twelve eye-tracking features were extracted and used to develop linear random intercept models for performance evaluation and multivariate linear regression models for the evaluation of the learning rate. Results showed that subject-wise standardization of features improved the R2 of the models. Average pupil diameter and rate of saccade were associated with performance in the Tubes task (multivariate analysis; p-value = 0.01 and p-value = 0.04, respectively). Entropy of pupil diameter was associated with performance in Dots and Needles task (multivariate analysis; p-value = 0.01). Average temporal flexibility and search information in several BAs and band frequencies were associated with performance and rate of learning. The models may be used to objectify performance and learning rate evaluation in RAS once validated with a broader sample size and tasks.

Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion

Existing vision-based fatigue detection methods commonly utilize RGB cameras to extract facial and physiological features for monitoring driver fatigue. These features often include single indicators such as eyelid movement, yawning frequency, and heart rate. However, the accuracy of RGB cameras can be affected by factors like varying lighting conditions and motion. To address these challenges, we propose a non-invasive method for multi-modal fusion fatigue detection called RPPMT-CNN-BiLSTM. This method incorporates a feature extraction enhancement module based on the improved Pan-Tompkins algorithm and 1D-MTCNN. This enhances the accuracy of heart rate signal extraction and eyelid features. Furthermore, we use one-dimensional neural networks to construct two models based on heart rate and PERCLOS values, forming a fatigue detection model. To enhance the robustness and accuracy of fatigue detection, the trained model data results are input into the BiLSTM network. This generates a time-fitting relationship between the data extracted from the CNN, allowing for effective dynamic modeling and achieving multi-modal fusion fatigue detection. Numerous experiments validate the effectiveness of the proposed method, achieving an accuracy of 98.2% on the self-made MDAD (Multi-Modal Driver Alertness Dataset). This underscores the feasibility of the algorithm. In comparison with traditional methods, our approach demonstrates higher accuracy and positively contributes to maintaining traffic safety, thereby advancing the field of smart transportation.

EEG-fest: few-shot based attention network for driver's drowsiness estimation with EEG signals

The leading factor behind most vehicular accidents is the driver's inattentiveness. To accurately determine driver's drowsiness, Electroencephalography (EEG) has been proven to be a reliable and effective method. Even though previous studies have developed accurate driver's drowsiness detection algorithms, certain challenges still persist, such as (a) limited training sample sizes, (b) detecting anomalous signals, and (c) achieving subject-independent classification. In this paper we propose a novel solution, names as EEG-Fest, which is a generalized few-shot model aimed at addressing the aforementioned limitations. The EEG-Fest has the ability to (a) classify a query sample's level of drowsiness with only a few support sample inputs (b) identify whether a query sample is anomalous signals or not, and (c) perform subject-independent classification. During the evaluation, our proposed EEG-Fest algorithm demonstrates better performance compared to other two conventional EEG algorithms in cross-subject validation.

Real driving environment EEG-based detection of driving fatigue using the wavelet scattering network

BACKGROUND

Driving fatigue is one of the main factors leading to traffic accidents. So, it is necessary to detect driver fatigue accurately and quickly.

NEW METHOD

To precisely detect driving fatigue in a real driving environment, this paper adopts a classification method for driving fatigue based on the wavelet scattering network (WSN). Firstly, electroencephalogram (EEG) signals of 12 subjects in the real driving environment are collected and categorized into two states: fatigue and awake. Secondly, the WSN algorithm extracts wavelet scattering coefficients of EEG signals, and these coefficients are used as input in support vector machine (SVM) as feature vectors for classification.

RESULTS

The results showed that the average classification accuracy of 12 subjects reached 99.33%; the average precision rate reached 99.28%; the average recall rate reached 98.27%; the average F1 score reached 98.74%; and the average classification accuracy of the public data set SEED-VIG reached 99.39%. The average precision, recall rate and F1 score reached 99.27%, 98.41% and 98.83% respectively.

COMPARISON WITH EXISTING METHODS

In addition, the WSN algorithm is compared with traditional convolutional neural network (CNN), Sparse-deep belief networks (SDBN), Spatio-temporal convolutional neural networks (STCNN), Long short-term memory (LSTM), and other methods, and it is found that WSN has higher classification accuracy.

CONCLUSION

Furthermore, this method has good versatility, providing excellent recognition effect on small sample data sets, and fast running time, making it convenient for real-time online monitoring of driver fatigue. Therefore, the WSN algorithm is promising in efficiently detecting driving fatigue state of drivers in real environments, contributing to improved traffic safety.

Mental fatigue assessment by an arbitrary channel EEG based on morphological features and LSTM-CNN

In order to achieve more sensitive mental fatigue assessment (MFA) based on an arbitrary channel EEG, this study proposed a series of feature extraction methods that combine mathematical morphology (MM), as well as an LSTM-CNN architecture. Firstly, 37 subjects had their resting-state EEGs collected at rested wakefulness (RW) and after 24 h of sleep deprivation (SD) using a 30-channel EEG acquisition device, the RW and SD groups were regarded as the negative and positive groups of mental fatigue, respectively, and the EEG collection were further categorized into two conditions: eye-opened state (EO) and eye-closed state (EC). Then, since MM can reflect the morphological characteristics of EEG rhythms and their potentials relatively independently of the time-frequency analysis and phase calculation, the MM methods were found to better reflect the mental fatigue after SD statistically, whether for single features (ANOVA: p<0.000001), multiple features (clustering by K-means, t-test: p<0.01), or time series feature spaces (calculating CD, t-test: p<0.01) of a single channel. Finally, the LSTM-CNN enhanced the generalization ability when dealing with different single-channel EEG by combining GRUs with convolutional layers: comparing the AUCs of different architectures for MFA based on an arbitrary channel, LSTM-CNN (0.992) > LSTM network (0.94) > CNN (0.831) > MLP (0.754). Moreover, the use of MM also improved the accuracy of analyzed architectures, and the true/false positive rate (TPR/FPR) of the LSTM-CNN architecture for MFA based on an arbitrary channel reached 97.024 %/3.497 %, which provided a feasible solution for the arbitrary channel EEG-based MFA.

Miner Fatigue Detection from Electroencephalogram-Based Relative Power Spectral Topography Using Convolutional Neural Network

Fatigue of miners is caused by intensive workloads, long working hours, and shift-work schedules. It is one of the major factors increasing the risk of safety problems and work mistakes. Examining the detection of miner fatigue is important because it can potentially prevent work accidents and improve working efficiency in underground coal mines. Many previous studies have introduced feature-based machine-learning methods to estimate miner fatigue. This work proposes a method that uses electroencephalogram (EEG) signals to generate topographic maps containing frequency and spatial information. It utilizes a convolutional neural network (CNN) to classify the normal state, critical state, and fatigue state of miners. The topographic maps are generated from the EEG signals and contrasted using power spectral density (PSD) and relative power spectral density (RPSD). These two feature extraction methods were applied to feature recognition and four representative deep-learning methods. The results showthat RPSD achieves better performance than PSD in classification accuracy with all deep-learning methods. The CNN achieved superior results to the other deep-learning methods, with an accuracy of 94.5%, precision of 97.0%, sensitivity of 94.8%, and F1 score of 96.3%. Our results also show that the RPSD-CNN method outperforms the current state of the art. Thus, this method might be a useful and effective miner fatigue detection tool for coal companies in the near future.

CSTNet: A Dual-Branch Convolutional Neural Network for Imaging of Reactive Flows Using Chemical Species Tomography

Chemical species tomography (CST) has been widely used for in situ imaging of critical parameters, e.g., species concentration and temperature, in reactive flows. However, even with state-of-the-art computational algorithms, the method is limited due to the inherently ill-posed and rank-deficient tomographic data inversion and by high computational cost. These issues hinder its application for real-time flow diagnosis. To address them, we present here a novel convolutional neural network, namely CSTNet, for high-fidelity, rapid, and simultaneous imaging of species concentration and temperature using CST. CSTNet introduces a shared feature extractor that incorporates the CST measurements and sensor layout into the learning network. In addition, a dual-branch decoder with internal crosstalk, which automatically learns the naturally correlated distributions of species concentration and temperature, is proposed for image reconstructions. The proposed CSTNet is validated both with simulated datasets and with measured data from real flames in experiments using an industry-oriented sensor. Superior performance is found relative to previous approaches in terms of reconstruction accuracy and robustness to measurement noise. This is the first time, to the best of our knowledge, that a deep learning-based method for CST has been experimentally validated for simultaneous imaging of multiple critical parameters in reactive flows using a low-complexity optical sensor with a severely limited number of laser beams.

Characterisation of Cognitive Load Using Machine Learning Classifiers of Electroencephalogram Data

A high cognitive load can overload a person, potentially resulting in catastrophic accidents. It is therefore important to ensure the level of cognitive load associated with safety-critical tasks (such as driving a vehicle) remains manageable for drivers, enabling them to respond appropriately to changes in the driving environment. Although electroencephalography (EEG) has attracted significant interest in cognitive load research, few studies have used EEG to investigate cognitive load in the context of driving. This paper presents a feasibility study on the simulation of various levels of cognitive load through designing and implementing four driving tasks. We employ machine learning-based classification techniques using EEG recordings to differentiate driving conditions. An EEG dataset containing these four driving tasks from a group of 20 participants was collected to investigate whether EEG can be used as an indicator of changes in cognitive load. The collected dataset was used to train four Deep Neural Networks and four Support Vector Machine classification models. The results showed that the best model achieved a classification accuracy of 90.37%, utilising statistical features from multiple frequency bands in 24 EEG channels. Furthermore, the Gamma and Beta bands achieved higher classification accuracy than the Alpha and Theta bands during the analysis. The outcomes of this study have the potential to enhance the Human-Machine Interface of vehicles, contributing to improved safety.

EEG-Based Cross-Subject Driver Drowsiness Recognition With an Interpretable Convolutional Neural Network

In the context of electroencephalogram (EEG)-based driver drowsiness recognition, it is still challenging to design a calibration-free system, since EEG signals vary significantly among different subjects and recording sessions. Many efforts have been made to use deep learning methods for mental state recognition from EEG signals. However, existing work mostly treats deep learning models as black-box classifiers, while what have been learned by the models and to which extent they are affected by the noise in EEG data are still underexplored. In this article, we develop a novel convolutional neural network combined with an interpretation technique that allows sample-wise analysis of important features for classification. The network has a compact structure and takes advantage of separable convolutions to process the EEG signals in a spatial-temporal sequence. Results show that the model achieves an average accuracy of 78.35% on 11 subjects for leave-one-out cross-subject drowsiness recognition, which is higher than the conventional baseline methods of 53.40%-72.68% and state-of-the-art deep learning methods of 71.75%-75.19%. Interpretation results indicate the model has learned to recognize biologically meaningful features from EEG signals, e.g., alpha spindles, as strong indicators of drowsiness across different subjects. In addition, we also explore reasons behind some wrongly classified samples with the interpretation technique and discuss potential ways to improve the recognition accuracy. Our work illustrates a promising direction on using interpretable deep learning models to discover meaningful patterns related to different mental states from complex EEG signals.

Psychophysiology of Monotonous Driving, Fatigue and Sleepiness in Train and Non-Professional Drivers: Driver Safety Implications

Fatigue and sleepiness are complex bodily states associated with monotony as well as physical and cognitive impairment, accidents, injury, and illness. Moreover, these states are often characteristic of professional driving. However, most existing work has focused on motor vehicle drivers, and research examining train drivers remains limited. As such, the present study psychophysiologically examined monotonous driving, fatigue, and sleepiness in a group of passenger train drivers and a group of non-professional drivers. Sixty-three train drivers and thirty non-professional drivers participated in the present study, which captured 32-lead electroencephalogram (EEG) data during a monotonous driving task. Fatigue and sleepiness were self-evaluated using the Pittsburgh Sleep Quality Index, the Epworth Sleepiness Scale, the Karolinksa Sleepiness Scale, and the Checklist of Individual Strength. Unexpectedly, fatigue and sleepiness scores did not significantly differ between the groups; however, train drivers generally scored lower than non-professional drivers, which may be indicative of individual and/or industry attempts to reduce fatigue. Across both groups, fatigue and sleepiness scores were negatively correlated with theta, alpha, and beta EEG variables clustered towards the fronto-central and temporal regions. Broadly, these associations may reflect a monotony-associated blunting of neural activity that is associated with a self-reported fatigue state.

Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction

Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.

Driver drowsiness detection methods using EEG signals: a systematic review

Electroencephalography (EEG) is a complex signal that may require several years of training, advanced signal processing, and feature extraction methodologies to interpret correctly. Recently, many methods have been used to extract and classify EEG data. This study reviews 62 papers that used EEG signals to detect driver drowsiness, published between January 2018 and 2022. We extract trends and highlight interesting approaches from this large body of literature to inform future research and formulate recommendations. To find relevant papers published in scientific journals, conferences, and electronic preprint repositories, researchers searched major databases covering the domains of science and engineering. For each investigation, many data items about (1) the data, (2) the channels used, (3) the extraction and classification procedure, and (4) the outcomes were extracted. These items were then analyzed one by one to uncover trends. Our analysis reveals that the amount of EEG data used across studies varies. We saw that more than half the studies used simulation driving experimental. About 21% of the studies used support vector machine (SVM), while 19% used convolutional neural networks (CNN). Overall, we can conclude that drowsiness and fatigue impair driving performance, resulting in drivers who are more exposed to risky situations.

Temporal-spatial convolutional residual network for decoding attempted movement related EEG signals of subjects with spinal cord injury

Brain Computer Interface (BCI) offers a promising approach to restoring hand functionality for people with cervical spinal cord injury (SCI). A reliable classification of brain activities based on appropriate flexibility in feature extraction could enhance BCI systems performance. In the present study, based on convolutional layers with temporal-spatial, Separable and Depthwise structures, we develop Temporal-Spatial Convolutional Residual Network)TSCR-Net(and Temporal-Spatial Convolutional Iterative Residual Network)TSCIR-Net(structures to classify electroencephalogram (EEG) signals. Using EEG signals in five different hand movement classes of SCI people, we compare the effectiveness of TSCIR-Net and TSCR-Net models with some competitive methods. We use the bayesian hyperparameter optimization algorithm to tune the hyperparameters of compact convolutional neural networks. In order to show the high generalizability of the proposed models, we compare the results of the models in different frequency ranges. Our proposed models decoded distinctive characteristics of different movement efforts and obtained higher classification accuracy than previous deep neural networks. Our findings indicate that TSCIR-Net and TSCR-Net models fulfills a better classification accuracy of 71.11%, and 64.55% for EEG_All and 57.74%, and 67.87% for EEG_Low frequency data sets than the compared methods in the literature.

A regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based cross-subject fatigue detection

Fatigue, one of the most important factors affecting road safety, has attracted many researchers' attention. Most existing fatigue detection methods are based on feature engineering and classification models. The feature engineering is greatly influenced by researchers' domain knowledge, which will lead to a poor performance in fatigue detection, especially in cross-subject experiment design. In addition, fatigue detection is often simplified as a classification problem of several discrete states. Models based on deep learning can realize automatic feature extraction without the limitation of researcher's domain knowledge. Therefore, this paper proposes a regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based (EEG-based) cross-subject fatigue detection. At the same time, a twofold random-offset zero-overlapping sampling method is proposed to train a bigger model and reduce overfitting. Compared with existing results, the proposed method achieves a much better result of 0.94 correlation coefficient (COR) and 0.09 root mean square error (RMSE) in a within-subject experiment design. What is more, there is no misclassification between awake and drowsy states. For cross-subject experiment design, the COR and RMSE are 0.79 and 0.15, respectively, which are close to the existing within-subject results and better than similar cross-subject results. The cross-subject regression model is very important for fatigue detection application since the fatigue indication is more precise than several discrete states and no model calibration is required for a new user. The twofold random-offset zero-overlapping sampling method can also be used as a reference by other EEG-based deep learning research.

Self-Attentive Channel-Connectivity Capsule Network for EEG-Based Driving Fatigue Detection

Deep neural networks have recently been successfully extended to EEG-based driving fatigue detection. Nevertheless, most existing models fail to reveal the intrinsic inter-channel relations that are known to be beneficial for EEG-based classification. Additionally, these models require substantial data for training, which is often impractical due to the high cost of data collection. To simultaneously address these two issues, we propose a Self-Attentive Channel-Connectivity Capsule Network (SACC-CapsNet) for EEG-based driving fatigue detection in this paper. SACC-CapsNet starts with a temporal-channel attention module to investigate the critical temporal information and important channels for driving fatigue detection, refining the input EEG signals. Subsequently, the refined EEG data are transformed into a channel covariance matrix to capture the inter-channel relations, followed by selective kernel attention to extract the highly discriminative channel-connectivity features. Finally, a capsule neural network is employed to effectively learn the relationships between connectivity features, which is more suitable for limited data. To confirm the effectiveness of SACC-CapsNet, we collected 24-channel EEG data from 31 subjects (mean age=23.13±2.68 years, male/female=18/13) in a simulated fatigue driving environment. Extensive experiments were conducted with the acquired data, and the comparison results show that our proposed model outperforms state-of-the-art methods. Additionally, the channel covariance matrix learned from SACC-CapsNet reveals that the frontal pole is most informative for detecting driving fatigue, followed by the parietal and central regions. Intriguingly, the temporal-channel attention module can enhance the significance of these critical regions, and the reconstructed channel covariance matrix generated by the decoder network of SACC-CapsNet can effectively preserve valuable information about them.

EEG Signal Epilepsy Detection With a Weighted Neighbor Graph Representation and Two-Stream Graph-Based Framework

Epilepsy is one of the most common neurological diseases. Clinically, epileptic seizure detection is usually performed by analyzing electroencephalography (EEG) signals. At present, deep learning models have been widely used for single-channel EEG signal epilepsy detection, but this method is difficult to explain the classification results. Researchers have attempted to solve interpretive problems by combining graph representation of EEG signals with graph neural network models. Recently, the combination of graph representations and graph neural network (GNN) models has been increasingly applied to single-channel epilepsy detection. By this methodology, the raw EEG signal is transformed to its graph representation, and a GNN model is used to learn latent features and classify whether the data indicates an epileptic seizure episode. However, existing methods are faced with two major challenges. First, existing graph representations tend to have high time complexity as they generally require each vertex to traverse all other vertices to construct a graph structure. Some of them also have high space complexity for being dense. Second, while separate graph representations can be derived from a single-channel EEG signal in both time and frequency domains, existing GNN models for epilepsy detection can learn from a single graph representation, which makes it hard to let the information from the two domains complement each other. For addressing these challenges, we propose a Weighted Neighbour Graph (WNG) representation for EEG signals. Reducing the redundant edges of the existing graph, WNG can be both time and space-efficient, and as informative as its less efficient counterparts. We then propose a two-stream graph-based framework to simultaneously learn features from WNG in both time and frequency domain. Extensive experiments demonstrate the effectiveness and efficiency of the proposed methods.

An EEG Channel Selection Framework for Driver Drowsiness Detection via Interpretability Guidance

Drowsy driving has a crucial influence on driving safety, creating an urgent demand for driver drowsiness detection. Electroencephalogram (EEG) signal can accurately reflect the mental fatigue state and thus has been widely studied in drowsiness monitoring. However, the raw EEG data is inherently noisy and redundant, which is neglected by existing works that just use single-channel EEG data or full-head channel EEG data for model training, resulting in limited performance of driver drowsiness detection. In this paper, we are the first to propose an Interpretability-guided Channel Selection (ICS) framework for the driver drowsiness detection task. Specifically, we design a two-stage training strategy to progressively select the key contributing channels with the guidance of interpretability. We first train a teacher network in the first stage using full-head channel EEG data. Then we apply the class activation mapping (CAM) to the trained teacher model to highlight the high-contributing EEG channels and further propose a channel voting scheme to select the top N contributing EEG channels. Finally, we train a student network with the selected channels of EEG data in the second stage for driver drowsiness detection. Experiments are designed on a public dataset, and the results demonstrate that our method is highly applicable and can significantly improve the performance of cross-subject driver drowsiness detection.

EEG-Based Driver Fatigue Monitoring within a Human-Ship-Environment System: Implications for Ship Braking Safety

To address the uncontrollable risks associated with the overreliance on ship operators' driving in current ship safety braking methods, this study aims to reduce the impact of operator fatigue on navigation safety. Firstly, this study established a human-ship-environment monitoring system with functional and technical architecture, emphasizing the investigation of a ship braking model that integrates brain fatigue monitoring using electroencephalography (EEG) to reduce braking safety risks during navigation. Subsequently, the Stroop task experiment was employed to induce fatigue responses in drivers. By utilizing principal component analysis (PCA) to reduce dimensionality across multiple channels of the data acquisition device, this study extracted centroid frequency (CF) and power spectral entropy (PSE) features from channels 7 and 10. Additionally, a correlation analysis was conducted between these features and the Fatigue Severity Scale (FSS), a five-point scale for assessing fatigue severity in the subjects. This study established a model for scoring driver fatigue levels by selecting the three features with the highest correlation and utilizing ridge regression. The human-ship-environment monitoring system and fatigue prediction model proposed in this study, combined with the ship braking model, achieve a safer and more controllable ship braking process. By real-time monitoring and prediction of driver fatigue, appropriate measures can be taken in a timely manner to ensure navigation safety and driver health.

Electroencephalogram based brain-computer interface: Applications, challenges, and opportunities

Brain-Computer Interfaces (BCI) is an exciting and emerging research area for researchers and scientists. It is a suitable combination of software and hardware to operate any device mentally. This review emphasizes the significant stages in the BCI domain, current problems, and state-of-the-art findings. This article also covers how current results can contribute to new knowledge about BCI, an overview of BCI from its early developments to recent advancements, BCI applications, challenges, and future directions. The authors pointed to unresolved issues and expressed how BCI is valuable for analyzing the human brain. Humans' dependence on machines has led humankind into a new future where BCI can play an essential role in improving this modern world.

T-A-MFFNet: Multi-feature fusion network for EEG analysis and driving fatigue detection based on time domain network and attention network

Driving fatigue detection based on EEG signals is a research hotspot in applying brain-computer interfaces. EEG signal is complex, unstable, and nonlinear. Most existing methods rarely analyze the data characteristics from multiple dimensions, so it takes work to analyze the data comprehensively. To analyze EEG signals more comprehensively, this paper evaluates a feature extraction strategy of EEG data based on differential entropy (DE). This method combines the characteristics of different frequency bands, extracts the frequency domain characteristics of EEG, and retains the spatial information between channels. This paper proposes a multi-feature fusion network (T-A-MFFNet) based on the time domain and attention network. The model is composed of a time domain network (TNet), channel attention network (CANet), spatial attention network (SANet), and multi-feature fusion network(MFFNet) based on a squeeze network. T-A-MFFNet aims to learn more valuable features from the input data to achieve good classification results. Specifically, the TNet network extracts high-level time series information from EEG data. CANet and SANet are used to fuse channel and spatial features. They use MFFNet to merge multi-dimensional features and realize classification. The validity of the model is verified on the SEED-VIG dataset. The experimental results show that the accuracy of the proposed method reaches 85.65 %, which is superior to the current popular model. The proposed method can learn more valuable information from EEG signals to improve the ability to identify fatigue status and promote the development of the research field of driving fatigue detection based on EEG signals.

Status of deep learning for EEG-based brain-computer interface applications

In the previous decade, breakthroughs in the central nervous system bioinformatics and computational innovation have prompted significant developments in brain-computer interface (BCI), elevating it to the forefront of applied science and research. BCI revitalization enables neurorehabilitation strategies for physically disabled patients (e.g., disabled patients and hemiplegia) and patients with brain injury (e.g., patients with stroke). Different methods have been developed for electroencephalogram (EEG)-based BCI applications. Due to the lack of a large set of EEG data, methods using matrix factorization and machine learning were the most popular. However, things have changed recently because a number of large, high-quality EEG datasets are now being made public and used in deep learning-based BCI applications. On the other hand, deep learning is demonstrating great prospects for solving complex relevant tasks such as motor imagery classification, epileptic seizure detection, and driver attention recognition using EEG data. Researchers are doing a lot of work on deep learning-based approaches in the BCI field right now. Moreover, there is a great demand for a study that emphasizes only deep learning models for EEG-based BCI applications. Therefore, we introduce this study to the recent proposed deep learning-based approaches in BCI using EEG data (from 2017 to 2022). The main differences, such as merits, drawbacks, and applications are introduced. Furthermore, we point out current challenges and the directions for future studies. We argue that this review study will help the EEG research community in their future research.

End-to-end fatigue driving EEG signal detection model based on improved temporal-graph convolution network

Fatigue driving is one of the leading causes of traffic accidents, so fatigue driving detection technology plays a crucial role in road safety. The physiological information-based fatigue detection methods have the advantage of objectivity and accuracy. Among many physiological signals, EEG signals are considered to be the most direct and promising ones. Most traditional methods are challenging to train and do not meet real-time requirements. To this end, we propose an end-to-end temporal and graph convolution-based (MATCN-GT) fatigue driving detection algorithm. The MATCN-GT model consists of a multi-scale attentional temporal convolutional neural network block (MATCN block) and a graph convolutional-Transformer block (GT block). Among them, the MATCN block extracts features directly from the original EEG signal without a priori information, and the GT block processes the features of EEG signals between different electrodes. In addition, we design a multi-scale attention module to ensure that valuable information on electrode correlations will not be lost. We add a Transformer module to the graph convolutional network, which can better capture the dependencies between long-distance electrodes. We conduct experiments on the public dataset SEED-VIG, and the accuracy of the MATCN-GT model reached 93.67%, outperforming existing algorithms. Furthermore, compared with the traditional graph convolutional neural network, the GT block has improved the accuracy rate by 3.25%. The accuracy of the MATCN block on different subjects is higher than the existing feature extraction methods.

EEG driving fatigue detection based on log-Mel spectrogram and convolutional recurrent neural networks

Driver fatigue detection is one of the essential tools to reduce accidents and improve traffic safety. Its main challenge lies in the problem of how to identify the driver's fatigue state accurately. Existing detection methods include yawning and blinking based on facial expressions and physiological signals. Still, lighting and the environment affect the detection results based on facial expressions. In contrast, the electroencephalographic (EEG) signal is a physiological signal that directly responds to the human mental state, thus reducing the impact on the detection results. This paper proposes a log-Mel spectrogram and Convolution Recurrent Neural Network (CRNN) model based on EEG to implement driver fatigue detection. This structure allows the advantages of the different networks to be exploited to overcome the disadvantages of using them individually. The process is as follows: first, the original EEG signal is subjected to a one-dimensional convolution method to achieve a Short Time Fourier Transform (STFT) and passed through a Mel filter bank to obtain a logarithmic Mel spectrogram, and then the resulting logarithmic Mel spectrogram is fed into a fatigue detection model to complete the fatigue detection task for the EEG signals. The fatigue detection model consists of a 6-layer convolutional neural network (CNN), bi-directional recurrent neural networks (Bi-RNNs), and a classifier. In the modeling phase, spectrogram features are transported to the 6-layer CNN to automatically learn high-level features, thereby extracting temporal features in the bi-directional RNN to obtain spectrogram-temporal information. Finally, the alert or fatigue state is obtained by a classifier consisting of a fully connected layer, a ReLU activation function, and a softmax function. Experiments were conducted on publicly available datasets in this study. The results show that the method can accurately distinguish between alert and fatigue states with high stability. In addition, the performance of four existing methods was compared with the results of the proposed method, all of which showed that the proposed method could achieve the best results so far.

A systematic comparison of deep learning methods for EEG time series analysis

Analyzing time series data like EEG or MEG is challenging due to noisy, high-dimensional, and patient-specific signals. Deep learning methods have been demonstrated to be superior in analyzing time series data compared to shallow learning methods which utilize handcrafted and often subjective features. Especially, recurrent deep neural networks (RNN) are considered suitable to analyze such continuous data. However, previous studies show that they are computationally expensive and difficult to train. In contrast, feed-forward networks (FFN) have previously mostly been considered in combination with hand-crafted and problem-specific feature extractions, such as short time Fourier and discrete wavelet transform. A sought-after are easily applicable methods that efficiently analyze raw data to remove the need for problem-specific adaptations. In this work, we systematically compare RNN and FFN topologies as well as advanced architectural concepts on multiple datasets with the same data preprocessing pipeline. We examine the behavior of those approaches to provide an update and guideline for researchers who deal with automated analysis of EEG time series data. To ensure that the results are meaningful, it is important to compare the presented approaches while keeping the same experimental setup, which to our knowledge was never done before. This paper is a first step toward a fairer comparison of different methodologies with EEG time series data. Our results indicate that a recurrent LSTM architecture with attention performs best on less complex tasks, while the temporal convolutional network (TCN) outperforms all the recurrent architectures on the most complex dataset yielding a 8.61% accuracy improvement. In general, we found the attention mechanism to substantially improve classification results of RNNs. Toward a light-weight and online learning-ready approach, we found extreme learning machines (ELM) to yield comparable results for the less complex tasks.

EEG-based mental fatigue detection using linear prediction cepstral coefficients and Riemann spatial covariance matrix

Objective. Establishing a mental fatigue monitoring system is of great importance as for severe fatigue may cause unimaginable consequences. Electroencephalogram (EEG) is often utilized for mental fatigue detection because of its high temporal resolution and ease of use. However, many EEG-based approaches for detecting mental fatigue only take into account the feature extraction of a single domain and do not fully exploit the information that EEG may offer.Approach. In our work, we propose a new algorithm for mental fatigue detection based on multi-domain feature extraction and fusion. EEG components representing fatigue are closely related in the past and present because fatigue is a dynamic and gradual process. Accordingly, the idea of linear prediction is used to fit the current value with a set of sample values in the past to calculate the linear prediction cepstral coefficients (LPCCs) as the time domain feature. Moreover, in order to better capture fatigue-related spatial domain information, the spatial covariance matrix of the original EEG signal is projected into the Riemannian tangent space using the Riemannian geometric method. Then multi-domain features are fused to obtain comprehensive spatio-temporal information.Main results. Experimental results prove the suggested algorithm outperforms existing state-of-the-art methods, achieving an average accuracy of 87.10% classification on the public dataset SEED-VIG (three categories) and 97.40% classification accuracy (two categories) on the dataset made by self-designed experiments.Significance. These findings show that our proposed strategy perform more effectively for mental fatigue detection based on EEG.

Driving Fatigue Detection with Three Non-Hair-Bearing EEG Channels and Modified Transformer Model

Driving fatigue is the main cause of traffic accidents, which seriously affects people's life and property safety. Many researchers have applied electroencephalogram (EEG) signals for driving fatigue detection to reduce negative effects. The main challenges are the practicality and accuracy of the EEG-based driving fatigue detection method when it is applied on the real road. In our previous study, we attempted to improve the practicality of fatigue detection based on the proposed non-hair-bearing (NHB) montage with fewer EEG channels, but the recognition accuracy was only 76.47% with the random forest (RF) model. In order to improve the accuracy with NHB montage, this study proposed an improved transformer architecture for one-dimensional feature vector classification based on introducing the Gated Linear Unit (GLU) in the Attention sub-block and Feed-Forward Networks (FFN) sub-block of a transformer, called GLU-Oneformer. Moreover, we constructed an NHB-EEG-based feature set, including the same EEG features (power ratio, approximate entropy, and mutual information (MI)) in our previous study, and the lateralization features of the power ratio and approximate entropy based on the strategy of brain lateralization. The results indicated that our GLU-Oneformer method significantly improved the recognition performance and achieved an accuracy of 86.97%. Our framework demonstrated that the combination of the NHB montage and the proposed GLU-Oneformer model could well support driving fatigue detection.

Wearable electroencephalography and multi-modal mental state classification: A systematic literature review

BACKGROUND

Wearable multi-modal time-series classification applications outperform their best uni-modal counterparts and hold great promise. A modality that directly measures electrical correlates from the brain is electroencephalography. Due to varying noise sources, different key brain regions, key frequency bands, and signal characteristics like non-stationarity, techniques for data pre-processing and classification algorithms are task-dependent.

METHOD

Here, a systematic literature review on mental state classification for wearable electroencephalography is presented. Four search terms in different combinations were used for an in-title search. The search was executed on the 29th of June 2022, across Google Scholar, PubMed, IEEEXplore, and ScienceDirect. 76 most relevant publications were set into context as the current state-of-the-art in mental state time-series classification.

RESULTS

Pre-processing techniques, features, and time-series classification models were analyzed. Across publications, a window length of one second was mainly chosen for classification and spectral features were utilized the most. The achieved performance per time-series classification model is analyzed, finding linear discriminant analysis, decision trees, and k-nearest neighbors models outperform support-vector machines by a factor of up to 1.5. A historical analysis depicts future trends while under-reported aspects relevant to practical applications are discussed.

CONCLUSIONS

Five main conclusions are given, covering utilization of available area for electrode placement on the head, most often or scarcely utilized features and time-series classification model architectures, baseline reporting practices, as well as explainability and interpretability of Deep Learning. The importance of a 'test battery' assessing the influence of data pre-processing and multi-modality on time-series classification performance is emphasized.

Fatigue Detection of Pilots' Brain Through Brains Cognitive Map and Multilayer Latent Incremental Learning Model

This work proposes a nonparametric prior induced deep sum-logarithmic-multinomial mixture (DSLMM) model to detect pilots' cognitive states through the developed brain power map. DSLMM uses multinormal distribution to infer the latent variable of each neuron in the first layer of the network. These latent variables obeyed a sum-logarithmic distribution that is backpropagated to its observation vector and the number of neurons in the next layer. Multinormal distribution is used to segment the extended observation vector to form a matrix associated with the width of the next layer. This work also proposes an adaptive topic-layer stochastic gradient Riemann (ATL-SGR) Markov chain Monte Carlo (MCMC) inference method to learn its global parameters without heuristic assumptions. The experimental results indicate that DSLMM can extract more probability distribution contained in the brain power map layer by layer, and achieve higher pilot cognition detection accuracy.

Psychological distance and user engagement in online exhibitions: Visualization of moiré patterns based on electroencephalography signals

The COVID-19 pandemic has significantly affected the exhibition of artworks in museums and galleries. Many have displayed their collection online. In this context, experiencing an online exhibition is essential for visitors to appreciate and understand the artwork. Compared with offline exhibitions, visitors to online exhibitions are often unable to communicate their experiences with other visitors. Therefore, in this study, by facilitating communication via Zoom call, we established a system that allows two people to visit the museum together through the Google Arts and Culture (GA&C) website. To reduce the psychological distance between online visitors and help increase user engagement, we designed and developed a media device based on moiré pattern visualization of electroencephalography (EEG) signals. The participants were divided into two groups to visit the online museum, communicating remotely through Zoom. The objective of this study was to test whether a real-time EEG signal visualization device could help close the psychological distance between participants and whether it could increase user engagement with the online exhibition. Participants were randomly assigned to either the normal online exhibition experience (NOEE) group or EEG signal visualization device (ESVD) group. Participants in the NOEE group experienced four online exhibitions (Task1, Task2, Task3, and Task4) together (two participants per test unit). The conditions for participants in the ESVD group remained the same, apart from adding a media device to enable them to visualize EEG signals. A total of 40 university students participated in this study. Independent samples t-tests revealed that participants in the ESVD group perceived a significantly closer psychological distance between themselves and the participants on the opposite side than those in the NOEE group (t = −2.699; p = 0.008 < 0.05). A one-way ANOVA revealed that participants experienced Task3 with significantly closer psychological distance assessments than Task1 (p = 0.002 < 0.05), Task2 (p = 0.000 < 0.05), and Task4 (p = 0.001 < 0.05). Repeated ANOVAs revealed that participants in the ESVD group had higher overall user engagement than those in the NOEE group, with marginal significance (p = 0.056 < 0.1). Thus, this study shows that EEG visualization media devices can reduce the psychological distance between two participants when experiencing an online exhibition. Moreover, it can increase user engagement to some extent.

Modulating break types induces divergent low band EEG processes during post-break improvement: A power spectral analysis

Conventional wisdom suggests mid-task rest as a potential approach to relieve the time-on-task (TOT) effect while accumulating evidence indicated that acute exercise might also effectively restore mental fatigue. However, few studies have explored the neural mechanism underlying these different break types, and the results were scattered. This study provided one of the first looks at how different types of fatigue-recovery break exerted influence on the cognitive processes by evaluating the corresponding behavioral improvement and neural response (EEG power spectral) in a sustained attention task. Specifically, 19 participants performed three sessions of psychomotor vigilance tasks (PVT), with one session including a continuous 30-min PVT while the other two sessions additionally inserted a 15-min mid-task cycling and rest break, respectively. For behavioral performance, both types of break could restore objective vigilance transiently, while subjective feeling was only maintained after mid-task rest. Moreover, divergent patterns of EEG change were observed during post-break improvement. In detail, relative theta decreased and delta increased immediately after mid-task exercise, while decreased delta was found near the end of the rest-inserted task. Meanwhile, theta and delta could serve as neurological indicators to predict the reaction time change for exercise and rest intervention, respectively. In sum, our findings provided novel evidence to demonstrate divergent neural patterns following the mid-task exercise and rest intervention to counter TOT effects, which might lead to new insights into the nascent field of neuroergonomics for mental fatigue restoration.

Cross-subject emotion recognition using visibility graph and genetic algorithm-based convolution neural network

An efficient emotion recognition model is an important research branch in electroencephalogram (EEG)-based brain-computer interfaces. However, the input of the emotion recognition model is often a whole set of EEG channels obtained by electrodes placed on subjects. The unnecessary information produced by redundant channels affects the recognition rate and depletes computing resources, thereby hindering the practical applications of emotion recognition. In this work, we aim to optimize the input of EEG channels using a visibility graph (VG) and genetic algorithm-based convolutional neural network (GA-CNN). First, we design an experiment to evoke three types of emotion states using movies and collect the multi-channel EEG signals of each subject under different emotion states. Then, we construct VGs for each EEG channel and derive nonlinear features representing each EEG channel. We employ the genetic algorithm (GA) to find the optimal subset of EEG channels for emotion recognition and use the recognition results of the CNN as fitness values. The experimental results show that the recognition performance of the proposed method using a subset of EEG channels is superior to that of the CNN using all channels for each subject. Last, based on the subset of EEG channels searched by the GA-CNN, we perform cross-subject emotion recognition tasks employing leave-one-subject-out cross-validation. These results demonstrate the effectiveness of the proposed method in recognizing emotion states using fewer EEG channels and further enrich the methods of EEG classification using nonlinear features.

Improving EEG Decoding via Clustering-Based Multitask Feature Learning

Accurate electroencephalogram (EEG) pattern decoding for specific mental tasks is one of the key steps for the development of brain-computer interface (BCI), which is quite challenging due to the considerably low signal-to-noise ratio of EEG collected at the brain scalp. Machine learning provides a promising technique to optimize EEG patterns toward better decoding accuracy. However, existing algorithms do not effectively explore the underlying data structure capturing the true EEG sample distribution and, hence, can only yield a suboptimal decoding accuracy. To uncover the intrinsic distribution structure of EEG data, we propose a clustering-based multitask feature learning algorithm for improved EEG pattern decoding. Specifically, we perform affinity propagation-based clustering to explore the subclasses (i.e., clusters) in each of the original classes and then assign each subclass a unique label based on a one-versus-all encoding strategy. With the encoded label matrix, we devise a novel multitask learning algorithm by exploiting the subclass relationship to jointly optimize the EEG pattern features from the uncovered subclasses. We then train a linear support vector machine with the optimized features for EEG pattern decoding. Extensive experimental studies are conducted on three EEG data sets to validate the effectiveness of our algorithm in comparison with other state-of-the-art approaches. The improved experimental results demonstrate the outstanding superiority of our algorithm, suggesting its prominent performance for EEG pattern decoding in BCI applications.

2020 International brain-computer interface competition: A review

The brain-computer interface (BCI) has been investigated as a form of communication tool between the brain and external devices. BCIs have been extended beyond communication and control over the years. The 2020 international BCI competition aimed to provide high-quality neuroscientific data for open access that could be used to evaluate the current degree of technical advances in BCI. Although there are a variety of remaining challenges for future BCI advances, we discuss some of more recent application directions: (i) few-shot EEG learning, (ii) micro-sleep detection (iii) imagined speech decoding, (iv) cross-session classification, and (v) EEG(+ear-EEG) detection in an ambulatory environment. Not only did scientists from the BCI field compete, but scholars with a broad variety of backgrounds and nationalities participated in the competition to address these challenges. Each dataset was prepared and separated into three data that were released to the competitors in the form of training and validation sets followed by a test set. Remarkable BCI advances were identified through the 2020 competition and indicated some trends of interest to BCI researchers.

A Spike Neural Network Model for Lateral Suppression of Spike-Timing-Dependent Plasticity with Adaptive Threshold

Aiming at the practical constraints of high resource occupancy and complex calculations in the existing Spike Neural Network (SNN) image classification model, in order to seek a more lightweight and efficient machine vision solution, this paper proposes an adaptive threshold Spike Neural Network (SNN) model of lateral inhibition of Spike-Timing-Dependent Plasticity (STDP). The conversion from grayscale image to pulse sequence is completed by convolution normalization and first pulse time coding. The network self-classification is realized by combining the classical Spike-Timing-Dependent Plasticity algorithm (STDP) and lateral suppression algorithm. The occurrence of overfitting is effectively suppressed by introducing an adaptive threshold. The experimental results on the MNIST data set show that compared with the traditional SNN classification model, the complexity of the weight update algorithm is reduced from O(n2) to O(1), and the accuracy rate can still remain stable at about 96%. The provided model is conducive to the migration of software algorithms to the bottom layer of the hardware platform, and can provide a reference for the realization of edge computing solutions for small intelligent hardware terminals with high efficiency and low power consumption.

Behavioral Change Prediction from Physiological Signals Using Deep Learned Features

Predicting change from multivariate time series has relevant applications ranging from the medical to engineering fields. Multisensory stimulation therapy in patients with dementia aims to change the patient's behavioral state. For example, patients who exhibit a baseline of agitation may be paced to change their behavioral state to relaxed. This study aimed to predict changes in one's behavioral state from the analysis of the physiological and neurovegetative parameters to support the therapist during the stimulation session. In order to extract valuable indicators for predicting changes, both handcrafted and learned features were evaluated and compared. The handcrafted features were defined starting from the CATCH22 feature collection, while the learned ones were extracted using a temporal convolutional network, and the behavioral state was predicted through bidirectional long short-term memory auto-encoder, operating jointly. From the comparison with the state of the art, the learned features-based approach exhibits superior performance with accuracy rates of up to 99.42% with a time window of 70 seconds and up to 98.44% with a time window of 10 seconds.

Discriminative Frequencies and Temporal EEG Segmentation in the Motor Imagery Classification Approach

A linear discriminant analysis transformation-based approach to the classification of three different motor imagery types for brain–computer interfaces was considered. The study involved 16 conditionally healthy subjects (12 men, 4 women, mean age of 21.5 years). First, the search for subject-specific discriminative frequencies was conducted in the task of movement-related activity. This procedure was shown to increase the classification accuracy compared to the conditional common spatial pattern (CSP) algorithm, followed by a linear classifier considered as a baseline approach. In addition, an original approach to finding discriminative temporal segments for each motor imagery was tested. This led to a further increase in accuracy under the conditions of using Hjorth parameters and interchannel correlation coefficients as features calculated for the EEG segments. In particular, classification by the latter feature led to the best accuracy of 71.6%, averaged over all subjects (intrasubject classification), and, surprisingly, it also allowed us to obtain a comparable value of intersubject classification accuracy of 68%. Furthermore, scatter plots demonstrated that two out of three pairs of motor imagery were discriminated by the approach presented.

Brain functional and effective connectivity based on electroencephalography recordings: A review

Functional connectivity and effective connectivity of the human brain, representing statistical dependence and directed information flow between cortical regions, significantly contribute to the study of the intrinsic brain network and its functional mechanism. Many recent studies on electroencephalography (EEG) have been focusing on modeling and estimating brain connectivity due to increasing evidence that it can help better understand various brain neurological conditions. However, there is a lack of a comprehensive updated review on studies of EEG-based brain connectivity, particularly on visualization options and associated machine learning applications, aiming to translate those techniques into useful clinical tools. This article reviews EEG-based functional and effective connectivity studies undertaken over the last few years, in terms of estimation, visualization, and applications associated with machine learning classifiers. Methods are explored and discussed from various dimensions, such as either linear or nonlinear, parametric or nonparametric, time-based, and frequency-based or time-frequency-based. Then it is followed by a novel review of brain connectivity visualization methods, grouped by Heat Map, data statistics, and Head Map, aiming to explore the variation of connectivity across different brain regions. Finally, the current challenges of related research and a roadmap for future related research are presented.

A Hybrid Driver Fatigue and Distraction Detection Model Using AlexNet Based on Facial Features

Modern cities have imposed a fast-paced lifestyle where more drivers on the road suffer from fatigue and sleep deprivation. Consequently, road accidents have increased, becoming one of the leading causes of injuries and death among young adults and children. These accidents can be prevented if fatigue symptoms are diagnosed and detected sufficiently early. For this reason, we propose and compare two AlexNet CNN-based models to detect drivers’ fatigue behaviors, relying on head position and mouth movements as behavioral measures. We used two different approaches. The first approach is transfer learning, specifically, fine-tuning AlexNet, which allowed us to take advantage of what the model had already learned without developing it from scratch. The newly trained model was able to predict drivers’ drowsiness behaviors. The second approach is the use of AlexNet to extract features by training the top layers of the network. These features were reduced using non-negative matrix factorization (NMF) and classified with a support vector machine (SVM) classifier. The experiments showed that our proposed transfer learning model achieved an accuracy of 95.7%, while the feature extraction SVM-based model performed better, with an accuracy of 99.65%. Both models were trained on a simulated NTHU Driver Drowsiness Detection dataset.