Contrastive fine-grained domain adaptation network for EEG-based vigilance estimation

Vigilance state is crucial for the effective performance of users in brain-computer interface (BCI) systems. Most vigilance estimation methods rely on a large amount of labeled data to train a satisfactory model for the specific subject, which limits the practical application of the methods. This study aimed to build a reliable vigilance estimation method using a small amount of unlabeled calibration data. We conducted a vigilance experiment in the designed BCI-based cursor-control task. Electroencephalogram (EEG) signals of eighteen participants were recorded in two sessions on two different days. And, we proposed a contrastive fine-grained domain adaptation network (CFGDAN) for vigilance estimation. Here, an adaptive graph convolution network (GCN) was built to project the EEG data of different domains into a common space. The fine-grained feature alignment mechanism was designed to weight and align the feature distributions across domains at the EEG channel level, and the contrastive information preservation module was developed to preserve the useful target-specific information during the feature alignment. The experimental results show that the proposed CFGDAN outperforms the compared methods in our BCI vigilance dataset and SEED-VIG dataset. Moreover, the visualization results demonstrate the efficacy of the designed feature alignment mechanisms. These results indicate the effectiveness of our method for vigilance estimation. Our study is helpful for reducing calibration efforts and promoting the practical application potential of vigilance estimation methods.

An improved CapsNet based on data augmentation for driver vigilance estimation with forehead single-channel EEG

Various studies have shown that it is necessary to estimate the drivers' vigilance to reduce the occurrence of traffic accidents. Most existing EEG-based vigilance estimation studies have been performed on intra-subject and multi-channel signals, and these methods are too costly and complicated to implement in practice. Hence, aiming at the problem of cross-subject vigilance estimation of single-channel EEG signals, an estimation algorithm based on capsule network (CapsNet) is proposed. Firstly, we propose a new construction method of the input feature maps to fit the characteristics of CapsNet to improve the algorithm accuracy. Meanwhile, the self-attention mechanism is incorporated in the algorithm to focus on the key information in feature maps. Secondly, we propose substituting the traditional multi-channel signals with the single-channel signals to improve the utility of algorithm. Thirdly, since the single-channel signals carry fewer dimensions of the information compared to the multi-channel signals, we use the conditional generative adversarial network to improve the accuracy of single-channel signals by increasing the amount of data. The proposed algorithm is verified on the SEED-VIG, and Root-mean-square-error (RMSE) and Pearson Correlation Coefficient (PCC) are used as the evaluation metrics. The results show that the proposed algorithm improves the computing speed while the RMSE is reduced by 3%, and the PCC is improved by 12% compared to the mainstream algorithm. Experiment results prove the feasibility of using forehead single-channel EEG signals for cross-subject vigilance estimation and offering the possibility of lightweight EEG vigilance estimation devices for practical applications.

Multimodal Vigilance Estimation With Modality-Pairwise Contrastive Loss

Nowadays, how to estimate vigilance with higher accuracy has become a hot field of research direction. Although the increasing available modalities opens the door for amazing new possibilities to achieve good performance, the uncertain cross-modal interaction still poses a real challenge to the multimodal fusion. In this paper, a cross-modality alignment method has been proposed based on the contrastive learning for extracting shared but not the same information among modalities. The contrastive learning is adopted to minimize the intermodal differences by maximizing the similarity of semantic representation of modalities. Applying our proposed modeling framework, we evaluated our approach on SEED-VIG dataset consisting of EEG and EOG signals. Experiments showed that our study achieved state-of-the-art multimodal vigilance estimation performance both in intra-subject and inter-subject situations, the average of RMSE/CORR were improved to 0.092/0.893 and 0.144/0.887, respectively. In addition, analysis on the frequency bands showed that theta and alpha activities contain valuable information for vigilance estimation, and the correlation between them and PERCLOS can be significantly improved by contrastive learning. We argue that the proposed method in the inter-subject case could offer the possibility of reducing the high-cost of data annotation, and further analysis may provide an idea for the application of multimodal vigilance regression.

Human Eye Activity Monitoring Using Continuous Wave Doppler Radar: A Feasibility Study

Human eye activity has been widely studied in many fields such as psychology, neuroscience, medicine, and human-computer interaction engineering. In previous studies, monitoring of human eye activity mainly depends on electrooculogram (EOG) that requires a contact sensor. This article proposes a novel eye movement monitoring method called continuous wave doppler oculogram (cDOG). Unlike the conventional EOG-based eye movement monitoring methods, cDOG based on continuous wave doppler radar sensor (cDRS) can remotely measure human eye activity without placing electrodes on the head. To verify the feasibility of using cDOG for eye movement monitoring, we first theoretically analyzed the association between the radar signal and the corresponding eye movements measured with EOG. Afterward, we conducted an experiment to compare EOG and cDOG measurements under the conditions of eyes closure and opening. In addition, different eye movement states were considered, including right-left saccade, up-down saccade, eye-blink, and fixation. Several representative time domain and frequency domain features obtained from cDOG and from EOG were compared in these states, allowing us to demonstrate the feasibility of using cDOG for monitoring eye movements. The experimental results show that there is a correlation between cDOG and EOG in the time and frequency domain features, the average time error of single eye movement is less than 280.5 ms, and the accuracy of cDOG in eye movement detection is higher than 92.35%, when the distance between the cDRS and the face is 10 cm and eyes is facing the radar directly.

Vigilance estimation using truncated l1 distance kernel-based sparse representation regression with physiological signals

BACKGROUND

With a large number of accidents caused by the decline in the vigilance of operators, finding effective automatic vigilance monitoring methods is a work of great significance in recent years. Based on physiological signals and machine learning algorithms, researchers have opened up a path for objective vigilance estimation.

METHODS

Sparse representation (SR)-based recognition algorithms with excellent performance and simple models are very promising approaches in this field. This paper aims to study the adaptability and performance improvement of truncated l1 distance (TL1) kernel on SR-based algorithm in the context of physiological signal vigilance estimation. Compared with the traditional radial basis function (RBF), the TL1 kernel has good adaptiveness to nonlinearity and is suitable for the discrimination of complex physiological signals. A recognition framework based on TL1 and SR theory is proposed. Firstly, the inseparable physiological features are mapped to the reproducing kernel Kreĭn space through the infinite-dimensional projection of the TL1 kernel. Then the obtained kernel matrix is converted into the symmetric positive definite matrix according to the eigenspectrum approaches. Finally, the final prediction result is obtained through the sparse representation regression process.

RESULTS

We verified the performance of the proposed framework on the popular SEED-VIG dataset containing physiological signals (electroencephalogram and electrooculogram) associated with vigilance. In the experimental results, the TL1 kernel is superior to the RBF kernel in both performance and kernel parameter stability.

CONCLUSIONS

This demonstrates the effectiveness of the TL1 kernel in distinguishing physiological signals and the excellent vigilance estimation capability of the proposed framework. Moreover, the contribution of our research motivates the development of physiological signal recognition based on kernel methods.

EEG-fest: few-shot based attention network for driver's drowsiness estimation with EEG signals

The leading factor behind most vehicular accidents is the driver's inattentiveness. To accurately determine driver's drowsiness, Electroencephalography (EEG) has been proven to be a reliable and effective method. Even though previous studies have developed accurate driver's drowsiness detection algorithms, certain challenges still persist, such as (a) limited training sample sizes, (b) detecting anomalous signals, and (c) achieving subject-independent classification. In this paper we propose a novel solution, names as EEG-Fest, which is a generalized few-shot model aimed at addressing the aforementioned limitations. The EEG-Fest has the ability to (a) classify a query sample's level of drowsiness with only a few support sample inputs (b) identify whether a query sample is anomalous signals or not, and (c) perform subject-independent classification. During the evaluation, our proposed EEG-Fest algorithm demonstrates better performance compared to other two conventional EEG algorithms in cross-subject validation.

Semisupervised Manifold Regularization via a Subnetwork-Based Representation Learning Model

Semisupervised classification with a few labeled training samples is a challenging task in the area of data mining. Moore-Penrose inverse (MPI)-based manifold regularization (MR) is a widely used technique in tackling semisupervised classification. However, most of the existing MPI-based MR algorithms can only generate loosely connected feature encoding, which is generally less effective in data representation and feature learning. To alleviate this deficiency, we introduce a new semisupervised multilayer subnet neural network called SS-MSNN. The key contributions of this article are as follows: 1) a novel MPI-based MR model using the subnetwork structure is introduced. The subnet model is utilized to enrich the latent space representations iteratively; 2) a one-step training process to learn the discriminative encoding is proposed. The proposed SS-MSNN learns parameters by directly optimizing the entire network, accepting input from one end, and producing output at the other end; and 3) a new semisupervised dataset called HFSWR-RDE is built for this research. Experimental results on multiple domains show that the SS-MSNN achieves promising performance over the other semisupervised learning algorithms, demonstrating fast inference speed and better generalization ability.

A Methodology for Enhancing SSVEP Features Using Adaptive Filtering Based on the Spatial Distribution of EEG Signals

In this paper, we propose a classification algorithm of EEG signal based on canonical correlation analysis (CCA) and integrated with adaptive filtering. It can enhance the detection of steady-state visual evoked potentials (SSVEPs) in a brain-computer interface (BCI) speller. An adaptive filter is employed in front of the CCA algorithm to improve the signal-to-noise ratio (SNR) of SSVEP signals by removing background electroencephalographic (EEG) activities. The ensemble method is developed to integrate recursive least squares (RLS) adaptive filter corresponding to multiple stimulation frequencies. The method is tested by the SSVEP signal recorded from six targets by actual experiment and the EEG in a public SSVEP dataset of 40 targets from Tsinghua University. The accuracy rates of the CCA method and the CCA-based integrated RLS filter algorithm (RLS-CCA method) are compared. Experiment results show that the proposed RLS-CCA-based method significantly improves the classification accuracy compared with the pure CCA method. Especially when the number of EEG leads is low (three occipital electrodes and five non occipital electrodes), its advantage is more significant, and accuracy reaches 91.23%, which is more suitable for wearable environments where high-density EEG is not easy to collect.

An Electro-Oculogram (EOG) Sensor's Ability to Detect Driver Hypovigilance Using Machine Learning

Driving safely is crucial to avoid death, injuries, or financial losses that can be sustained in an accident. Thus, a driver's physical state should be monitored to prevent accidents, rather than vehicle-based or behavioral measurements, and provide reliable information in this regard. Electrocardiography (ECG), electroencephalography (EEG), electrooculography (EOG), and surface electromyography (sEMG) signals are used to monitor a driver's physical state during a drive. The purpose of this study was to detect driver hypovigilance (drowsiness, fatigue, as well as visual and cognitive inattention) using signals collected from 10 drivers while they were driving. EOG signals from the driver were preprocessed to remove noise, and 17 features were extracted. ANOVA (analysis of variance) was used to select statistically significant features that were then loaded into a machine learning algorithm. We then reduced the features by using principal component analysis (PCA) and trained three classifiers: support vector machine (SVM), k-nearest neighbor (KNN), and ensemble. A maximum accuracy of 98.7% was obtained for the classification of normal and cognitive classes under the category of two-class detection. Upon considering hypovigilance states as five-class, a maximum accuracy of 90.9% was achieved. In this case, the number of detection classes increased, resulting in a reduction in the accuracy of detecting more driver states. However, with the possibility of incorrect identification and the presence of issues, the ensemble classifier's performance produced an enhanced accuracy when compared to others.

Feasibility study of personalized speed adaptation method based on mental state for teleoperated robots

The teleoperated robotic system can support humans to complete tasks in high-risk, high-precision and difficult special environments. Because this kind of special working environment is easy to cause stress, high mental workload, fatigue and other mental states of the operator, which will reduce the quality of operation and even cause safety accidents, so the mental state of the people in this system has received extensive attention. However, the existence of individual differences and mental state diversity is often ignored, so that most of the existing adjustment strategy is out of a match between mental state and adaptive decision, which cannot effectively improve operational quality and safety. Therefore, a personalized speed adaptation (PSA) method based on policy gradient reinforcement learning was proposed in this paper. It can use electroencephalogram and electro-oculogram to accurately perceive the operator’s mental state, and adjust the speed of the robot individually according to the mental state of different operators, in order to perform teleoperation tasks efficiently and safely. The experimental results showed that the PSA method learns the mapping between the mental state and the robot’s speed regulation action by means of rewards and punishments, and can adjust the speed of the robot individually according to the mental state of different operators, thereby improving the operating quality of the system. And the feasibility and superiority of this method were proved. It is worth noting that the PSA method was validated on 6 real subjects rather than a simulation model. To the best of our knowledge, the PSA method is the first implementation of online reinforcement learning control of teleoperated robots involving human subjects.

Fatigue Driving Detection Method Based on Combination of BP Neural Network and Time Cumulative Effect

Fatigue driving has always received a lot of attention, but few studies have focused on the fact that human fatigue is a cumulative process over time, and there are no models available to reflect this phenomenon. Furthermore, the problem of incorrect detection due to facial expression is still not well addressed. In this article, a model based on BP neural network and time cumulative effect was proposed to solve these problems. Experimental data were used to carry out this work and validate the proposed method. Firstly, the Adaboost algorithm was applied to detect faces, and the Kalman filter algorithm was used to trace the face movement. Then, a cascade regression tree-based method was used to detect the 68 facial landmarks and an improved method combining key points and image processing was adopted to calculate the eye aspect ratio (EAR). After that, a BP neural network model was developed and trained by selecting three characteristics: the longest period of continuous eye closure, number of yawns, and percentage of eye closure time (PERCLOS), and then the detection results without and with facial expressions were discussed and analyzed. Finally, by introducing the Sigmoid function, a fatigue detection model considering the time accumulation effect was established, and the drivers’ fatigue state was identified segment by segment through the recorded video. Compared with the traditional BP neural network model, the detection accuracies of the proposed model without and with facial expressions increased by 3.3% and 8.4%, respectively. The number of incorrect detections in the awake state also decreased obviously. The experimental results show that the proposed model can effectively filter out incorrect detections caused by facial expressions and truly reflect that driver fatigue is a time accumulating process.

Multimodel Feature Reinforcement Framework Using Moore-Penrose Inverse for Big Data Analysis

Fully connected representation learning (FCRL) is one of the widely used network structures in multimodel image classification frameworks. However, most FCRL-based structures, for instance, stacked autoencoder encode features and find the final cognition with separate building blocks, resulting in loosely connected feature representation. This article achieves a robust representation by considering a low-dimensional feature and the classifier model simultaneously. Thus, a new hierarchical subnetwork-based neural network (HSNN) is proposed in this article. The novelties of this framework are as follows: 1) it is an iterative learning process, instead of stacking separate blocks to obtain the discriminative encoding and the final classification results. In this sense, the optimal global features are generated; 2) it applies Moore-Penrose (MP) inverse-based batch-by-batch learning strategy to handle large-scale data sets, so that large data set, such as Place365 containing 1.8 million images, can be processed effectively. The experimental results on multiple domains with a varying number of training samples from  ∼  1 K to  ∼ 2 M show that the proposed feature reinforcement framework achieves better generalization performance compared with most state-of-the-art FCRL methods.

Deep Coupling Recurrent Auto-Encoder with Multi-Modal EEG and EOG for Vigilance Estimation

Vigilance estimation of drivers is a hot research field of current traffic safety. Wearable devices can monitor information regarding the driver’s state in real time, which is then analyzed by a data analysis model to provide an estimation of vigilance. The accuracy of the data analysis model directly affects the effect of vigilance estimation. In this paper, we propose a deep coupling recurrent auto-encoder (DCRA) that combines electroencephalography (EEG) and electrooculography (EOG). This model uses a coupling layer to connect two single-modal auto-encoders to construct a joint objective loss function optimization model, which consists of single-modal loss and multi-modal loss. The single-modal loss is measured by Euclidean distance, and the multi-modal loss is measured by a Mahalanobis distance of metric learning, which can effectively reflect the distance between different modal data so that the distance between different modes can be described more accurately in the new feature space based on the metric matrix. In order to ensure gradient stability in the long sequence learning process, a multi-layer gated recurrent unit (GRU) auto-encoder model was adopted. The DCRA integrates data feature extraction and feature fusion. Relevant comparative experiments show that the DCRA is better than the single-modal method and the latest multi-modal fusion. The DCRA has a lower root mean square error (RMSE) and a higher Pearson correlation coefficient (PCC).

Capsule Attention for Multimodal EEG-EOG Representation Learning With Application to Driver Vigilance Estimation

Driver vigilance estimation is an important task for transportation safety. Wearable and portable brain-computer interface devices provide a powerful means for real-time monitoring of the vigilance level of drivers to help with avoiding distracted or impaired driving. In this paper, we propose a novel multimodal architecture for in-vehicle vigilance estimation from Electroencephalogram and Electrooculogram. To enable the system to focus on the most salient parts of the learned multimodal representations, we propose an architecture composed of a capsule attention mechanism following a deep Long Short-Term Memory (LSTM) network. Our model learns hierarchical dependencies in the data through the LSTM and capsule feature representation layers. To better explore the discriminative ability of the learned representations, we study the effect of the proposed capsule attention mechanism including the number of dynamic routing iterations as well as other parameters. Experiments show the robustness of our method by outperforming other solutions and baseline techniques, setting a new state-of-the-art. We then provide an analysis on different frequency bands and brain regions to evaluate their suitability for driver vigilance estimation. Lastly, an analysis on the role of capsule attention, multimodality, and robustness to noise is performed, highlighting the advantages of our approach.