Car Driver's Sympathetic Reaction Detection Through Electrodermal Activity and Electrocardiogram Measurements

Instantaneous estimation of momentary affective responses using neurophysiological signals and a spatiotemporal emotional intensity regression network

Previous studies in affective computing often use a fixed emotional label to train an emotion classifier with electroencephalography (EEG) from individuals experiencing an affective stimulus. However, EEGs encode emotional dynamics that include varying intensities within a given emotional category. To investigate these variations in emotional intensity, we propose a framework that obtains momentary affective labels for fine-grained segments of EEGs with human feedback. We then model these labeled segments using a novel spatiotemporal emotional intensity regression network (STEIR-Net). It integrates temporal EEG patterns from nine predefined cortical regions to provide a continuous estimation of emotional intensity. We demonstrate that the STEIR-Net outperforms classical regression models by reducing the root mean square error (RMSE) by an average of 4∼9 % and 2∼4 % for the SEED and SEED-IV databases, respectively. We find that the frontal and temporal cortical regions contribute significantly to the affective intensity's variation. Higher absolute values of the Spearman correlation coefficient between the model estimation and momentary affective labels under happiness (0.2114) and fear (0.2072) compared to neutral (0.1694) and sad (0.1895) emotions were observed. Besides, increasing the input length of the EEG segments from 4 to 20 s further reduces the RMSE from 1.3548 to 1.3188.

Real-World Driver Stress Recognition and Diagnosis Based on Multimodal Deep Learning and Fuzzy EDAS Approaches

Mental stress is known as a prime factor in road crashes. The devastation of these crashes often results in damage to humans, vehicles, and infrastructure. Likewise, persistent mental stress could lead to the development of mental, cardiovascular, and abdominal disorders. Preceding research in this domain mostly focuses on feature engineering and conventional machine learning approaches. These approaches recognize different levels of stress based on handcrafted features extracted from various modalities including physiological, physical, and contextual data. Acquiring good quality features from these modalities using feature engineering is often a difficult job. Recent developments in the form of deep learning (DL) algorithms have relieved feature engineering by automatically extracting and learning resilient features. This paper proposes different CNN and CNN-LSTSM-based fusion models using physiological signals (SRAD dataset) and multimodal data (AffectiveROAD dataset) for the driver's two and three stress levels. The fuzzy EDAS (evaluation based on distance from average solution) approach is used to evaluate the performance of the proposed models based on different classification metrics (accuracy, recall, precision, F-score, and specificity). Fuzzy EDAS performance estimation shows that the proposed CNN and hybrid CNN-LSTM models achieved the first ranks based on the fusion of BH, E4-Left (E4-L), and E4-Right (E4-R). Results showed the significance of multimodal data for designing an accurate and trustworthy stress recognition diagnosing model for real-world driving conditions. The proposed model can also be used for the diagnosis of the stress level of a subject during other daily life activities.

Machine Learning Techniques for Arousal Classification from Electrodermal Activity: A Systematic Review

This article introduces a systematic review on arousal classification based on electrodermal activity (EDA) and machine learning (ML). From a first set of 284 articles searched for in six scientific databases, fifty-nine were finally selected according to various criteria established. The systematic review has made it possible to analyse all the steps to which the EDA signals are subjected: acquisition, pre-processing, processing and feature extraction. Finally, all ML techniques applied to the features of these signals for arousal classification have been studied. It has been found that support vector machines and artificial neural networks stand out within the supervised learning methods given their high-performance values. In contrast, it has been shown that unsupervised learning is not present in the detection of arousal through EDA. This systematic review concludes that the use of EDA for the detection of arousal is widely spread, with particularly good results in classification with the ML methods found.

Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics

With the development of the autopilot system, the main task of a pilot has changed from controlling the aircraft to supervising the autopilot system and making critical decisions. Therefore, the human–machine interaction system needs to be improved accordingly. A key step to improving the human–machine interaction system is to improve its understanding of the pilots’ status, including fatigue, stress, workload, etc. Monitoring pilots’ status can effectively prevent human error and achieve optimal human–machine collaboration. As such, there is a need to recognize pilots’ status and predict the behaviors responsible for changes of state. For this purpose, in this study, 14 Air Force cadets fly in an F-35 Lightning II Joint Strike Fighter simulator through a series of maneuvers involving takeoff, level flight, turn and hover, roll, somersault, and stall. Electro cardio (ECG), myoelectricity (EMG), galvanic skin response (GSR), respiration (RESP), and skin temperature (SKT) measurements are derived through wearable physiological data collection devices. Physiological indicators influenced by the pilot’s behavioral status are objectively analyzed. Multi-modality fusion technology (MTF) is adopted to fuse these data in the feature layer. Additionally, four classifiers are integrated to identify pilots’ behaviors in the strategy layer. The results indicate that MTF can help to recognize pilot behavior in a more comprehensive and precise way.

Analysis of Physiological Signals for Stress Recognition with Different Car Handling Setups

When designing a car, the vehicle dynamics and handling are important aspects, as they can satisfy a purpose in professional racing, as well as contributing to driving pleasure and safety, real and perceived, in regular drivers. In this paper, we focus on the assessment of the emotional response in drivers while they are driving on a track with different car handling setups. The experiments were performed using a dynamic professional simulator prearranged with different car setups. We recorded various physiological signals, allowing us to analyze the response of the drivers and analyze which car setup is more influential in terms of stress arising in the subjects. We logged two skin potential responses (SPRs), the electrocardiogram (ECG) signal, and eye tracking information. In the experiments, three car setups were used (neutral, understeering, and oversteering). To evaluate how these affect the drivers, we analyzed their physiological signals using two statistical tests (t-test and Wilcoxon test) and various machine learning (ML) algorithms. The results of the Wilcoxon test show that SPR signals provide higher statistical significance when evaluating stress among different drivers, compared to the ECG and eye tracking signals. As for the ML classifiers, we count the number of positive or “stress” labels of 15 s SPR time intervals for each subject and each particular car setup. With the support vector machine classifier, the mean value of the number of positive labels for the four subjects is equal to 13.13% for the base setup, 44.16% for the oversteering setup, and 39.60% for the understeering setup. In the end, our findings show that the base car setup appears to be the least stressful, and that our system enables us to effectively recognize stress while the subjects are driving in the different car configurations.

Exploring Physiological Signal Responses to Traffic-Related Stress in Simulated Driving

In this paper, we propose a relatively noninvasive system that can automatically assess the impact of traffic conditions on drivers. We analyze the physiological signals recorded from a set of individuals while driving in a simulated urban scenario in two different traffic scenarios, i.e., with traffic and without traffic. The experiments were carried out in a laboratory located at the University of Udine, employing a driving simulator equipped with a moving platform. We acquired two Skin Potential Response (SPR) signals from the hands of the drivers, and an electrocardiogram (ECG) signal from their chest. In the proposed scheme, the SPR signals are then processed through a Motion Artifact (MA) removal algorithm such that possible motion artifacts arising during the drive are reduced. An analysis considering the scalogram of the single cleaned SPR signal is presented. This signal, along with the ECG, is then fed to various Machine Learning (ML) algorithms. More specifically, some statistical features are extracted from each signal segment which, after being analyzed through a binary ML model, are labeled as corresponding to a stressful situation or not. Our results confirm the applicability of the proposed approach to identify stress in the two scenarios. This is also in accordance with our findings considering the SPR signal scalograms.

Classification of Mental Stress Using CNN-LSTM Algorithms with Electrocardiogram Signals

The mental stress faced by many people in modern society is a factor that causes various chronic diseases, such as depression, cancer, and cardiovascular disease, according to stress accumulation. Therefore, it is very important to regularly manage and monitor a person's stress. In this study, we propose an ensemble algorithm that can accurately determine mental stress states using a modified convolutional neural network (CNN)- long short-term memory (LSTM) architecture. When a person is exposed to stress, a displacement occurs in the electrocardiogram (ECG) signal. It is possible to classify stress signals by analyzing ECG signals and extracting specific parameters. To maximize the performance of the proposed stress classification algorithm, fast Fourier transform (FFT) and spectrograms were applied to preprocess ECG signals and produce signals in both the time and frequency domains to aid the training process. As the performance evaluation benchmarks of the stress classification model, confusion matrices, receiver operating characteristic (ROC) curves, and precision-recall (PR) curves were used, and the accuracy achieved by the proposed model was 98.3%, which is an improvement of 14.7% compared to previous research results. Therefore, our model can help manage the mental health of people exposed to stress. In addition, if combined with various biosignals such as electromyogram (EMG) and photoplethysmography (PPG), it may have the potential for development in various healthcare systems, such as home training, sleep state analysis, and cardiovascular monitoring.

Stress Evaluation in Simulated Autonomous and Manual Driving through the Analysis of Skin Potential Response and Electrocardiogram Signals

The evaluation of car drivers' stress condition is gaining interest as research on Autonomous Driving Systems (ADS) progresses. The analysis of the stress response can be used to assess the acceptability of ADS and to compare the driving styles of different autonomous drive algorithms. In this contribution, we present a system based on the analysis of the Electrodermal Activity Skin Potential Response (SPR) signal, aimed to reveal the driver's stress induced by different driving situations. We reduce motion artifacts by processing two SPR signals, recorded from the hands of the subjects, and outputting a single clean SPR signal. Statistical features of signal blocks are sent to a Supervised Learning Algorithm, which classifies between stress and normal driving (non-stress) conditions. We present the results obtained from an experiment using a professional driving simulator, where a group of people is asked to undergo manual and autonomous driving on a highway, facing some unexpected events meant to generate stress. The results of our experiment show that the subjects generally appear more stressed during manual driving, indicating that the autonomous drive can possibly be well received by the public. During autonomous driving, however, significant peaks of the SPR signal are evident during unexpected events. By examining the electrocardiogram signal, the average heart rate is generally higher in the manual case compared to the autonomous case. This further supports our previous findings, even if it may be due, in part, to the physical activity involved in manual driving.