Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals

Ultralow-cost piezoelectric sensor constructed by thermal compression bonding for long-term biomechanical signal monitoring in chronic mental disorders

Wearable bioelectronic devices, which circumvent issues related to the large size and high cost of clinical equipment, have emerged as powerful tools for the auxiliary diagnosis and long-term monitoring of chronic psychiatric diseases. Current devices often integrate multiple intricate and expensive devices to ensure accurate diagnosis. However, their high cost and complexity hinder widespread clinical application and long-term user compliance. Herein, we developed an ultralow-cost poly(vinylidene fluoride)/zinc oxide nanofiber film-based piezoelectric sensor in a thermal compression bonding process. Our piezoelectric sensor exhibits remarkable sensitivity (13.4 mV N-1), rapid response (8 ms), and exceptional stability over 2000 compression/release cycles, all at a negligibly low fabrication cost. We demonstrate that pulse wave, blink, and speech signals can be acquired by the sensor, proposing a single biomechanical modality to monitor multiple physiological traits associated with bipolar disorder. This ultralow-cost and mass-producible piezoelectric sensor paves the way for extensive long-term monitoring and immediate feedback for bipolar disorder management.

A System of Emotion Recognition and Judgment and Its Application in Adaptive Interactive Game

A system of emotion recognition and judgment (SERJ) based on a set of optimal signal features is established, and an emotion adaptive interactive game (EAIG) is designed. The change in a player's emotion can be detected with the SERJ during the process of playing the game. A total of 10 subjects were selected to test the EAIG and SERJ. The results show that the SERJ and designed EAIG are effective. The game adapted itself by judging the corresponding special events triggered by a player's emotion and, as a result, enhanced the player's game experience. It was found that, in the process of playing the game, a player's perception of the change in emotion was different, and the test experience of a player had an effect on the test results. A SERJ that is based on a set of optimal signal features is better than a SERJ that is based on the conventional machine learning-based method.

Machine Learning Techniques for Arousal Classification from Electrodermal Activity: A Systematic Review

This article introduces a systematic review on arousal classification based on electrodermal activity (EDA) and machine learning (ML). From a first set of 284 articles searched for in six scientific databases, fifty-nine were finally selected according to various criteria established. The systematic review has made it possible to analyse all the steps to which the EDA signals are subjected: acquisition, pre-processing, processing and feature extraction. Finally, all ML techniques applied to the features of these signals for arousal classification have been studied. It has been found that support vector machines and artificial neural networks stand out within the supervised learning methods given their high-performance values. In contrast, it has been shown that unsupervised learning is not present in the detection of arousal through EDA. This systematic review concludes that the use of EDA for the detection of arousal is widely spread, with particularly good results in classification with the ML methods found.

A Spherical Phase Space Partitioning Based Symbolic Time Series Analysis (SPSP—STSA) for Emotion Recognition Using EEG Signals

Emotion recognition systems have been of interest to researchers for a long time. Improvement of brain-computer interface systems currently makes EEG-based emotion recognition more attractive. These systems try to develop strategies that are capable of recognizing emotions automatically. There are many approaches due to different features extractions methods for analyzing the EEG signals. Still, Since the brain is supposed to be a nonlinear dynamic system, it seems a nonlinear dynamic analysis tool may yield more convenient results. A novel approach in Symbolic Time Series Analysis (STSA) for signal phase space partitioning and symbol sequence generating is introduced in this study. Symbolic sequences have been produced by means of spherical partitioning of phase space; then, they have been compared and classified based on the maximum value of a similarity index. Obtaining the automatic independent emotion recognition EEG-based system has always been discussed because of the subject-dependent content of emotion. Here we introduce a subject-independent protocol to solve the generalization problem. To prove our method’s effectiveness, we used the DEAP dataset, and we reached an accuracy of 98.44% for classifying happiness from sadness (two- emotion groups). It was 93.75% for three (happiness, sadness, and joy), 89.06% for four (happiness, sadness, joy, and terrible), and 85% for five emotional groups (happiness, sadness, joy, terrible and mellow). According to these results, it is evident that our subject-independent method is more accurate rather than many other methods in different studies. In addition, a subject-independent method has been proposed in this study, which is not considered in most of the studies in this field.

Model of Emotion Judgment Based on Features of Multiple Physiological Signals

The model of emotion judgment based on features of multiple physiological signals was investi-gated. In total, 40 volunteers participated in the experiment by playing a computer game while their physiological signals (skin electricity, electrocardiogram (ECG), pulse wave, and facial electromy-ogram (EMG)) were acquired. The volunteers were asked to complete an emotion questionnaire where six typical events that appeared in the game were included, and each volunteer rated their own emotion when experiencing the six events. Based on the analysis of game events, the signal data were cut into segments and the emotional trends were classified. The correlation between data segments and emotional trends was built using a statistical method combined with the questionnaire responses. The set of optimal signal features was obtained by processing the data of physiological signals, extracting the features of signal data, reducing the dimensionality of signal features, and classifying the emotion based on the set of signal data. Finally, the model of emotion judgment was established by selecting the features with a significance of 0.01 based on the correlation between the features in the set of optimal signal features and emotional trends.

Emotion Recognition and Regulation Based on Stacked Sparse Auto-Encoder Network and Personalized Reconfigurable Music

Music can regulate and improve the emotions of the brain. Traditional emotional regulation approaches often adopt complete music. As is well-known, complete music may vary in pitch, volume, and other ups and downs. An individual’s emotions may also adopt multiple states, and music preference varies from person to person. Therefore, traditional music regulation methods have problems, such as long duration, variable emotional states, and poor adaptability. In view of these problems, we use different music processing methods and stacked sparse auto-encoder neural networks to identify and regulate the emotional state of the brain in this paper. We construct a multi-channel EEG sensor network, divide brainwave signals and the corresponding music separately, and build a personalized reconfigurable music-EEG library. The 17 features in the EEG signal are extracted as joint features, and the stacked sparse auto-encoder neural network is used to classify the emotions, in order to establish a music emotion evaluation index. According to the goal of emotional regulation, music fragments are selected from the personalized reconfigurable music-EEG library, then reconstructed and combined for emotional adjustment. The results show that, compared with complete music, the reconfigurable combined music was less time-consuming for emotional regulation (76.29% less), and the number of irrelevant emotional states was reduced by 69.92%. In terms of adaptability to different participants, the reconfigurable music improved the recognition rate of emotional states by 31.32%.

Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data

To further extend the applicability of wearable sensors in various domains such as mobile health systems and the automotive industry, new methods for accurately extracting subtle physiological information from these wearable sensors are required. However, the extraction of valuable information from physiological signals is still challenging—smartphones can count steps and compute heart rate, but they cannot recognize emotions and related affective states. This study analyzes the possibility of using end-to-end multimodal deep learning (DL) methods for affect recognition. Ten end-to-end DL architectures are compared on four different datasets with diverse raw physiological signals used for affect recognition, including emotional and stress states. The DL architectures specialized for time-series classification were enhanced to simultaneously facilitate learning from multiple sensors, each having their own sampling frequency. To enable fair comparison among the different DL architectures, Bayesian optimization was used for hyperparameter tuning. The experimental results showed that the performance of the models depends on the intensity of the physiological response induced by the affective stimuli, i.e., the DL models recognize stress induced by the Trier Social Stress Test more successfully than they recognize emotional changes induced by watching affective content, e.g., funny videos. Additionally, the results showed that the CNN-based architectures might be more suitable than LSTM-based architectures for affect recognition from physiological sensors.

Selecting transferrable neurophysiological features for inter-individual emotion recognition via a shared-subspace feature elimination approach

The interplay between human emotions, personality, and motivation results in individual specificity in neurophysiological data distributions for the same emotional category. To address this issue for building an emotion recognition system based on electroencephalogram (EEG) features, we propose a shared-subspace feature elimination (SSFE) approach to identify EEG variables with common characteristics across multiple individuals. In the SSFE framework, a low-dimensional space defined by a selected number of EEG features is created to represent the inter-emotion discriminant for different pairs of subjects evaluated based on the interclass margin. Using two public databases-DEAP and MAHNOB-HCI-the performance of the SSFE is validated according to the leave-one-subject-out paradigm. The performance of the proposed framework is compared with five other feature-selection methods. The effectiveness and computational cost of the SSFE is investigated across six machine learning models based on their optimal hyperparameters. In the end, the competitive binary classification accuracy from the SSFE of arousal and valence recognitions are determined to be 0.6521 and 0.6635, respectively, for DEAP, and 0.6520 and 0.6537, respectively for MAHNOB-HCI.

Systematic Analysis of a Military Wearable Device Based on a Multi-Level Fusion Framework: Research Directions

With the development of the Internet of Battlefield Things (IoBT), soldiers have become key nodes of information collection and resource control on the battlefield. It has become a trend to develop wearable devices with diverse functions for the military. However, although densely deployed wearable sensors provide a platform for comprehensively monitoring the status of soldiers, wearable technology based on multi-source fusion lacks a generalized research system to highlight the advantages of heterogeneous sensor networks and information fusion. Therefore, this paper proposes a multi-level fusion framework (MLFF) based on Body Sensor Networks (BSNs) of soldiers, and describes a model of the deployment of heterogeneous sensor networks. The proposed framework covers multiple types of information at a single node, including behaviors, physiology, emotions, fatigue, environments, and locations, so as to enable Soldier-BSNs to obtain sufficient evidence, decision-making ability, and information resilience under resource constraints. In addition, we systematically discuss the problems and solutions of each unit according to the frame structure to identify research directions for the development of wearable devices for the military.

Feature Extraction and Selection for Pain Recognition Using Peripheral Physiological Signals

In pattern recognition, the selection of appropriate features is paramount to both the performance and the robustness of the system. Over-reliance on machine learning-based feature selection methods can, therefore, be problematic; especially when conducted using small snapshots of data. The results of these methods, if adopted without proper interpretation, can lead to sub-optimal system design or worse, the abandonment of otherwise viable and important features. In this work, a deep exploration of pain-based emotion classification was conducted to better understand differences in the results of the related literature. In total, 155 different time domain and frequency domain features were explored, derived from electromyogram (EMG), skin conductance levels (SCL), and electrocardiogram (ECG) readings taken from the 85 subjects in response to heat-induced pain. To address the inconsistency in the optimal feature sets found in related works, an exhaustive and interpretable feature selection protocol was followed to obtain a generalizable feature set. Associations between features were then visualized using a topologically-informed chart, called Mapper, of this physiological feature space, including synthesis and comparison of results from previous literature. This topological feature chart was able to identify key sources of information that led to the formation of five main functional feature groups: signal amplitude and power, frequency information, nonlinear complexity, unique, and connecting. These functional groupings were used to extract further insight into observable autonomic responses to pain through a complementary statistical interaction analysis. From this chart, it was observed that EMG and SCL derived features could functionally replace those obtained from ECG. These insights motivate future work on novel sensing modalities, feature design, deep learning approaches, and dimensionality reduction techniques.

Recognition of Emotions Using Multichannel EEG Data and DBN-GC-Based Ensemble Deep Learning Framework

Fusing multichannel neurophysiological signals to recognize human emotion states becomes increasingly attractive. The conventional methods ignore the complementarity between time domain characteristics, frequency domain characteristics, and time-frequency characteristics of electroencephalogram (EEG) signals and cannot fully capture the correlation information between different channels. In this paper, an integrated deep learning framework based on improved deep belief networks with glia chains (DBN-GCs) is proposed. In the framework, the member DBN-GCs are employed for extracting intermediate representations of EEG raw features from multiple domains separately, as well as mining interchannel correlation information by glia chains. Then, the higher level features describing time domain characteristics, frequency domain characteristics, and time-frequency characteristics are fused by a discriminative restricted Boltzmann machine (RBM) to implement emotion recognition task. Experiments conducted on the DEAP benchmarking dataset achieve averaged accuracy of 75.92% and 76.83% for arousal and valence states classification, respectively. The results show that the proposed framework outperforms most of the above deep classifiers. Thus, potential of the proposed framework is demonstrated.

EEG-Based Emotion Recognition Using Quadratic Time-Frequency Distribution

Accurate recognition and understating of human emotions is an essential skill that can improve the collaboration between humans and machines. In this vein, electroencephalogram (EEG)-based emotion recognition is considered an active research field with challenging issues regarding the analyses of the nonstationary EEG signals and the extraction of salient features that can be used to achieve accurate emotion recognition. In this paper, an EEG-based emotion recognition approach with a novel time-frequency feature extraction technique is presented. In particular, a quadratic time-frequency distribution (QTFD) is employed to construct a high resolution time-frequency representation of the EEG signals and capture the spectral variations of the EEG signals over time. To reduce the dimensionality of the constructed QTFD-based representation, a set of 13 time- and frequency-domain features is extended to the joint time-frequency-domain and employed to quantify the QTFD-based time-frequency representation of the EEG signals. Moreover, to describe different emotion classes, we have utilized the 2D arousal-valence plane to develop four emotion labeling schemes of the EEG signals, such that each emotion labeling scheme defines a set of emotion classes. The extracted time-frequency features are used to construct a set of subject-specific support vector machine classifiers to classify the EEG signals of each subject into the different emotion classes that are defined using each of the four emotion labeling schemes. The performance of the proposed approach is evaluated using a publicly available EEG dataset, namely the DEAPdataset. Moreover, we design three performance evaluation analyses, namely the channel-based analysis, feature-based analysis and neutral class exclusion analysis, to quantify the effects of utilizing different groups of EEG channels that cover various regions in the brain, reducing the dimensionality of the extracted time-frequency features and excluding the EEG signals that correspond to the neutral class, on the capability of the proposed approach to discriminate between different emotion classes. The results reported in the current study demonstrate the efficacy of the proposed QTFD-based approach in recognizing different emotion classes. In particular, the average classification accuracies obtained in differentiating between the various emotion classes defined using each of the four emotion labeling schemes are within the range of 73.8%–86.2%. Moreover, the emotion classification accuracies achieved by our proposed approach are higher than the results reported in several existing state-of-the-art EEG-based emotion recognition studies.

Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals

Emotion recognition is an important pattern recognition problem that has inspired researchers for several areas. Various data from humans for emotion recognition have been developed, including visual, audio, and physiological signals data. This paper proposes a decision-level weight fusion strategy for emotion recognition in multichannel physiological signals. Firstly, we selected four kinds of physiological signals, including Electroencephalography (EEG), Electrocardiogram (ECG), Respiration Amplitude (RA), and Galvanic Skin Response (GSR). And various analysis domains have been used in physiological emotion features extraction. Secondly, we adopt feedback strategy for weight definition, according to recognition rate of each emotion of each physiological signal based on Support Vector Machine (SVM) classifier independently. Finally, we introduce weight in decision level by linear fusing weight matrix with classification result of each SVM classifier. The experiments on the MAHNOB-HCI database show the highest accuracy. The results also provide evidence and suggest a way for further developing a more specialized emotion recognition system based on multichannel data using weight fusion strategy.

Coverage of Emotion Recognition for Common Wearable Biosensors

The present research proposes a novel emotion recognition framework for the computer prediction of human emotions using common wearable biosensors. Emotional perception promotes specific patterns of biological responses in the human body, and this can be sensed and used to predict emotions using only biomedical measurements. Based on theoretical and empirical psychophysiological research, the foundation of autonomic specificity facilitates the establishment of a strong background for recognising human emotions using machine learning on physiological patterning. However, a systematic way of choosing the physiological data covering the elicited emotional responses for recognising the target emotions is not obvious. The current study demonstrates through experimental measurements the coverage of emotion recognition using common off-the-shelf wearable biosensors based on the synchronisation between audiovisual stimuli and the corresponding physiological responses. The work forms the basis of validating the hypothesis for emotional state recognition in the literature and presents coverage of the use of common wearable biosensors coupled with a novel preprocessing algorithm to demonstrate the practical prediction of the emotional states of wearers.

Fusion of heart rate variability and pulse rate variability for emotion recognition using lagged poincare plots

Designing an efficient automatic emotion recognition system based on physiological signals has attracted great interests within the research of human-machine interactions. This study was aimed to classify emotional responses by means of a simple dynamic signal processing technique and fusion frameworks. The electrocardiogram and finger pulse activity of 35 participants were recorded during rest condition and when subjects were listening to music intended to stimulate certain emotions. Four emotion categories, including happiness, sadness, peacefulness, and fear were chosen. Estimating heart rate variability (HRV) and pulse rate variability (PRV), 4 Poincare indices in 10 lags were extracted. The support vector machine classifier was used for emotion classification. Both feature level (FL) and decision level (DL) fusion schemes were examined. Significant differences have been observed between lag 1 Poincare plot indices and the other lagged measures. The mean accuracies of 84.1, 82.9, 79.68, and 76.05% were obtained for PRV, DL, FL, and HRV measures, respectively. However, DL outperformed others in discriminating sadness and peacefulness, using SD₁ and total features, correspondingly. In both cases, the classification rates improved up to 92% (with the sensitivity of 95% and specificity of 83.33%). Totally, DL resulted in better performances compared to FL. In addition, the impact of the fusion rules on the classification performances has been confirmed.

Cross-Subject EEG Feature Selection for Emotion Recognition Using Transfer Recursive Feature Elimination

Using machine-learning methodologies to analyze EEG signals becomes increasingly attractive for recognizing human emotions because of the objectivity of physiological data and the capability of the learning principles on modeling emotion classifiers from heterogeneous features. However, the conventional subject-specific classifiers may induce additional burdens to each subject for preparing multiple-session EEG data as training sets. To this end, we developed a new EEG feature selection approach, transfer recursive feature elimination (T-RFE), to determine a set of the most robust EEG indicators with stable geometrical distribution across a group of training subjects and a specific testing subject. A validating set is introduced to independently determine the optimal hyper-parameter and the feature ranking of the T-RFE model aiming at controlling the overfitting. The effectiveness of the T-RFE algorithm for such cross-subject emotion classification paradigm has been validated by DEAP database. With a linear least square support vector machine classifier implemented, the performance of the T-RFE is compared against several conventional feature selection schemes and the statistical significant improvement has been found. The classification rate and F-score achieve 0.7867, 0.7526, 0.7875, and 0.8077 for arousal and valence dimensions, respectively, and outperform several recent reported works on the same database. In the end, the T-RFE based classifier is compared against two subject-generic classifiers in the literature. The investigation of the computational time for all classifiers indicates the accuracy improvement of the T-RFE is at the cost of the longer training time.

Recognition of emotions using multimodal physiological signals and an ensemble deep learning model

BACKGROUND AND OBJECTIVE

Using deep-learning methodologies to analyze multimodal physiological signals becomes increasingly attractive for recognizing human emotions. However, the conventional deep emotion classifiers may suffer from the drawback of the lack of the expertise for determining model structure and the oversimplification of combining multimodal feature abstractions.

METHODS

In this study, a multiple-fusion-layer based ensemble classifier of stacked autoencoder (MESAE) is proposed for recognizing emotions, in which the deep structure is identified based on a physiological-data-driven approach. Each SAE consists of three hidden layers to filter the unwanted noise in the physiological features and derives the stable feature representations. An additional deep model is used to achieve the SAE ensembles. The physiological features are split into several subsets according to different feature extraction approaches with each subset separately encoded by a SAE. The derived SAE abstractions are combined according to the physiological modality to create six sets of encodings, which are then fed to a three-layer, adjacent-graph-based network for feature fusion. The fused features are used to recognize binary arousal or valence states.

RESULTS

DEAP multimodal database was employed to validate the performance of the MESAE. By comparing with the best existing emotion classifier, the mean of classification rate and F-score improves by 5.26%.

CONCLUSIONS

The superiority of the MESAE against the state-of-the-art shallow and deep emotion classifiers has been demonstrated under different sizes of the available physiological instances.

Indices from lagged poincare plots of heart rate variability: an efficient nonlinear tool for emotion discrimination

Interest in human emotion recognition, regarding physiological signals, has recently risen. In this study, an efficient emotion recognition system, based on geometrical analysis of autonomic nervous system signals, is presented. The electrocardiogram recordings of 47 college students were obtained during rest condition and affective visual stimuli. Pictures with four emotional contents, including happiness, peacefulness, sadness, and fear were selected. Then, ten lags of Poincare plot were constructed for heart rate variability (HRV) segments. For each lag, five geometrical indices were extracted. Next, these features were fed into an automatic classification system for the recognition of the four affective states and rest condition. The results showed that the Poincare plots have different shapes for different lags, as well as for different affective states. Considering higher lags, the greatest increment in SD₁ and decrements in SD₂ occurred during the happiness stimuli. In contrast, the minimum changes in the Poincare measures were perceived during the fear inducements. Therefore, the HRV geometrical shapes and dynamics were altered by the positive and negative values of valence-based emotion dimension. Using a probabilistic neural network, a maximum recognition rate of 97.45% was attained. Applying the proposed methodology based on lagged Poincare indices, a valuable tool for discriminating the emotional states was provided.

GENDER DIFFERENCES IN RESPONSE TO AFFECTIVE AUDIO AND VISUAL INDUCTIONS: EXAMINATION OF NONLINEAR DYNAMICS OF AUTONOMIC SIGNALS

Physiological reflection of emotions can be tracked by autonomic signals. Several studies have been conducted using autonomic signal processing to examine men and women differences during the exposure of affective stimuli. Emotional pictures and music are two commonly used methods to induce affects in an experimental setup. The biological changes have been commonly monitored during a certain emotional inducement protocol, solely. This study was aimed to examine two induction paradigms involved auditory and visual cues using nonlinear dynamical approaches. To this end, various nonlinear parameters of galvanic skin response (GSR) and pulse signals of men and women were examined. The nonlinear analysis was performed using lagged Poincare parameters, detrended fluctuation indices (DFAs), Lyapunov exponents (LEs), some entropy measures, and recurrence quantification analysis (RQA). The Wilcoxon rank sum test was used to show significant differences between the groups. The results indicate that besides the type of affect induction, physiological differences of men and women are notable in negative emotions (sadness and fear). Regardless to the inducements, lagged Poincare parameters of the pulse signals and DFA indices of the GSR have shown significant differences in gender affective responses. However, applying pictorial stimuli, LEs are appropriate indicators for gender discrimination. It is also concluded that GSR dynamics are intensely affected by the kind of stimuli; while this is not validated for the pulse. These findings suggest that different emotional inductions evoked different autonomic responses in men and women, which can be appropriately monitored using nonlinear signal processing approaches.