Fusion of heart rate variability and pulse rate variability for emotion recognition using lagged poincare plots

PPG2EMG: Estimating Upper-Arm Muscle Activities and EMG from Wrist PPG Values

The electromyogram (EMG) is a waveform representation of the action potential generated by muscle cells using electrodes. EMG acquired using surface electrodes is called surface EMG (sEMG), and it is the acquisition of muscle action potentials transmitted by volume conduction from the skin. Surface electrodes require disposable conductive gel or adhesive tape to be attached to the skin, which is costly to run, and the tape is hard on the skin when it is removed. Muscle activity can be evaluated by acquiring muscle potentials and analyzing quantitative, temporal, and frequency factors. It is also possible to evaluate muscle fatigue because the frequency of the EMG becomes lower as the muscle becomes fatigued. Research on human activity recognition from EMG signals has been actively conducted and applied to systems that support arm and hand functions. This paper proposes a method for recognizing the muscle activity state of the arm using pulse wave data (PPG: Photoplethysmography) and a method for estimating EMG using pulse wave data. This paper assumes that the PPG sensor is worn on the user's wrist to measure the heart rate. The user also attaches an elastic band to the upper arm, and when the user exerts a force on the arm, the muscles of the upper arm contract. The arteries are then constricted, and the pulse wave measured at the wrist becomes weak. From the change in the pulse wave, the muscle activity of the arm can be recognized and the number of action potentials of the muscle can be estimated. From the evaluation experiment with five subjects, three types of muscle activity were recognized with 80+%, and EMG was estimated with approximately 20% error rate.

A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

Multimodal human–computer interaction (HCI) systems pledge a more human–human-like interaction between machines and humans. Their prowess in emanating an unambiguous information exchange between the two makes these systems more reliable, efficient, less error prone, and capable of solving complex tasks. Emotion recognition is a realm of HCI that follows multimodality to achieve accurate and natural results. The prodigious use of affective identification in e-learning, marketing, security, health sciences, etc., has increased demand for high-precision emotion recognition systems. Machine learning (ML) is getting its feet wet to ameliorate the process by tweaking the architectures or wielding high-quality databases (DB). This paper presents a survey of such DBs that are being used to develop multimodal emotion recognition (MER) systems. The survey illustrates the DBs that contain multi-channel data, such as facial expressions, speech, physiological signals, body movements, gestures, and lexical features. Few unimodal DBs are also discussed that work in conjunction with other DBs for affect recognition. Further, VIRI, a new DB of visible and infrared (IR) images of subjects expressing five emotions in an uncontrolled, real-world environment, is presented. A rationale for the superiority of the presented corpus over the existing ones is instituted.

Innovative Poincare's plot asymmetry descriptors for EEG emotion recognition

Given the importance of emotion recognition in both medical and non-medical applications, designing an automatic system has captured the attention of several scholars. Currently, EEG-based emotion recognition has a special position, which has not fulfilled the desired accuracy rates yet. This experiment intended to provide novel EEG asymmetry measures to improve emotion recognition rates. Four emotional states have been classified using the k-nearest neighbor (kNN), support vector machine, and Naïve Bayes. Feature selection has been performed, and the role of employing a different number of top-ranked features on emotion recognition rates has been assessed. To validate the efficiency of the proposed scheme, two public databases, including the SJTU Emotion EEG Dataset-IV (SEED-IV) and a Database for Emotion Analysis using Physiological signals (DEAP) were evaluated. The experimental results indicated that kNN outperformed the other classifiers with a maximum accuracy of 95.49 and 98.63% using SEED-IV and DEAP datasets, respectively. In conclusion, the results of the proposed novel EEG-asymmetry measures make the framework a superior one compared to the state-of-art EEG emotion recognition approaches.

Verhulst map measures: new biomarkers for heart rate classification

Recording, monitoring, and analyzing biological signals has received significant attention in medicine. A fundamental phase for understanding a bio-system under various conditions is to process the corresponding bio-signal appropriately. To this effect, different conventional and nonlinear approaches have been proposed. However, since the non-stationary properties of the bio-signals are not revealed by traditional linear methods, nonlinear dynamical techniques play a crucial role in examining the behavior of a bio-system. This work proposes new bio-markers based on the chaotic nature of the biomedical signals. These measures were introduced using the Verhulst map, a simple tool for characterizing the morphology of the reconstructed phase space. For this purpose, we extracted the features from the heart rate (HR) signals of six groups of meditators and non-meditators. For a typical classification problem, the performance of some conventional classifiers, including the k-nearest neighbor, support vector machine, and Naïve Bayes, was appraised separately. In addition, the competence of a hybrid classification strategy was inspected using majority voting. The results indicated a maximum accuracy, F1-score, and sensitivity of 100%. These findings reveal that the proposed framework is eminently capable of analyzing and classifying the HR signals of the groups. In conclusion, the Verhulst diagram-based measures are simple and based on the dynamics of the bio-signals, which can be served for quantifying different signals in medical systems.

Human Emotion Recognition using Polar-Based Lagged Poincare Plot Indices of Eye-Blinking Data

Emotion recognition using bio-signals is currently a hot and challenging topic in human–computer interferences, robotics, and affective computing. A broad range of literature has been published by analyzing the internal/external behaviors of the subjects in confronting emotional events/stimuli. Eye movements, as an external behavior, are frequently used in the multi-modal emotion recognition system. On the other hand, classic statistical features of the signal have generally been assessed, and the evaluation of its dynamics has been neglected so far. For the first time, the dynamics of single-modal eye-blinking data are characterized. Novel polar-based indices of the lagged Poincare plots were introduced. The optimum lag was estimated using mutual information. After reconstruction of the plot, the polar measures of all points were characterized using statistical measures. The support vector machine (SVM), decision tree, and Naïve Bayes were implemented to complete the process of classification. The highest accuracy of 100% with an average accuracy of 84.17% was achieved for fear/sad discrimination using SVM. The suggested framework provided outstanding performances in terms of recognition rates, simplicity of the methodology, and less computational cost. Our results also showed that eye-blinking data possesses the potential for emotion recognition, especially in classifying fear emotion.

Asymmetry of lagged Poincare plot in heart rate signals during meditation

Background and aim

Heart rate variability (HRV) quantifies the variability in the heart’s beat-to-beat intervals. This signal is a potential marker of cardiac function in normal, pathological, and psychological states. Signal asymmetry refers to an unequal distribution in the signal, which can be found by a two-dimensional Poincare plot. Earlier, heart rate asymmetry (HRA) was assessed using a conventional Poincare plot (lag of 1). In this study, we have investigated the effect of delay on the phase space asymmetry using lagged Poincare’s plot.

Experimental procedure

This study compared the presence/lack of asymmetries in the HRV data of 12 meditators (four Kundalini yoga (Yoga) at an advanced level of meditation, eight Chinese Chi meditators (Chi) ∼1–3 months) to 25 non-meditators (11 spontaneous nocturnal breathing (Normal) and 14 metronomic breathing (Metron)). Poincare’s plots were constructed with six different lags, and HRA was calculated. The analysis was conducted using HRV data provided in the Physionet database.

Results

The results showed that using conventional Poincare’s plot (lag of 1), the lowest HRA was observed in the Metron group. In addition, the HRA index was different between meditators and non-meditator groups. Moreover, as the most significant difference between groups was observed in a delay of 6, the role of the delay selection on the signal asymmetry was revealed.

Conclusion

The difference between lagged HRA responses on Yoga in comparison with other groups can be an emphasis on the importance of choosing the type of meditation technique and its effects on the cardiovascular system.

•

Asymmetries in HRV was assessed in different meditator and non-meditator groups.

•

The role of delay selection was explored on the phase space asymmetry using lagged Poincare plot.

•

A weaker asymmetry was observed in the metronomic breathing group.

•

The most significant difference between groups was perceived in a delay of six.

A Multimodal Facial Emotion Recognition Framework through the Fusion of Speech with Visible and Infrared Images

The exigency of emotion recognition is pushing the envelope for meticulous strategies of discerning actual emotions through the use of superior multimodal techniques. This work presents a multimodal automatic emotion recognition (AER) framework capable of differentiating between expressed emotions with high accuracy. The contribution involves implementing an ensemble-based approach for the AER through the fusion of visible images and infrared (IR) images with speech. The framework is implemented in two layers, where the first layer detects emotions using single modalities while the second layer combines the modalities and classifies emotions. Convolutional Neural Networks (CNN) have been used for feature extraction and classification. A hybrid fusion approach comprising early (feature-level) and late (decision-level) fusion, was applied to combine the features and the decisions at different stages. The output of the CNN trained with voice samples of the RAVDESS database was combined with the image classifier’s output using decision-level fusion to obtain the final decision. An accuracy of 86.36% and similar recall (0.86), precision (0.88), and f-measure (0.87) scores were obtained. A comparison with contemporary work endorsed the competitiveness of the framework with the rationale for exclusivity in attaining this accuracy in wild backgrounds and light-invariant conditions.

Schizophrenia diagnosis using innovative EEG feature-level fusion schemes

Electroencephalogram (EEG) has become a practical tool for monitoring and diagnosing pathological/psychological brain states. To date, an increasing number of investigations considered differences between brain dynamic of patients with schizophrenia and healthy controls. However, insufficient studies have been performed to provide an intelligent and accurate system that detects the schizophrenia using EEG signals. This paper concerns this issue by providing new feature-level fusion algorithms. Firstly, we analyze EEG dynamics using three well-known nonlinear measures, including complexity (Cx), Higuchi fractal dimension (HFD), and Lyapunov exponents (Lya). Next, we propose some innovative feature-level fusion strategies to combine the information of these indices. We evaluate the effect of the classifier parameter (σ) adjustment and the cross-validation partitioning criteria on classification accuracy. The performance of EEG classification using combined features was compared with the non-combined attributes. Experimental results showed higher classification accuracy when feature-level features were utilized, compared to when each feature was used individually or all fed to the classifier simultaneously. Using the proposed algorithm, the classification accuracy increased up to 100%. These results establish the suggested framework as a superior scheme compared to the state-of-the-art EEG schizophrenia diagnosis tool.

The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features

Recently, developing an accurate automatic emotion recognition system using a minimum number of bio-signals has become a challenging issue in "affective computing." This study aimed to propose a reliable system by examining nonlinear dynamics of photoplethysmogram (PPG) and galvanic skin response (GSR). To address this goal, two strategies were adopted. First, the efficiency of each signal in valence/arousal based emotion categorization was examined. Then, the proficiency of a hybrid feature, by combining both GSR and PPG features was studied. Lyapunov exponents, lagged Poincare's measures, and approximate entropy were extracted to characterize the irregularity and chaotic behavior of the phase space. To discriminate two levels of arousal and two levels of the valence, a probabilistic neural network (PNN) with different sigma adjustment parameter was examined. The results showed that the phase space geometry and consequently, the signal dynamics are influenced by the emotional music video. Additionally, distinctive patterns of the phase space behavior were observed under the influence of different lags. For both signals, the most irregularity was observed during the high valence, and the least irregularity was seen during the low valence. Consequently, signals' irregularity is affected by the valence dimension. The results showed that the fusion has more potential for emotion recognition than that of using each signal separately. For sigma = 0.1, the highest recognition rate was 100% in a subject-dependent mode. In a subject-independent mode, the maximum accuracies of 88.57 and 86.8% were obtained for arousal and valence dimensions, respectively.

Fractal dimension of EEG signals and heart dynamics in discrete emotional states

In this study, we explored the fractal dimension (FD; a measure of signal complexity) of 28 EEG channels with positive and negative emotional states. The EEG of participants and their ECG were registered while watching short video clips that induced fear, disgust, humour, or neutral emotions. In order to better understand the nature of these emotions, the Higuchi FD of EEG segments and the heart rate variability (HRV) of the ECG associated with each emotion were obtained. Our results exhibited similar patterns of results with both measures. Humour elicited the highest FD scores in most EEG channels and the highest HRV, while fear, among all emotions, produced the lowest scores in both measures. These results may contribute to the understanding of the relationship between cortical and heart dynamics and their role on emotion perception.

Different Types of Sounds and Their Relationship With the Electrocardiographic Signals and the Cardiovascular System - Review

Background: For some time now, the effects of sound, noise, and music on the human body have been studied. However, despite research done through time, it is still not completely clear what influence, interaction, and effects sounds have on human body. That is why it is necessary to conduct new research on this topic. Thus, in this paper, a systematic review is undertaken in order to integrate research related to several types of sound, both pleasant and unpleasant, specifically noise and music. In addition, it includes as much research as possible to give stakeholders a more general vision about relevant elements regarding methodologies, study subjects, stimulus, analysis, and experimental designs in general. This study has been conducted in order to make a genuine contribution to this area and to perhaps to raise the quality of future research about sound and its effects over ECG signals. Methods: This review was carried out by independent researchers, through three search equations, in four different databases, including: engineering, medicine, and psychology. Inclusion and exclusion criteria were applied and studies published between 1999 and 2017 were considered. The selected documents were read and analyzed independently by each group of researchers and subsequently conclusions were established between all of them. Results: Despite the differences between the outcomes of selected studies, some common factors were found among them. Thus, in noise studies where both BP and HR increased or tended to increase, it was noted that HRV (HF and LF/HF) changes with both sound and noise stimuli, whereas GSR changes with sound and musical stimuli. Furthermore, LF also showed changes with exposure to noise. Conclusion: In many cases, samples displayed a limitation in experimental design, and in diverse studies, there was a lack of a control group. There was a lot of variability in the presented stimuli providing a wide overview of the effects they could produce in humans. In the listening sessions, there were numerous examples of good practice in experimental design, such as the use of headphones and comfortable positions for study subjects, while the listening sessions lasted 20 min in most of the studies.