Incorporating Interpersonal Synchronization Features for Automatic Emotion Recognition from Visual and Audio Data during Communication

Exploring emotional climate recognition in peer conversations through bispectral features and affect dynamics

BACKGROUND AND OBJECTIVE

Emotion recognition in conversations using artificial intelligence (AI) has gained significant attention due to its potential to provide insights into human social behavior. This study extends AI-based emotion recognition to the recognition of emotional climate (EC), which reflects the joint emotional atmosphere dynamically created and perceived by peers during conversations. The objective is to propose and evaluate a novel approach, MLBispec, for EC recognition using speech signals.

METHODS

The MLBispec approach involves time-windowed bispectral analysis of conversational speech signals to extract features related to nonlinear harmonic interactions. These features are combined with peers' affect dynamics, derived from emotion labeling for the same time windows, to form an extended feature set. The combined feature set is then fed into machine learning (ML) classifiers. MLBispec was evaluated on the IEMOCAP, K-EmoCon, and SEWA open-access datasets, which provide 2D emotion annotations (arousal and valence) divided into low/high classes. Additionally, cross-lingual experiments were conducted to test the framework's generalization across languages.

RESULTS

Experimental results demonstrated that MLBispec outperformed previous deep learning-based state-of-the-art approaches in speech emotion recognition, achieving accuracies of 82.6% for arousal and 75.4% for valence. The framework's incorporation of both qualitative and quantitative EC measurements enhanced its ability to characterize the dynamic speech representations of conversational affective structures. Cross-lingual experiments further validated the robustness of MLBispec.

CONCLUSIONS

The findings highlight the effectiveness of MLBispec in objectively recognizing peers' EC during conversations, setting a new standard for practical emotionally-aware applications. These include point-of-care healthcare, human-computer interfaces (HCI), and large-language models (LLMs). By enabling dynamic and reliable EC recognition, MLBispec paves the way for advancements in emotionally intelligent systems.

Incorporating Multimodal Directional Interpersonal Synchrony into Empathetic Response Generation

This study investigates how interpersonal (speaker-partner) synchrony contributes to empathetic response generation in communication scenarios. To perform this investigation, we propose a model that incorporates multimodal directional (positive and negative) interpersonal synchrony, operationalized using the cosine similarity measure, into empathetic response generation. We evaluate how incorporating specific synchrony affects the generated responses at the language and empathy levels. Based on comparison experiments, models with multimodal synchrony generate responses that are closer to ground truth responses and more diverse than models without synchrony. This demonstrates that these features are successfully integrated into the models. Additionally, we find that positive synchrony is linked to enhanced emotional reactions, reduced exploration, and improved interpretation. Negative synchrony is associated with reduced exploration and increased interpretation. These findings shed light on the connections between multimodal directional interpersonal synchrony and empathy's emotional and cognitive aspects in artificial intelligence applications.

EmoNet: Deep Learning-based Emotion Climate Recognition Using Peers' Conversational Speech, Affect Dynamics, and Physiological Data

Understanding the emotional dynamics within social interactions is crucial for meaningful interpretation. Despite progress in emotion recognition systems, recognizing the collective emotional climate among peers has been understudied. Addressing this gap, we propose EmoNet, an AI model transcending traditional emotion identification. EmoNet employs Mel-frequency cepstral coefficients and a Temporal Convolutional Network to extract deep features from speech signals. It uniquely integrates affect dynamics and physiological inputs (heart rate, electrodermal activity), providing a holistic view of emotion climates. Tested on K-EmoCon dataset, EmoNet excels in arousal and valence classification, achieving 87.82% and 83.79% accuracy, respectively. These results position EmoNet as a valuable tool for understanding and influencing emotion climates in real-world conversations, with applications in healthcare and human-computer interactions.

Cross-subject EEG-based emotion recognition through dynamic optimization of random forest with sparrow search algorithm

The objective of EEG-based emotion recognition is to classify emotions by decoding signals, with potential applications in the fields of artificial intelligence and bioinformatics. Cross-subject emotion recognition is more difficult than intra-subject emotion recognition. The poor adaptability of classification model parameters is a significant factor of low accuracy in cross-subject emotion recognition. We propose a model of a dynamically optimized Random Forest based on the Sparrow Search Algorithm (SSA-RF). The decision trees number (DTN) and the leave minimum number (LMN) of the RF are dynamically optimized by the SSA. 12 features are used to construct feature combinations for selecting the optimal feature combination. DEAP and SEED datasets are employed for testing the performance of SSA-RF. The experimental results show that the accuracy of binary classification is 76.81% on DEAP, and the accuracy of triple classification is 75.96% on SEED based on SSA-RF, which are both higher than that of traditional RF. This study provides new insights for the development of cross-subject emotion recognition, and has significant theoretical value.

Analytics and Applications of Audio and Image Sensing Techniques

Nowadays, with numerous sensors placed everywhere around us, we can obtain signals collected from a variety of environment-based sensors, including the ones placed on the ground, cased in the air or water, etc [...].