Synchronization in Interpersonal Speech

Multimodal Approach for Characterizing the Quality of Parent-Child Interaction: A Single Synchronization Source May Not Tell the Whole Story

The interaction between the parent and child is essential for the child's cognitive and emotional development and sets the path for future well-being. These interactions, starting from birth, are necessary for providing the sensory stimulation the child needs in the critical time window of brain development. The characterization of parent-child interactions is traditionally performed by human decoding. This approach is considered the leading and most accurate way of characterizing the quality of these interactions. However, the development of computational tools and especially the concept of parent-child synchronization opened up an additional source of data characterizing these interactions in an objective, less human-labor manner. Such sources include brain-to-brain, voice/speech, eye contact, motor, and heart-rate synchronization. However, can a single source synchronization dataset accurately represent parent-child interaction? Will attending to the same stimulation, often resulting in a higher brain-to-brain synchronization, be considered an interactive condition? In this perspective, we will try to convey a new concept of the child-parent interaction synchronization (CHIPS) matrix, which includes the different sources of signals generated during an interaction. Such a model may assist in explaining the source of interaction alterations in the case of child/parent developmental/emotional or sensory deficits and may open up new ways of assessing interventions and changes in parent-child interactions along development. We will discuss this interaction during one of the parent-child joint activities providing opportunities for interaction, i.e., storytelling.

Incorporating Interpersonal Synchronization Features for Automatic Emotion Recognition from Visual and Audio Data during Communication

During social interaction, humans recognize others’ emotions via individual features and interpersonal features. However, most previous automatic emotion recognition techniques only used individual features—they have not tested the importance of interpersonal features. In the present study, we asked whether interpersonal features, especially time-lagged synchronization features, are beneficial to the performance of automatic emotion recognition techniques. We explored this question in the main experiment (speaker-dependent emotion recognition) and supplementary experiment (speaker-independent emotion recognition) by building an individual framework and interpersonal framework in visual, audio, and cross-modality, respectively. Our main experiment results showed that the interpersonal framework outperformed the individual framework in every modality. Our supplementary experiment showed—even for unknown communication pairs—that the interpersonal framework led to a better performance. Therefore, we concluded that interpersonal features are useful to boost the performance of automatic emotion recognition tasks. We hope to raise attention to interpersonal features in this study.

Interactional synchrony: signals, mechanisms and benefits

Many group-living animals, humans included, occasionally synchronize their behavior with that of conspecifics. Social psychology and neuroscience have attempted to explain this phenomenon. Here we sought to integrate results around three themes: the stimuli, the mechanisms and the benefits of interactional synchrony. As regards stimuli, we asked what characteristics, apart from temporal regularity, prompt synchronization and found that stimulus modality and complexity are important. The high temporal resolution of the auditory system and the relevance of socio-emotional information endow auditory, multimodal, emotional and somewhat variable and adaptive sequences with particular synchronizing power. Looking at the mechanisms revealed that traditional perspectives emphasizing beat-based representations of others' signals conflict with more recent work investigating the perception of temporal regularity. Timing processes supported by striato-cortical loops represent any kind of repetitive interval sequence fairly automatically. Additionally, socio-emotional processes supported by posterior superior temporal cortex help endow such sequences with value motivating the extent of synchronizing. Synchronizing benefits arise from an increased predictability of incoming signals and include many positive outcomes ranging from basic information processing at the individual level to the bonding of dyads and larger groups.