Advancing Naturalistic Affective Science with Deep Learning

Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

The Future of Affective Science: Introduction to the Special Issue

Modern affective science-the empirical study of emotional responding and affective experience-has been active for a half-century. The Future of Affective Science special issue considers the history of this field and proposes new directions for the decades ahead. Contributors represent diverse theoretical perspectives, methodological expertise, and domains of study, and the special issue includes both literature reviews and new empirical studies as illustrations. This introductory article synthesizes the contributions, articulating the broader context of the current status of our field and highlighting common themes across articles as well as gaps notable even in this special issue. Sections of the article address theoretical and conceptual issues, research methodology, the questions we ask, and translation of basic affective science to applied domains. We conclude that much has been learned from the first 50 years of affective science, and it is now time for new theories, new research questions, and innovative methods for the decades ahead.