A data-driven investigation of human action representations

Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision

We effortlessly extract behaviorally relevant information from dynamic visual input in order to understand the actions of others. In the current study, we develop and test a number of models to better understand the neural representational geometries supporting action understanding. Using fMRI, we measured brain activity as participants viewed a diverse set of 90 different video clips depicting social and nonsocial actions in real-world contexts. We developed five behavioral models using arrangement tasks: two models reflecting behavioral judgments of the purpose (transitivity) and the social content (sociality) of the actions depicted in the video stimuli; and three models reflecting behavioral judgments of the visual content (people, objects, and scene) depicted in still frames of the stimuli. We evaluated how well these models predict neural representational geometry and tested them against semantic models based on verb and nonverb embeddings and visual models based on gaze and motion energy. Our results revealed that behavioral judgments of similarity better reflect neural representational geometry than semantic or visual models throughout much of cortex. The sociality and transitivity models in particular captured a large portion of unique variance throughout the action observation network, extending into regions not typically associated with action perception, like ventral temporal cortex. Overall, our findings expand the action observation network and indicate that the social content and purpose of observed actions are predominant in cortical representation.

Perceptual discrimination of action formidableness and friendliness and the impact of autistic traits

The ability to determine whether the actions of other individuals are friendly or formidable are key decisions we need to make to successfully navigate our complex social environment. In this study we measured perceptual performance when discriminating actions that vary in their friendliness or formidableness, and whether performance was related to the autistic traits of individuals. To do this, we developed an action morphing method to generate novel actions that lied along the action quality dimensions of formidableness and friendliness. In Experiment 1 we show that actions that vary along the formidableness or friendliness continua were rated as varying monotonically along the respective quality. In Experiment 2 we measured the ability of individuals with different levels of autistic traits to discriminate action formidableness and friendliness using adaptive 2-AFC procedures. We found considerable variation in perceptual thresholds when discriminating action formidableness (~ 540% interindividual variation) or friendliness (~ 1100% interindividual variation). Importantly, we found no evidence that autistic traits influenced perceptual discrimination of these action qualities. These results confirm that sensory enhancements with autistic traits are limited to lower level stimuli, and suggest that the perceptual processing of these complex social signals are not affected by autistic traits.

Overlapping representations of observed actions and action-related features

The lateral occipitotemporal cortex (LOTC) has been shown to capture the representational structure of a smaller range of actions. In the current study, we carried out an fMRI experiment in which we presented human participants with images depicting 100 different actions and used representational similarity analysis (RSA) to determine which brain regions capture the semantic action space established using judgments of action similarity. Moreover, to determine the contribution of a wide range of action-related features to the neural representation of the semantic action space we constructed an action feature model on the basis of ratings of 44 different features. We found that the semantic action space model and the action feature model are best captured by overlapping activation patterns in bilateral LOTC and ventral occipitotemporal cortex (VOTC). An RSA on eight dimensions resulting from principal component analysis carried out on the action feature model revealed partly overlapping representations within bilateral LOTC, VOTC, and the parietal lobe. Our results suggest spatially overlapping representations of the semantic action space of a wide range of actions and the corresponding action-related features. Together, our results add to our understanding of the kind of representations along the LOTC that support action understanding.