Action perception as hypothesis testing

Bridging flexible goal-directed cognition and consciousness: The Goal-Aligning Representation Internal Manipulation theory

Goal-directed manipulation of internal representations is a key element of human flexible behaviour, while consciousness is commonly associated with higher-order cognition and human flexibility. Current perspectives have only partially linked these processes, thus preventing a clear understanding of how they jointly generate flexible cognition and behaviour. Moreover, these limitations prevent an effective exploitation of this knowledge for technological scopes. We propose a new theoretical perspective that extends our 'three-component theory of flexible cognition' toward higher-order cognition and consciousness, based on the systematic integration of key concepts from Cognitive Neuroscience and AI/Robotics. The theory proposes that the function of conscious processes is to support the alignment of representations with multi-level goals. This higher alignment leads to more flexible and effective behaviours. We analyse here our previous model of goal-directed flexible cognition (validated with more than 20 human populations) as a starting GARIM-inspired model. By bridging the main theories of consciousness and goal-directed behaviour, the theory has relevant implications for scientific and technological fields. In particular, it contributes to developing new experimental tasks and interpreting clinical evidence. Finally, it indicates directions for improving machine learning and robotics systems and for informing real-world applications (e.g., in digital-twin healthcare and roboethics).

Excitatory/inhibitory motor balance reflects individual differences during joint action coordination

Joint action (JA) is a continuous process of motor co-regulation based on the integration of contextual (top-down) and kinematic (bottom-up) cues from partners. The fine equilibrium between excitation and inhibition in sensorimotor circuits is, thus, central to such a dynamic process of action selection and execution. In a bimanual task adapted to become a unimanual JA task, the participant held a bottle (JA), while a confederate had to reach and unscrew either that bottle or another stabilized by a mechanical clamp (No_JA). Prior knowledge was manipulated in each trial such that the participant knew (K) or not (No_K) the target bottle in advance. Online transcranial magnetic stimulation (TMS) was administered at action-relevant landmarks to explore corticospinal excitability (CSE) and inhibition (cortical silent period [cSP]). CSE was modulated early on before the action started if prior information was available. In contrast, cSP modulation emerged later during the reaching action, regardless of prior information. These two indexes could thus reflect the concurrent elaboration of contextual priors (top-down) and the online sampling of partner's kinematic cues (bottom-up). Furthermore, participants selected either one of two possible behavioural strategies, preferring early or late force exertion on the bottle. One translates into a reduced risk of motor coordination failure and the other into reduced metabolic expenditure. Each strategy was characterised by a specific excitatory/inhibitory profile. In conclusion, the study of excitatory/inhibitory balance paves the way for the neurophysiological determination of individual differences in the combination of top-down and bottom-up processing during JA coordination.

How and Why People Synchronize: An Integrated Perspective

Academic AbstractInterpersonal synchrony, the alignment of behavior and/or physiology during interactions, is a pervasive phenomenon observed in diverse social contexts. Here we synthesize across contexts and behaviors to classify the different forms and functions of synchrony. We provide a concise framework for classifying the manifold forms of synchrony along six dimensions: periodicity, discreteness, spatial similarity, directionality, leader-follower dynamics, and observability. We also distill the various proposed functions of interpersonal synchrony into four interconnected functions: reducing complexity and improving understanding, accomplishing joint tasks, strengthening social connection, and influencing partners' behavior. These functions derive from first principles, emerge from each other, and are accomplished by some forms of synchrony more than others. Effective synchrony flexibly adapts to social goals and more synchrony is not always better. Our synthesis offers a shared framework and language for the field, allowing for better cross-context and cross-behavior comparisons, generating new hypotheses, and highlighting future research directions.

Predicting others' actions from their social contexts

Contextual cues have been shown to inform our understanding and predictions of others' actions. In this study, we tested whether observers' predictions about unfolding actions depend upon the social context in which they occur. Across five experiments, we showed participants videos of an actor walking toward a piece of furniture either with (joint context) or without (solo context) a partner standing by it. We found greater predictive bias, indicative of stronger action expectations when videos contained a second actor (Experiment 1), even when the solo condition had a perceptually-matched control object in place of the actor (Experiment 2). Critically, belief manipulations about the actions the walking actor would perform suppressed the difference between social context conditions when the manipulation specified an action possible in both contexts (Experiment 5) but not when the action was one that would be difficult without a partner (Experiment 4). Interestingly, the social context effect persisted when the belief manipulation specified an unlikely action given the depicted scene (Experiment 3). These findings provide novel evidence that kinematically-identical actions can elicit different predictions depending on the social context in which they occur.

An active inference model of hierarchical action understanding, learning and imitation

We advance a novel active inference model of the cognitive processing that underlies the acquisition of a hierarchical action repertoire and its use for observation, understanding and imitation. We illustrate the model in four simulations of a tennis learner who observes a teacher performing tennis shots, forms hierarchical representations of the observed actions, and imitates them. Our simulations show that the agent's oculomotor activity implements an active information sampling strategy that permits inferring the kinematic aspects of the observed movement, which lie at the lowest level of the action hierarchy. In turn, this low-level kinematic inference supports higher-level inferences about deeper aspects of the observed actions: proximal goals and intentions. Finally, the inferred action representations can steer imitative responses, but interfere with the execution of different actions. Our simulations show that hierarchical active inference provides a unified account of action observation, understanding, learning and imitation and helps explain the neurobiological underpinnings of visuomotor cognition, including the multiple routes for action understanding in the dorsal and ventral streams and mirror mechanisms.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

Kinematic priming of action predictions

The ability to anticipate what others will do next is crucial for navigating social, interactive environments. Here, we develop an experimental and analytical framework to measure the implicit readout of prospective intention information from movement kinematics. Using a primed action categorization task, we first demonstrate implicit access to intention information by establishing a novel form of priming, which we term kinematic priming: subtle differences in movement kinematics prime action prediction. Next, using data collected from the same participants in a forced-choice intention discrimination task 1 h later, we quantify single-trial intention readout-the amount of intention information read by individual perceivers in individual kinematic primes-and assess whether it can be used to predict the amount of kinematic priming. We demonstrate that the amount of kinematic priming, as indexed by both response times (RTs) and initial fixations to a given probe, is directly proportional to the amount of intention information read by the individual perceiver at the single-trial level. These results demonstrate that human perceivers have rapid, implicit access to intention information encoded in movement kinematics and highlight the potential of our approach to reveal the computations that permit the readout of this information with single-subject, single-trial resolution.

From fear of falling to choking under pressure: A predictive processing perspective of disrupted motor control under anxiety

Under the Predictive Processing Framework, perception is guided by internal models that map the probabilistic relationship between sensory states and their causes. Predictive processing has contributed to a new understanding of both emotional states and motor control but is yet to be fully applied to their interaction during the breakdown of motor movements under heightened anxiety or threat. We bring together literature on anxiety and motor control to propose that predictive processing provides a unifying principle for understanding motor breakdowns as a disruption to the neuromodulatory control mechanisms that regulate the interactions of top-down predictions and bottom-up sensory signals. We illustrate this account using examples from disrupted balance and gait in populations who are anxious/fearful of falling, as well as 'choking' in elite sport. This approach can explain both rigid and inflexible movement strategies, as well as highly variable and imprecise action and conscious movement processing, and may also unite the apparently opposing self-focus and distraction approaches to choking. We generate predictions to guide future work and propose practical recommendations.

The implicit effect of action mental simulation on action evaluation

The main assumption underlying the present investigation is that action observation elicits a mandatory mental simulation representing the action forward in time. In Experiment 1, participants observed pairs of photos portraying the initial and the final still frames of an action video; then they observed a photo depicting the very same action but either forward or backward in time. Their task was to tell whether the action in the photo portrayed something happened before or after the action seen at encoding. In this explicit task, the evaluation was faster for forward photos than for backward photos. Crucially, the effect was replicated when instructions asked only to evaluate at test whether the photo depicted a scene congruent with the action seen at encoding (implicit task from two still frames, Experiment 2), and when at encoding, they were presented a single still frame and evaluated at test whether a photo depicted a scene congruent with the action seen at encoding (implicit task from single still frame; Experiment 3). Overall, the results speak in favour of a mandatory mechanism through which our brain simulates the action also in tasks that do not explicitly require action simulation.

The acute effects of action observation training on upper extremity functions, cognitive processes and reaction times: a randomized controlled trial

Aim: To investigate the acute effects of action observation training on upper extremity functions, cognitive functions and response time in healthy, young adults. Materials & methods: A total of 60 participants were randomly divided into five groups: the self-action observation group, action observation group, action practice group, non-action observation group and control group. The Jebsen-Taylor hand function test (JTHFT), nine-hole peg test, serial reaction time task and d2 test of attention were applied to the participants before and after the interventions. Results: JTHFT performance with both non-dominant and dominant hands improved significantly compared with baseline in all groups (p < 0.001). JTHFT performance with non-dominant and dominant hands differed between the groups (p < 0.001). Conclusion: Action observation training seems to enhance the performance of upper extremity-related functions. Observing self-actions resulted in statistically significant positive changes in more variables compared with other methods. However, its clinical effectiveness over the other methods should be investigated in future long-term studies. Clinical Trial Registration: NCT04932057 (ClinicalTrials.gov).

Effective Connectivity of the Human Mirror Neuron System During Social Cognition

The human mirror neuron system (MNS) can be considered the neural basis of social cognition. Identifying the global network structure of this system can provide significant progress in the field. In this study, we use dynamic causal modeling (DCM) to determine the effective connectivity between central regions of the MNS for the first time during different social cognition tasks. Sixty-seven healthy participants completed fMRI scanning while performing social cognition tasks, including imitation, empathy and theory of mind. Superior temporal sulcus (STS), inferior parietal lobule (IPL) and Brodmann area 44 (BA44) formed the regions of interest for DCM. Varying connectivity patterns, 540 models were built and fitted for each participant. By applying group-level analysis, Bayesian model selection and Bayesian model averaging, the optimal family and model for all experimental tasks were found. For all social-cognitive processes, effective connectivity from STS to IPL and from STS to BA44 was found. For imitation, additional mutual connections occurred between STS and BA44, as well as BA44 and IPL. The results suggest inverse models in which the motor regions BA44 and IPL receive sensory information from the STS. In contrast, for imitation, a sensory loop with an exchange of motor-to-sensory and sensory-to-motor information seems to exist.

Can the early visual processing of others' actions be related to social power and dominance?

Although goals often drive action understanding, this ability is also prone to important variability among individuals, which may have its origin in individual social characteristics. The present study aimed at evaluating the relationship between the tendency to prioritize goal information over grip information during early visual processing of action and several social dimensions. Visual processing of grip and goal information during action recognition was evaluated in 64 participants using the priming protocol developed by Decroix and Kalénine (Exp Brain Res 236(8):2411-2426, 2018). Object-directed action photographs were primed by photographs sharing the same goal and/or the same grip. The effects of goal and grip priming on action recognition were evaluated for different prime durations. The same participants further fulfilled questionnaires characterizing the way individuals deal with their social environment, namely their sense of social power, dominance, perspective taking, and construal level. At the group level, results confirmed greater goal than grip priming effects on action recognition for the shortest prime duration. Regression analyses between the pattern of response times in the action priming protocol and scores at the questionnaires further showed that the advantage of goal over grip priming was associated with higher sense of social power, and possibly to lower dominance. Overall, data confirm that observers tend to prioritize goal-related information when processing visual actions but further indicate that this tendency is sensitive to individual social characteristics. Results suggest that goal information may not always drive action understanding and point out the connection between low-level processing of observed actions and more general individual characteristics.

Spontaneous perception: a framework for task-free, self-paced perception

Flipping through social media feeds, viewing exhibitions in a museum, or walking through the botanical gardens, people consistently choose to engage with and disengage from visual content. Yet, in most laboratory settings, the visual stimuli, their presentation duration, and the task at hand are all controlled by the researcher. Such settings largely overlook the spontaneous nature of human visual experience, in which perception takes place independently from specific task constraints and its time course is determined by the observer as a self-governing agent. Currently, much remains unknown about how spontaneous perceptual experiences unfold in the brain. Are all perceptual categories extracted during spontaneous perception? Does spontaneous perception inherently involve volition? Is spontaneous perception segmented into discrete episodes? How do different neural networks interact over time during spontaneous perception? These questions are imperative to understand our conscious visual experience in daily life. In this article we propose a framework for spontaneous perception. We first define spontaneous perception as a task-free and self-paced experience. We propose that spontaneous perception is guided by four organizing principles that grant it temporal and spatial structures. These principles include coarse-to-fine processing, continuity and segmentation, agency and volition, and associative processing. We provide key suggestions illustrating how these principles may interact with one another in guiding the multifaceted experience of spontaneous perception. We point to testable predictions derived from this framework, including (but not limited to) the roles of the default-mode network and slow cortical potentials in underlying spontaneous perception. We conclude by suggesting several outstanding questions for future research, extending the relevance of this framework to consciousness and spontaneous brain activity. In conclusion, the spontaneous perception framework proposed herein integrates components in human perception and cognition, which have been traditionally studied in isolation, and opens the door to understand how visual perception unfolds in its most natural context.

The extrafoveal preview paradigm as a measure of predictive, active sampling in visual perception

A key feature of visual processing in humans is the use of saccadic eye movements to look around the environment. Saccades are typically used to bring relevant information, which is glimpsed with extrafoveal vision, into the high-resolution fovea for further processing. With the exception of some unusual circumstances, such as the first fixation when walking into a room, our saccades are mainly guided based on this extrafoveal preview. In contrast, the majority of experimental studies in vision science have investigated "passive" behavioral and neural responses to suddenly appearing and often temporally or spatially unpredictable stimuli. As reviewed here, a growing number of studies have investigated visual processing of objects under more natural viewing conditions in which observers move their eyes to a stationary stimulus, visible previously in extrafoveal vision, during each trial. These studies demonstrate that the extrafoveal preview has a profound influence on visual processing of objects, both for behavior and neural activity. Starting from the preview effect in reading research we follow subsequent developments in vision research more generally and finally argue that taking such evidence seriously leads to a reconceptualization of the nature of human visual perception that incorporates the strong influence of prediction and action on sensory processing. We review theoretical perspectives on visual perception under naturalistic viewing conditions, including theories of active vision, active sensing, and sampling. Although the extrafoveal preview paradigm has already provided useful information about the timing of, and potential mechanisms for, the close interaction of the oculomotor and visual systems while reading and in natural scenes, the findings thus far also raise many new questions for future research.

Disentangling effects of expectancy, accuracy, and empathy on the processing of observed actions

A number of studies suggest that event-related potential (ERP) components previously associated with error processing might represent expectation violation instead of valence. When observing others, these processes might further be modulated by trait empathy. We suggest that trait empathy modulates expectancy formation and that these expectancies then influence observed response processing as reflected in a frontocentral negative ERP component resembling the previously described observer error-related negativity. We acquired single trial ERPs of participants who observed another person in a true- or false-belief condition answering correctly or erroneously. Additionally, we prompted participants' expectancy in some trials. Using linear mixed model analyses, we found that for low empathy participants, expectations for the false-belief condition decreased throughout the experiment, so that expectations were more pronounced in participants with higher empathy toward the end of the experiment. We also found that single trial expectancy measures derived from regression models of the measured expectancies predicted the amplitude of the frontocentral negative ERP component, and that neither the addition of empathy nor accuracy or trial type (true- or false-belief) led to the explanation of significantly more variance compared with the model just containing expectancy as predictor. These results suggest that empathy modulates the processing of observed responses indirectly via its effect on expectancy of the response.

The role of trait empathy in the processing of observed actions in a false-belief task

Empathic brain responses are characterized by overlapping activations between active experience and observation of an emotion in another person, with the pattern for observation being modulated by trait empathy. Also for self-performed and observed errors, similar brain activity has been described, but findings concerning the role of empathy are mixed. We hypothesized that trait empathy modulates the processing of observed responses if expectations concerning the response are based on the beliefs of the observed person. In the present study, we utilized a false-belief task in which observed person’s and observer’s task-related knowledge were dissociated and errors and correct responses could be expected or unexpected. While theta power was generally modulated by the expectancy of the observed response, a negative mediofrontal event-related potential (ERP) component was more pronounced for unexpected observed actions only in participants with higher trait empathy (assessed by the Empathy Quotient), as revealed by linear mixed effects analyses. Cognitive and affective empathy, assessed by the Interpersonal Reactivity Index, were not significantly related to the ERP component. The results suggest that trait empathy can facilitate the generation of predictions and thereby modulate specific aspects of the processing of observed actions, while the contributions of specific empathy components remain unclear.

Predictive Processing in Cognitive Robotics: A Review

Predictive processing has become an influential framework in cognitive sciences. This framework turns the traditional view of perception upside down, claiming that the main flow of information processing is realized in a top-down, hierarchical manner. Furthermore, it aims at unifying perception, cognition, and action as a single inferential process. However, in the related literature, the predictive processing framework and its associated schemes, such as predictive coding, active inference, perceptual inference, and free-energy principle, tend to be used interchangeably. In the field of cognitive robotics, there is no clear-cut distinction on which schemes have been implemented and under which assumptions. In this letter, working definitions are set with the main aim of analyzing the state of the art in cognitive robotics research working under the predictive processing framework as well as some related nonrobotic models. The analysis suggests that, first, research in both cognitive robotics implementations and nonrobotic models needs to be extended to the study of how multiple exteroceptive modalities can be integrated into prediction error minimization schemes. Second, a relevant distinction found here is that cognitive robotics implementations tend to emphasize the learning of a generative model, while in nonrobotics models, it is almost absent. Third, despite the relevance for active inference, few cognitive robotics implementations examine the issues around control and whether it should result from the substitution of inverse models with proprioceptive predictions. Finally, limited attention has been placed on precision weighting and the tracking of prediction error dynamics. These mechanisms should help to explore more complex behaviors and tasks in cognitive robotics research under the predictive processing framework.

Effects of trait empathy and expectation on the processing of observed actions

Recent evidence suggests that the processing of observed actions may reflect an action prediction error, with more pronounced mediofrontal negative event-related potentials (ERPs) for unexpected actions. This evidence comes from an application of a false-belief task, where unexpected correct responses elicited high ERP amplitudes. An alternative interpretation is that the ERP component reflects vicarious error processing, as objectively correct responses were errors from the observed person's perspective. In this study, we aimed to disentangle the two possibilities by adding the factor task difficulty, which varied expectations without affecting the definition of (vicarious) errors, and to explore the role of empathy in action observation. We found that the relationship between empathy and event-related potentials (ERPs) mirrored the relationship between empathy and behavioral expectancy measures. Only in the easy task condition did higher empathy lead to stronger expectancy of correct responses in the true-belief and of errors in the false-belief condition. A compatible pattern was found for an early ERP component (150-200 ms) after the observed response, with a larger negativity for error than correct responses in the true-belief and the reverse pattern in the false-belief condition, but only in highly empathic participants. We conclude that empathy facilitates the formation of expectations regarding the actions of others. These expectations then modulate the processing of observed actions, as indicated by the ERPs in the present study.

Transcranial Magnetic Stimulation-Induced Motor Cortex Activity Influences Visual Awareness Judgments

The influence of non-visual information on visual awareness judgments has recently gained substantial interest. Using single-pulse transcranial magnetic stimulation (TMS), we investigate the potential contribution of evidence from the motor system to judgment of visual awareness. We hypothesized that TMS-induced activity in the primary motor cortex (M1) would increase reported visual awareness as compared to the control condition. Additionally, we investigated whether TMS-induced motor-evoked potential (MEP) could measure accumulated evidence for stimulus perception. Following stimulus presentation and TMS, participants first rated their visual awareness verbally using the Perceptual Awareness Scale (PAS), after which they responded manually to a Gabor orientation identification task. Delivering TMS to M1 resulted in higher average awareness ratings as compared to the control condition, in both correct and incorrect identification task response trials, when the hand with which participants responded was contralateral to the stimulated hemisphere (TMS-response-congruent trials). This effect was accompanied by longer PAS response times (RTs), irrespective of the congruence between TMS and identification response. Moreover, longer identification RTs were observed in TMS-response-congruent trials in the M1 condition as compared to the control condition. Additionally, the amplitudes of MEPs were related to the awareness ratings when response congruence was taken into account. We argue that MEP can serve as an indirect measure of evidence accumulated for stimulus perception and that longer PAS RTs and higher amplitudes of MEPs in the M1 condition reflect integration of additional evidence with visual awareness judgment. In conclusion, we advocate that motor activity influences perceptual awareness judgments.

Affordance matching predictively shapes the perceptual representation of others' ongoing actions

Predictive processing accounts of social perception argue that action observation is a predictive process, in which inferences about others' goals are tested against the perceptual input, inducing a subtle perceptual confirmation bias that distorts observed action kinematics toward the inferred goals. Here we test whether such biases are induced even when goals are not explicitly given but have to be derived from the unfolding action kinematics. In 2 experiments, participants briefly saw an actor reach ambiguously toward a large object and a small object, with either a whole-hand power grip or an index-finger and thumb precision grip. During its course, the hand suddenly disappeared, and participants reported its last seen position on a touch-screen. As predicted, judgments were consistently biased toward apparent action targets, such that power grips were perceived closer to large objects and precision grips closer to small objects, even if the reach kinematics were identical. Strikingly, these biases were independent of participants' explicit goal judgments. They were of equal size when action goals had to be explicitly derived in each trial (Experiment 1) or not (Experiment 2) and, across trials and across participants, explicit judgments and perceptual biases were uncorrelated. This provides evidence, for the first time, that people make online adjustments of observed actions based on the match between hand grip and object goals, distorting their perceptual representation toward implied goals. These distortions may not reflect high-level goal assumptions, but emerge from relatively low-level processing of kinematic features within the perceptual system. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Neural dynamics of grip and goal integration during the processing of others' actions with objects: An ERP study

Recent behavioural evidence suggests that when processing others’ actions, motor acts and goal-related information both contribute to action recognition. Yet the neuronal mechanisms underlying the dynamic integration of the two action dimensions remain unclear. This study aims to elucidate the ERP components underlying the processing and integration of grip and goal-related information. The electrophysiological activity of 28 adults was recorded during the processing of object-directed action photographs (e.g., writing with pencil) containing either grip violations (e.g. upright pencil grasped with atypical-grip), goal violations (e.g., upside-down pencil grasped with typical-grip), both grip and goal violations (e.g., upside-down pencil grasped with atypical-grip), or no violations. Participants judged whether actions were overall typical or not according to object typical use. Brain activity was sensitive to the congruency between grip and goal information on the N400, reflecting the semantic integration between the two dimensions. On earlier components, brain activity was affected by grip and goal typicality independently. Critically, goal typicality but not grip typicality affected brain activity on the N300, supporting an earlier role of goal-related representations in action recognition. Findings provide new insights on the neural temporal dynamics of the integration of motor acts and goal-related information during the processing of others’ actions.

What first drives visual attention during the recognition of object-directed actions? The role of kinematics and goal information

The recognition of others' object-directed actions is known to involve the decoding of both the visual kinematics of the action and the action goal. Yet whether action recognition is first guided by the processing of visual kinematics or by a prediction about the goal of the actor remains debated. In order to provide experimental evidence to this issue, the present study aimed at investigating whether visual attention would be preferentially captured by visual kinematics or by action goal information when processing others' actions. In a visual search task, participants were asked to find correct actions (e.g., drinking from glass) among distractor actions. Distractors actions contained grip and/or goal violations and could therefore share the correct goal and/or the correct grip with the target. The time course of fixation proportion on each distractor action has been taken as an indicator of visual attention allocation. Results show that visual attention is first captured by the distractor action with similar goal. Then the withdrawal of visual attention from the action distractor with similar goal suggests a later attentional capture by the action distractor with similar grip. Overall, results are in line with predictive approaches of action understanding, which assume that observers first make a prediction about the actor's goal before verifying this prediction using the visual kinematics of the action.

Bringing action into the picture. How action influences visual awareness

This article discusses how the analysis of interactions between action and awareness allows us to better understand the mechanisms of visual awareness. We argue that action is one of several factors that influence visual awareness and we provide a number of examples. We also discuss the possible mechanisms that underlie these influences on both the cognitive and the neural levels. We propose that action affects visual awareness for the following reasons: (1) it serves as additional information in the process of evidence accumulation; (2) it restricts the number of alternatives in the decisional process; (3) it enables error detection and performance monitoring; and (4) it triggers attentional mechanisms that modify stimulus perception. We also discuss the possible neuronal mechanisms of the aforementioned effects, including feedback-dependent prefrontal cortex modulation of the activity of visual areas, error-based modulation, interhemispheric inhibition of motor cortices, and attentional modulation of visual cortex activity triggered by motor processing.

Multi-layer adaptation of group coordination in musical ensembles

Group coordination passes through an efficient integration of multimodal sources of information. This study examines complex non-verbal communication by recording movement kinematics from conductors and two sections of violinists of an orchestra adapting to a perturbation affecting their normal pattern of sensorimotor communication (rotation of half a turn of the first violinists’ section). We show that different coordination signals are channeled through ancillary (head kinematics) and instrumental movements (bow kinematics). Each one of them affect coordination either at the inter-group or intra-group levels, therefore tapping into different modes of cooperation: complementary versus imitative coordination. Our study suggests that the co-regulation of group behavior is based on the exchange of information across several layers, each one of them tuned to carry specific coordinative signals. Multi-layer sensorimotor communication may be the key musicians and, more generally humans, use to flexibly communicate between each other in interactive sensorimotor tasks.

Making the Environment an Informative Place: A Conceptual Analysis of Epistemic Policies and Sensorimotor Coordination

How do living organisms decide and act with limited and uncertain information? Here, we discuss two computational approaches to solving these challenging problems: a "cognitive" and a "sensorimotor" enrichment of stimuli, respectively. In both approaches, the key notion is that agents can strategically modulate their behavior in informative ways, e.g., to disambiguate amongst alternative hypotheses or to favor the perception of stimuli providing the information necessary to later act appropriately. We discuss how, despite their differences, both approaches appeal to the notion that actions must obey both epistemic (i.e., information-gathering or uncertainty-reducing) and pragmatic (i.e., goal- or reward-maximizing) imperatives and balance them. Our computationally-guided analysis reveals that epistemic behavior is fundamental to understanding several facets of cognitive processing, including perception, decision making, and social interaction.

The neural oscillatory markers of phonetic convergence during verbal interaction

During a conversation, the neural processes supporting speech production and perception overlap in time and, based on context, expectations and the dynamics of interaction, they are also continuously modulated in real time. Recently, the growing interest in the neural dynamics underlying interactive tasks, in particular in the language domain, has mainly tackled the temporal aspects of turn-taking in dialogs. Besides temporal coordination, an under-investigated phenomenon is the implicit convergence of the speakers toward a shared phonetic space. Here, we used dual electroencephalography (dual-EEG) to record brain signals from subjects involved in a relatively constrained interactive task where they were asked to take turns in chaining words according to a phonetic rhyming rule. We quantified participants' initial phonetic fingerprints and tracked their phonetic convergence during the interaction via a robust and automatic speaker verification technique. Results show that phonetic convergence is associated to left frontal alpha/low-beta desynchronization during speech preparation and by high-beta suppression before and during listening to speech in right centro-parietal and left frontal sectors, respectively. By this work, we provide evidence that mutual adaptation of speech phonetic targets, correlates with specific alpha and beta oscillatory dynamics. Alpha and beta oscillatory dynamics may index the coordination of the "when" as well as the "how" speech interaction takes place, reinforcing the suggestion that perception and production processes are highly interdependent and co-constructed during a conversation.

Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis

While the neurobiology of simple and habitual choices is relatively well known, our current understanding of goal-directed choices and planning in the brain is still limited. Theoretical work suggests that goal-directed computations can be productively associated to model-based (reinforcement learning) computations, yet a detailed mapping between computational processes and neuronal circuits remains to be fully established. Here we report a computational analysis that aligns Bayesian nonparametrics and model-based reinforcement learning (MB-RL) to the functioning of the hippocampus (HC) and the ventral striatum (vStr)-a neuronal circuit that increasingly recognized to be an appropriate model system to understand goal-directed (spatial) decisions and planning mechanisms in the brain. We test the MB-RL agent in a contextual conditioning task that depends on intact hippocampus and ventral striatal (shell) function and show that it solves the task while showing key behavioral and neuronal signatures of the HC-vStr circuit. Our simulations also explore the benefits of biological forms of look-ahead prediction (forward sweeps) during both learning and control. This article thus contributes to fill the gap between our current understanding of computational algorithms and biological realizations of (model-based) reinforcement learning.

Error, rather than its probability, elicits specific electrocortical signatures: a combined EEG-immersive virtual reality study of action observation

Detecting errors in one’s own actions, and in the actions of others, is a crucial ability for adaptable and flexible behavior. Studies show that specific EEG signatures underpin the monitoring of observed erroneous actions (error-related negativity, error positivity, mid-frontal theta oscillations). However, the majority of studies on action observation used sequences of trials where erroneous actions were less frequent than correct actions. Therefore, it was not possible to disentangle whether the activation of the performance monitoring system was due to an error, as a violation of the intended goal, or to a surprise/novelty effect, associated with a rare and unexpected event. Combining EEG and immersive virtual reality (IVR-CAVE system), we recorded the neural signal of 25 young adults who observed, in first-person perspective, simple reach-to-grasp actions performed by an avatar aiming for a glass. Importantly, the proportion of erroneous actions was higher than correct actions. Results showed that the observation of erroneous actions elicits the typical electrocortical signatures of error monitoring, and therefore the violation of the action goal is still perceived as a salient event. The observation of correct actions elicited stronger alpha suppression. This confirmed the role of the alpha-frequency band in the general orienting response to novel and infrequent stimuli. Our data provide novel evidence that an observed goal error (the action slip) triggers the activity of the performance-monitoring system even when erroneous actions, which are, typically, relevant events, occur more often than correct actions and thus are not salient because of their rarity.

NEW & NOTEWORTHY Activation of the performance-monitoring system (PMS) is typically investigated when errors in a sequence are comparatively rare. However, whether the PMS is activated by errors per se or by their infrequency is not known. Combining EEG-virtual reality techniques, we found that observing frequent (70%) action errors performed by avatars elicits electrocortical error signatures suggesting that deviation from the prediction of how learned actions should correctly deploy, rather than its frequency, is coded in the PMS.

Intelligence and uncertainty: Implications of hierarchical predictive processing for the neuroscience of cognitive ability

Hierarchical predictive processing (PP) has recently emerged as a candidate theoretical paradigm for neurobehavioral research. To date, PP has found support through its success in offering compelling explanations for a number of perceptual, cognitive, and psychiatric phenomena, as well as from accumulating neurophysiological evidence. However, its implications for understanding intelligence and its neural basis have received relatively little attention. The present review outlines the key tenets and evidence for PP, and assesses its implications for intelligence research. It is argued that PP suggests indeterminacy as a unifying principle from which to investigate the cognitive hierarchy and brain-ability correlations. The resulting framework not only accommodates prominent psychometric models of intelligence, but also incorporates key findings from neuroanatomical and functional activation research, and motivates new predictions via the mechanisms of prediction-error minimization. Because PP also suggests unique neural signatures of experience-dependent activity, it may also help clarify environmental contributions to intellectual development. It is concluded that PP represents a plausible, integrative framework that could enhance progress in the neuroscience of intelligence.

Computational Techniques for Eye Movements Analysis towards Supporting Early Diagnosis of Alzheimer's Disease: A Review

An opportune early diagnosis of Alzheimer's disease (AD) would help to overcome symptoms and improve the quality of life for AD patients. Research studies have identified early manifestations of AD that occur years before the diagnosis. For instance, eye movements of people with AD in different tasks differ from eye movements of control subjects. In this review, we present a summary and evolution of research approaches that use eye tracking technology and computational analysis to measure and compare eye movements under different tasks and experiments. Furthermore, this review is targeted to the feasibility of pioneer work on developing computational tools and techniques to analyze eye movements under naturalistic scenarios. We describe the progress in technology that can enhance the analysis of eye movements everywhere while subjects perform their daily activities and give future research directions to develop tools to support early AD diagnosis through analysis of eye movements.

Real-Time Prediction of Observed Action Requires Integrity of the Dorsal Premotor Cortex: Evidence From Repetitive Transcranial Magnetic Stimulation

Studying brain mechanisms underlying the prediction of observed action, the dorsal premotor cortex (PMd) has been suggested a key area. The present study probed this notion using repetitive transcranial magnetic stimulation (rTMS) to test whether interference in this area would affect the accuracy in predicting the time course of object directed actions performed with the right hand. Young and healthy participants observed actions in short videos. These were briefly occluded from view for 600 ms and resumed immediately afterwards. The task was to continue the action mentally and to indicate after each occlusion, whether the action was resumed at the right moment (condition in-time) or shifted. In a first run, single-pulse transcranial magnetic stimulation (sTMS) was delivered over the left primary hand-area during occlusion. In the second run, rTMS over the left PMd was applied during occlusion in half of the participants [experimental group (EG)]. The control group (CG) received sham-rTMS over the same area. Under rTMS, the EG predicted less trials correctly than in the sTMS run. Sham-rTMS in the CG had no effects on prediction. The interference in PMd interacted with the type of manipulation applied to the action’s time course occasionally during occlusion. The performance decrease of the EG was most pronounced in conditions in which the continuations after occlusions were too late in the action’s course. The present results extend earlier findings suggesting that real-time action prediction requires the integrity of the PMd. Different functional roles of this area are discussed. Alternative interpretations consider either simulation of specific motor programming functions or the involvement of a feature-unspecific predictor.

Action Prediction Allows Hypothesis Testing via Internal Forward Models at 6 Months of Age

We propose that action prediction provides a cornerstone in a learning process known as internal forward models. According to this suggestion infants' predictions (looking to the mouth of someone moving a spoon upward) will moments later be validated or proven false (spoon was in fact directed toward a bowl), information that is directly perceived as the distance between the predicted and actual goal. Using an individual difference approach we demonstrate that action prediction correlates with the tendency to react with surprise when social interactions are not acted out as expected (action evaluation). This association is demonstrated across tasks and in a large sample (n = 118) at 6 months of age. These results provide the first indication that infants might rely on internal forward models to structure their social world. Additional analysis, consistent with prior work and assumptions from embodied cognition, demonstrates that the latency of infants' action predictions correlate with the infant's own manual proficiency.

Early modulation of intra-cortical inhibition during the observation of action mistakes

Errors while performing an action are fundamental for learning. During interaction others' errors must be monitored and taken into account to allow joint action coordination and imitation learning. This monitoring relies on an action observation network (AON) mainly based on parietofrontal recurrent circuits. Although different studies suggest that inappropriate actions may rapidly be inhibited during execution, little is known about the modulation of the AON when an action misstep is shown. Here we used single and paired pulse transcranial magnetic stimulation to assess corticospinal excitability, intracortical facilitation and intracortical inhibition at different time intervals (120, 180, 240 ms) after the visual presentation of a motor execution error. Results show a specific and early (120 ms) decrease of intracortical inhibition likely because of a significant mismatch between the observed erroneous action and observer's expectations. Indeed, as proposed by the top-down predictive framework, the motor system may be involved in the generation of these error signals and our data show that this mechanism could rely on the early decrease of intracortical inhibition within the corticomotor system.

The influence of communication mode on written language processing and beyond

Empirical evidence suggests a broad impact of communication mode on cognition at large, beyond language processing. Using a sign language since infancy might shape the representation of words and other linguistic stimuli - for example, incorporating in it the movements and signs used to express them. Once integrated into linguistic representations, this visuo-motor content can affect deaf signers' linguistic and cognitive processing.

Increased heart rate after exercise facilitates the processing of fearful but not disgusted faces

Embodied theories of emotion assume that emotional processing is grounded in bodily and affective processes. Accordingly, the perception of an emotion re-enacts congruent sensory and affective states; and conversely, bodily states congruent with a specific emotion facilitate emotional processing. This study tests whether the ability to process facial expressions (faces having a neutral expression, expressing fear, or disgust) can be influenced by making the participants’ body state congruent with the expressed emotion (e.g., high heart rate in the case of faces expressing fear). We designed a task requiring participants to categorize pictures of male and female faces that either had a neutral expression (neutral), or expressed emotions whose linkage with high heart rate is strong (fear) or significantly weaker or absent (disgust). Critically, participants were tested in two conditions: with experimentally induced high heart rate (Exercise) and with normal heart rate (Normal). Participants processed fearful faces (but not disgusted or neutral faces) faster when they were in the Exercise condition than in the Normal condition. These results support the idea that an emotionally congruent body state facilitates the automatic processing of emotionally-charged stimuli and this effect is emotion-specific rather than due to generic factors such as arousal.

Where Are You Throwing the Ball? I Better Watch Your Body, Not Just Your Arm!

The ability to intercept or avoid a moving object, whether to catch a ball, snatch one's prey, or avoid the path of a predator, is a skill that has been acquired throughout evolution by many species in the animal kingdom. This requires processing early visual cues in order to program anticipatory motor responses tuned to the forthcoming event. Here, we explore the nature of the early kinematics cues that could inform an observer about the future direction of a ball projected with an unconstrained overarm throw. Our goal was to pinpoint the body segments that, throughout the temporal course of the throwing action, could provide key cues for accurately predicting the side of the outgoing ball. We recorded whole-body kinematics from twenty non-expert participants performing unconstrained overarm throws at four different targets placed on a vertical plane at 6 m distance. In order to characterize the spatiotemporal structure of the information embedded in the kinematics of the throwing action about the outgoing ball direction, we introduced a novel combination of dimensionality reduction and machine learning techniques. The recorded kinematics clearly shows that throwing styles differed considerably across individuals, with corresponding inter-individual differences in the spatio-temporal structure of the thrower predictability. We found that for most participants it is possible to predict the region where the ball hit the target plane, with an accuracy above 80%, as early as 400-500 ms before ball release. Interestingly, the body parts that provided the most informative cues about the action outcome varied with the throwing style and during the time course of the throwing action. Not surprisingly, at the very end of the action, the throwing arm is the most informative body segment. However, cues allowing for predictions to be made earlier than 200 ms before release are typically associated to other body parts, such as the lower limbs and the contralateral arm. These findings are discussed in the context of the sport-science literature on throwing and catching interactive tasks, as well as from the wider perspective of the role of sensorimotor coupling in interpersonal social interactions.

Internally generated hippocampal sequences as a vantage point to probe future-oriented cognition

Information processing in the rodent hippocampus is fundamentally shaped by internally generated sequences (IGSs), expressed during two different network states: theta sequences, which repeat and reset at the ∼8 Hz theta rhythm associated with active behavior, and punctate sharp wave-ripple (SWR) sequences associated with wakeful rest or slow-wave sleep. A potpourri of diverse functional roles has been proposed for these IGSs, resulting in a fragmented conceptual landscape. Here, we advance a unitary view of IGSs, proposing that they reflect an inferential process that samples a policy from the animal's generative model, supported by hippocampus-specific priors. The same inference affords different cognitive functions when the animal is in distinct dynamical modes, associated with specific functional networks. Theta sequences arise when inference is coupled to the animal's action-perception cycle, supporting online spatial decisions, predictive processing, and episode encoding. SWR sequences arise when the animal is decoupled from the action-perception cycle and may support offline cognitive processing, such as memory consolidation, the prospective simulation of spatial trajectories, and imagination. We discuss the empirical bases of this proposal in relation to rodent studies and highlight how the proposed computational principles can shed light on the mechanisms of future-oriented cognition in humans.

You cannot speak and listen at the same time: a probabilistic model of turn-taking

Turn-taking is a preverbal skill whose mastering constitutes an important precondition for many social interactions and joint actions. However, the cognitive mechanisms supporting turn-taking abilities are still poorly understood. Here, we propose a computational analysis of turn-taking in terms of two general mechanisms supporting joint actions: action prediction (e.g., recognizing the interlocutor's message and predicting the end of turn) and signaling (e.g., modifying one's own speech to make it more predictable and discriminable). We test the hypothesis that in a simulated conversational scenario dyads using these two mechanisms can recognize the utterances of their co-actors faster, which in turn permits them to give and take turns more efficiently. Furthermore, we discuss how turn-taking dynamics depend on the fact that agents cannot simultaneously use their internal models for both action (or messages) prediction and production, as these have different requirements-or, in other words, they cannot speak and listen at the same time with the same level of accuracy. Our results provide a computational-level characterization of turn-taking in terms of cognitive mechanisms of action prediction and signaling that are shared across various interaction and joint action domains.