The fraction of an action is more than a movement: Neural signatures of event segmentation in fMRI

Action Segmentation in the Brain: The Role of Object-Action Associations

Motion information has been argued to be central to the subjective segmentation of observed actions. Concerning object-directed actions, object-associated action information might as well inform efficient action segmentation and prediction. The present study compared the segmentation and neural processing of object manipulations and equivalent dough ball manipulations to elucidate the effect of object-action associations. Behavioral data corroborated that objective relational changes in the form of (un-)touchings of objects, hand, and ground represent meaningful anchor points in subjective action segmentation rendering them objective marks of meaningful event boundaries. As expected, segmentation behavior became even more systematic for the weakly informative dough. fMRI data were modeled by critical subjective, and computer-vision-derived objective event boundaries. Whole-brain as well as planned ROI analyses showed that object information had significant effects on how the brain processes these boundaries. This was especially pronounced at untouchings, that is, events that announced the beginning of the upcoming action and might be the point where competing predictions are aligned with perceptual input to update the current action model. As expected, weak object-action associations at untouching events were accompanied by increased biological motion processing, whereas strong object-action associations came with an increased contextual associative information processing, as indicated by increased parahippocampal activity. Interestingly, anterior inferior parietal lobule activity increased for weak object-action associations at untouching events, presumably because of an unrestricted number of candidate actions for dough manipulation. Our findings offer new insights into the significance of objects for the segmentation of action.

Updating predictions in a complex repertoire of actions and its neural representation

Even though actions we observe in everyday life seem to unfold in a continuous manner, they are automatically divided into meaningful chunks, that are single actions or segments, which provide information for the formation and updating of internal predictive models. Specifically, boundaries between actions constitute a hub for predictive processing since the prediction of the current action comes to an end and calls for updating of predictions for the next action. In the current study, we investigated neural processes which characterize such boundaries using a repertoire of complex action sequences with a predefined probabilistic structure. Action sequences consisted of actions that started with the hand touching an object (T) and ended with the hand releasing the object (U). These action boundaries were determined using an automatic computer vision algorithm. Participants trained all action sequences by imitating demo videos. Subsequently, they returned for an fMRI session during which the original action sequences were presented in addition to slightly modified versions thereof. Participants completed a post-fMRI memory test to assess the retention of original action sequences. The exchange of individual actions, and thus a violation of action prediction, resulted in increased activation of the action observation network and the anterior insula. At U events, marking the end of an action, increased brain activation in supplementary motor area, striatum, and lingual gyrus was indicative of the retrieval of the previously encoded action repertoire. As expected, brain activation at U events also reflected the predefined probabilistic branching structure of the action repertoire. At T events, marking the beginning of the next action, midline and hippocampal regions were recruited, reflecting the selected prediction of the unfolding action segment. In conclusion, our findings contribute to a better understanding of the various cerebral processes characterizing prediction during the observation of complex action repertoires.

Distinct brain network organizations between club players and novices under different difficulty levels

SIGNIFICANT

Chunk memory is one of the essential cognitive functions for high-expertise (HE) player to make efficient decisions. However, it remains unknown how the neural mechanisms of chunk memory processes mediate or alter chess players' performance when facing different opponents.

AIM

This study aimed at inspecting the significant brain networks associated with chunk memory, which would vary between club players and novices.

APPROACH

Functional networks and topological features of 20 club players (HE) and 20 novice players (LE) were compared at different levels of difficulty by means of functional near-infrared spectroscopy.

RESULTS

Behavioral performance indicated that the club player group was unaffected by differences in difficulty. Furthermore, the club player group demonstrated functional connectivity among the dorsolateral prefrontal cortex, the frontopolar cortex, the supramarginal gyrus, and the subcentral gyrus, as well as higher clustering coefficients and lower path lengths in the high-difficulty task.

CONCLUSIONS

The club player group illustrated significant frontal-parietal functional connectivity patterns and topological characteristics, suggesting enhanced chunking processes for improved chess performance.

Aberrant neural processing of event boundaries in persons with Parkinson's disease

The perception of everyday events implies the segmentation into discrete sub-events (i.e. event segmentation). This process is relevant for the prediction of upcoming events and for the recall of recent activities. It is thought to involve dopaminergic networks which are strongly compromised in Parkinson's disease (PD). Indeed, deficits of event segmentation have been previously shown in PD, but underlying neuronal mechanisms remain unknown. We therefore investigated 22 persons with PD and 22 age-matched healthy controls, who performed an event segmentation task with simultaneous electroencephalography (EEG). Both groups had to indicate by button press the beginning of sub-events within three movies showing persons performing everyday activities. The segmentation performance of persons with PD deviated significantly from that of controls. Neurophysiologically, persons with PD expressed reduced theta (4-7 Hz) activity around identified event boundaries compared to healthy controls. Together, these results point to disturbed event processing in PD. According to functions attributed to EEG activities in particular frequency ranges, the PD-related theta reduction could reflect impaired matching of perceptual input with stored event representations and decreased updating processes of event information in working memory and, thus, event boundary identification.

Relation between event segmentation and memory dysfunction in Parkinson's disease

The perception of everyday events is thought to imply the segmentation into discrete sub-events. Involvement of dopaminergic networks in this process could relate to particular problems of persons with Parkinson's disease (PD) to recall recent activities. In an event segmentation task, persons with PD and healthy controls had to indicate the beginning of sub-events within three movies showing persons performing everyday activities. In a subsequent recognition task, they should judge whether presented pictures of sub-events were part of the watched movies. In a final order memory task, they had to arrange pictures in the sequence in which they had occurred. With respect to the overall segmentation behavior, persons with PD diverged from healthy controls only in the most familiar of the three demonstrated everyday activities. Moreover, persons with PD compared to healthy controls showed generally worse event recognition and committed more errors in the order memory task. These memory deficits were the higher, the more the segmentation moved away from the 'normative' segmentation pattern identified in healthy controls. The findings suggest that dysfunctional structuring of sensory event information contributes to deficient event representations of ongoing everyday activities and recall problems of these recently perceived events in persons with PD.

What Happened When? Cerebral Processing of Modified Structure and Content in Episodic Cueing

Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.

Shared Representations in Athletes: Segmenting Action Sequences From Taekwondo Reveals Implicit Agreement

How do athletes represent actions from their sport? How are these representations structured and which knowledge is shared among experts in the same discipline? To address these questions, the event segmentation task was used. Experts in Taekwondo and novices indicated how they would subjectively split videos of Taekwondo form sequences into meaningful units. In previous research, this procedure was shown to unveil the structure of internal action representations and to be affected by sensorimotor knowledge. Without specific instructions on the grain size of segmentation, experts tended to integrate over longer episodes which resulted in a lower number of single units. Moreover, in accordance with studies in figure-skating and basketball, we expected higher agreement among experts on where to place segmentation marks, i.e., boundaries. In line with this hypothesis, significantly more overlap of boundaries was found within the expert group as compared to the control group. This was observed even though the interindividual differences in the selected grain size were huge and expertise had no systematic influence here. The absence of obvious goals or objects to structure Taekwondo forms underlines the importance of shared expert knowledge. Further, experts might have benefited from sensorimotor skills which allowed to simulate the observed actions more precisely. Both aspects may explain stronger agreement among experts even in unfamiliar Taekwondo forms. These interpretations are descriptively supported by the participants’ statements about features which guided segmentation and by an overlap of the group’s agreed boundaries with those of an experienced referee. The study shows that action segmentation can be used to provide insights into structure and content of action representations specific to experts. The mechanisms underlying shared knowledge among Taekwondoists and among experts in general are discussed on the background of current theoretic frameworks.

Touching events predict human action segmentation in brain and behavior

Recognizing the actions of others depends on segmentation into meaningful events. After decades of research in this area, it remains still unclear how humans do this and which brain areas support underlying processes. Here we show that a computer vision-based model of touching and untouching events can predict human behavior in segmenting object manipulation actions with high accuracy. Using this computational model and functional Magnetic Resonance Imaging (fMRI), we pinpoint the neural networks underlying this segmentation behavior during an implicit action observation task. Segmentation was announced by a strong increase of visual activity at touching events followed by the engagement of frontal, hippocampal and insula regions, signaling updating expectation at subsequent untouching events. Brain activity and behavior show that touching-untouching motifs are critical features for identifying the key elements of actions including object manipulations.

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

Predicting other people’s upcoming action is key to successful social interactions. Previous studies have started to disentangle the various sources of information that action observers exploit, including objects, movements, contextual cues and features regarding the acting person’s identity. We here focus on the role of static and dynamic inter-object spatial relations that change during an action. We designed a virtual reality setup and tested recognition speed for ten different manipulation actions. Importantly, all objects had been abstracted by emulating them with cubes such that participants could not infer an action using object information. Instead, participants had to rely only on the limited information that comes from the changes in the spatial relations between the cubes. In spite of these constraints, participants were able to predict actions in, on average, less than 64% of the action’s duration. Furthermore, we employed a computational model, the so-called enriched Semantic Event Chain (eSEC), which incorporates the information of different types of spatial relations: (a) objects’ touching/untouching, (b) static spatial relations between objects and (c) dynamic spatial relations between objects during an action. Assuming the eSEC as an underlying model, we show, using information theoretical analysis, that humans mostly rely on a mixed-cue strategy when predicting actions. Machine-based action prediction is able to produce faster decisions based on individual cues. We argue that human strategy, though slower, may be particularly beneficial for prediction of natural and more complex actions with more variable or partial sources of information. Our findings contribute to the understanding of how individuals afford inferring observed actions’ goals even before full goal accomplishment, and may open new avenues for building robots for conflict-free human-robot cooperation.

Expert Event Segmentation of Dance Is Genre-Specific and Primes Verbal Memory

By chunking continuous streams of action into ordered, discrete, and meaningful units, event segmentation facilitates motor learning. While expertise in the observed repertoire reduces the frequency of event borders, generalization of this effect to unfamiliar genres of dance and among other sensorimotor experts (musicians, athletes) remains unknown, and was the first aim of this study. Due to significant overlap in visuomotor, language, and memory processing brain networks, the second aim of this study was to investigate whether visually priming expert motor schemas improves memory for words related to one’s expertise. A total of 112 participants in six groups (ballet, Bharatanatyam, and “other” dancers, athletes, musicians, and non-experts) segmented a ballet dance, a Bharatanatyam dance, and a non-dance control sequence. To test verbal memory, participants performed a retrieval-induced forgetting task between segmentation blocks. Dance, instrument, and sport word categories were included to probe the second study aim. Results of the event segmentation paradigm clarify that previously-established expert segmentation effects are specific to familiar genres of dance, and do not transfer between different types of experts or to non-dance sequences. Greater recall of dance category words among ballet and Bharatanatyam dancers provides novel evidence for improved verbal memory primed by activating familiar sensorimotor representations.

Episodes, events, and models

We describe a novel computational theory of how individuals segment perceptual information into representations of events. The theory is inspired by recent findings in the cognitive science and cognitive neuroscience of event segmentation. In line with recent theories, it holds that online event segmentation is automatic, and that event segmentation yields mental simulations of events. But it posits two novel principles as well: first, discrete episodic markers track perceptual and conceptual changes, and can be retrieved to construct event models. Second, the process of retrieving and reconstructing those episodic markers is constrained and prioritized. We describe a computational implementation of the theory, as well as a robotic extension of the theory that demonstrates the processes of online event segmentation and event model construction. The theory is the first unified computational account of event segmentation and temporal inference. We conclude by demonstrating now neuroimaging data can constrain and inspire the construction of process-level theories of human reasoning.

Intact action segmentation in Parkinson's disease: Hypothesis testing using a novel computational approach

Action observation is known to trigger predictions of the ongoing course of action and thus considered a hallmark example for predictive perception. A related task, which explicitly taps into the ability to predict actions based on their internal representations, is action segmentation; the task requires participants to demarcate where one action step is completed and another one begins. It thus benefits from a temporally precise prediction of the current action. Formation and exploitation of these temporal predictions of external events is now closely associated with a network including the basal ganglia and prefrontal cortex. Because decline of dopaminergic innervation leads to impaired function of the basal ganglia and prefrontal cortex in Parkinson's disease (PD), we hypothesised that PD patients would show increased temporal variability in the action segmentation task, especially under medication withdrawal (hypothesis 1). Another crucial aspect of action segmentation is its reliance on a semantic representation of actions. There is no evidence to suggest that action representations are substantially altered, or cannot be accessed, in non-demented PD patients. We therefore expected action segmentation judgments to follow the same overall patterns in PD patients and healthy controls (hypothesis 2), resulting in comparable segmentation profiles. Both hypotheses were tested with a novel classification approach. We present evidence for both hypotheses in the present study: classifier performance was slightly decreased when it was tested for its ability to predict the identity of movies segmented by PD patients, and a measure of normativity of response behaviour was decreased when patients segmented movies under medication-withdrawal without access to an episodic memory of the sequence. This pattern of results is consistent with hypothesis 1. However, the classifier analysis also revealed that responses given by patients and controls create very similar action-specific patterns, thus delivering evidence in favour hypothesis 2. In terms of methodology, the use of classifiers in the present study allowed us to establish similarity of behaviour across groups (hypothesis 2). The approach opens up a new avenue that standard statistical methods often fail to provide and is discussed in terms of its merits to measure hypothesised similarities across study populations.

Dissociating dynamic probability and predictability in observed actions-an fMRI study

The present fMRI study investigated whether human observers spontaneously exploit the statistical structure underlying continuous action sequences. In particular, we tested whether two different statistical properties can be distinguished with regard to their neural correlates: an action step's predictability and its probability. To assess these properties we used measures from information theory. Predictability of action steps was operationalized by its inverse, conditional entropy, which combines the number of possible action steps with their respective probabilities. Probability of action steps was assessed using conditional surprisal, which increases with decreasing probability. Participants were trained in an action observation paradigm with video clips showing sequences of 9–33 s length with varying numbers of action steps that were statistically structured according to a Markov chain. Behavioral tests revealed that participants implicitly learned this statistical structure, showing that humans are sensitive toward these probabilistic regularities. Surprisal (lower probability) enhanced the BOLD signal in the anterior intraparietal sulcus. In contrast, high conditional entropy, i.e., low predictability, was correlated with higher activity in dorsomedial prefrontal cortex, orbitofrontal gyrus, and posterior intraparietal sulcus. Furthermore, we found a correlation between the anterior hippocampus' response to conditional entropy with the extent of learning, such that the more participants had learnt the structure, the greater the magnitude of hippocampus activation in response to conditional entropy. Findings show that two aspects of predictions can be dissociated: an action's predictability is reflected in a top-down modulation of attentional focus, evident in increased fronto-parietal activation. In contrast, an action's probability depends on the identity of the stimulus itself, resulting in bottom-up driven processing costs in the parietal cortex.

The tool in the brain: apraxia in ADL. Behavioral and neurological correlates of apraxia in daily living

Humans differ from other animals in the way they can skilfully and precisely operate or invent tools to facilitate their everyday life. Tools have dominated our home, travel and work environment, becoming an integral step for our motor skills development. What happens when the part of the brain responsible for tool use is damaged in our adult life due to a cerebrovascular accident? How does daily life change when we lose the previously mastered ability to make use of the objects around us? How do patients suffering from compromised tool use cope with food preparation, personal hygiene, grooming, housework, or use of home appliances? In this literature review we present a state of the art for single and multiple tool use research, with a focus on the impact that apraxia (impaired ability to perform tool-based actions) and action disorganization syndrome (ADS; impaired ability to carry out multi-step actions) have on activities of daily living (ADL). Firstly, we summarize the behavioral studies investigating the impact of apraxia and other comorbidity syndromes, such as neglect or visual extinction, on ADL. We discuss the hallmarks of the compromised tool use in terms of the sequencing of action steps, conceptual errors committed, spatial motor control, and temporal organization of the movement. In addition, we present an up-to-date overview of the neuroimaging and lesion analyses studies that provide an insight into neural correlates of tool use in the human brain and functional changes in the neural organization following a stroke, in the context of ADL. Finally we discuss the current practice in neurorehabilitation of ADL in apraxia and ADS aiming at increasing patients' independence.

Event-related potentials to intact and disrupted actions in children and adults

The current research used event-related potentials (ERPs) to investigate neurophysiological responses to intact and disrupted actions embedded within an event in children and adults. Responses were recorded as children (24-month-olds) and adults observed a relatively novel event composed of three actions. In one condition pauses were inserted at intact boundaries (i.e., at the endpoint of each action), whereas in the other condition they were inserted at breakpoints that disrupted the action (i.e., in the middle of each action). Evoked responses revealed differences across conditions in both groups; disrupted actions elicited a prolonged negative slow wave from 100 to 700 ms in children, whereas adults demonstrated two distinct negative peaks between 50-150 and 250-350 ms. These findings contribute the first electrophysiological evidence that children readily detect disruptions to ongoing events by the end of the second year, even with limited exposure to the event itself. Furthermore, they suggest that adults rely on two distinct mechanisms when processing novel events. Results are discussed in relation to the role of perceptual and conceptual levels of analysis in the development of action processing.

Surprisingly correct: unexpectedness of observed actions activates the medial prefrontal cortex

Not only committing errors, but also observing errors has been shown to activate the dorsal medial prefrontal cortex, particularly BA 8 and adjacent rostral cingulate zone (RCZ). Currently, there is a debate on whether this activity reflects a response to the incorrectness of the committed action or to its unexpectedness. This article reports two studies investigating whether activity in BA 8/RCZ is due to the unexpectedness of observed errors or the incorrectness of the specific observed action. Both studies employed an action observation paradigm reliant on the observation of an actor tying sailing knots. The reported behavioral experiment delivered evidence that the paradigm successfully induced the expectation of incorrect actions as well as the expectation of correct actions. The functional magnetic resonance imaging study revealed that unexpectedly correct as well as unexpectedly incorrect actions activate the BA 8/RCZ. The same result was confirmed for a coordinate in the vicinity that has been previously reported to be activated in separate studies either by the error observation or by the unexpectedness of committed errors, and has been associated with the error-related negativity. The present results suggest that unexpectedness has an impact on the medial prefrontal correlate of observed errors.