Perceiving, remembering, and communicating structure in events

Temporal integration as an adaptive process in visual perception, attention, and working memory

I propose that temporal integration is ubiquitous in visual perception, because it serves an adaptive role. To support this idea, I draw together evidence from historically separated research fields that target different timescales. At one extreme, this concerns the detection and discrimination of successive stimuli within intervals of less than a quarter of a second. At an intermediate level, associated with attentional episodes, intervals between half a second up to a few seconds are considered. Finally, at the other extreme, this involves high-level, conceptual events across intervals of multiple seconds or even minutes. Across such varying intervals, the nature of temporal integration and the resultant perceptual events are clearly different. I nevertheless propose that temporal integration should be understood as a continuous process that serves a common adaptive goal: To maximize the amount of useful information, at minimal costs, tailored to the observer's current needs and circumstances. Emerging from this viewpoint are several research directions that might be pursued on the topic of temporal integration, and on its consequences for perception and memory.

Opportune moments for task interruptions: examining the cognitive mechanisms underlying interruption-timing effects

Introduction

Several studies showed that task interruptions at high mental workload moments are more harmful than task interruptions at low mental workload moments. In the present study, we used a theory-driven approach to define the mental workload during primary-task execution and to examine the effects of the interruption timing on primary-task performance.

Methods

Participants performed a primary task comprising a pre-defined sequence of six subtasks, with task interruptions occasionally occurring before the second, third, or fourth subtasks. Critically, the subtasks were organized either in two lag-2 repetition triplets or in two lag-2 switch triplets (e.g., ABA-CBC vs. CBA-CAB). This set-up allowed us to test two predictions about the effects of interruption timing on the resumption costs (i.e., the performance in subtasks following an interruption compared to the performance in the same subtask in non-interrupted primary tasks). First, we expected task interruptions before the fourth subtask being the less detrimental due to the presumed chunking of the six subtasks into two triplets. Second, in lag-2 switch triplets, task interruptions before the second and third subtasks were predicted to result in comparable resumption costs. In contrast, in lag-2 repetition triplets, task interruptions before the third subtask were hypothesized to be more disruptive than those before the second subtask. This is because the mental workload should be higher due to the need to overcome subtask inhibition.

Results

We found an interruption-timing effect with higher resumption costs for task interruptions occurring before the third subtask compared to interruptions before the second and the fourth subtasks. However, this effect did not differ across lag-2 repetition sequences and lag-2 switch sequences.

Discussion

These findings are discussed from a memory perspective and a context reconstruction perspective.

Aberrant neural event segmentation during a continuous social narrative in trauma-exposed older adolescents and young adults

Post-traumatic stress and major depressive disorders are associated with "overgeneral" autobiographical memory, or impaired recall of specific life events. Interpersonal trauma exposure, a risk factor for both conditions, may influence how symptomatic trauma-exposed (TE) individuals segment everyday events. The ability to parse experience into units (event segmentation) supports memory. Neural state transitions occur within a cortical hierarchy and play a key role in event segmentation, with regions like the occipital cortex, angular gyrus, and striatum involved in parsing event structure. We examined whether interpersonal trauma exposure was associated with alterations in the cortical hierarchy and striatal activity at neural state transitions in symptomatic TE versus healthy control (HC) individuals. Fifty older adolescents and young adults (29 TE, 21 HC) viewed the film "Partly Cloudy" during functional magnetic resonance imaging. A greedy-state boundary search algorithm assessed the optimal number of events, quality, and segmentation agreement of neural state transitions in the occipital cortex and angular gyrus. Striatal (nucleus accumbens, caudate, and putamen) activity was assessed at occipital and angular gyrus-evoked state transitions. Compared to HCs, TE participants displayed less occipital and greater angular gyrus-evoked optimal number of neural state transitions. TE participants also displayed lower quality of neural state segmentation solutions in occipital and angular cortices compared to HCs. Additionally, TE participants had less putamen activity at angular gyrus-evoked state transitions than HCs. This investigation provides neurobiological insights into aberrant event segmentation in symptomatic TE individuals, shedding light on mechanisms influencing overgeneral memory in trauma-related disorders.

Effects of Language Proficiency on Selective Attention Patterns at Segmenting Boundaries in English Audio Sentences

BACKGROUND/OBJECTIVES

Normative perceptual segmentation facilitates event perception, comprehension, and memory. Given that native English listeners' normative perceptual segmentation of English speech streams coexists with a highly selective attention pattern at segmentation boundaries, it is significant to test whether Chinese learners of English have a different attention pattern at boundaries, thereby checking whether they perform a normative segmentation.

METHODS

Thirty Chinese learners of English with relatively higher language proficiency (CLH) and 26 with relatively lower language proficiency (CLL) listened to a series of English audio sentences. Meanwhile, they were asked to press the key whenever a phonetic probe "ba" occurred. Response time to "ba" reflects the attention where "ba" is located at segmentation boundaries.

RESULTS

The results revealed that, (1) relative to native English listeners' highly selective attention pattern, the CLH group showed a relatively selective attention pattern, while the CLL group displayed a non-selective attention pattern. (2) Both the CLH group and natives had better recognition memory than the CLL group. (3) Both the CLH group and natives' attention at segmentation boundaries was not correlated with their memory for sentences, while the CLL group's attention at boundaries was correlated with memory.

CONCLUSIONS

These findings suggest that (1) Chinese learners of English did not perform a normative segmentation, which shows the effect of English proficiency on perceptual segmentation; (2) English proficiency has a superior effect on memory for sentences, while perceptual segmentation would come next to support memory by providing structure for memory construction if English proficiency is not high; (3) a comparison of attention patterns between Chinese learners and natives can provide a reference for potential intervention to rectify non-natives, thereby improving their perception of English speech streams.

The Relationship between Event Boundary Strength and Pattern Shifts across the Cortical Hierarchy during Naturalistic Movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: Phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multivoxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-participant agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile in both group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual participant level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. In addition, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Recall as a Window into Hippocampally Defined Events

We experience the present as a continuous stream of information, but often experience the past in parcels of unique events or episodes. Decades of research have helped to articulate how we perform this event segmentation in the moment, as well as how events and their boundaries influence what we later remember. More recently, neuroscientific research has suggested that the hippocampus plays a role at critical moments during event formation alongside its established role in enabling subsequent recall. Here, we review and explore the relationship between event processing and recall with the perspective that it can be uniquely characterized by the contributions of the hippocampus and its interactions with the rest of the brain. Specifically, we highlight a growing number of empirical studies suggesting that the hippocampus is important for processing events that have just ended, bridging the gap between the prior and current event, and influencing the contents and trajectories of recalled information. We also catalogue and summarize the multifaceted sets of findings concerning how recall is influenced by event structure. Lastly, we discuss several exciting directions for future research and how our understanding of events might be enriched by characterizing them in terms of the operations of different regions of the brain.

Turns around periodic spatial boundaries facilitate increasing event segmentation over time

Event segmentation is a neurocognitive process bridging perception and episodic memory. To our knowledge, almost all segmentation work is framed towards humans, yet evolutionarily conserved mechanisms in event cognition exist across species. Here, we addressed segmentation in a way that is applicable to humans and non-human animals, inspired by research in rats; specifically, the fragmentation of grid-cell spatial representations following the insertion of boundaries into an environment (forming a corridor maze). Participants indicated when they felt a meaningful unit of activity ended and another began, while watching an agent traverse from a first-person perspective. A virtual corridor maze (experiment 1) and two other mazes were used (experiment 2), with participants viewing/segmenting the same stimuli twice. We found that people segmented more during turns relative to corridors, with elevated segmentation occurring in discrete moments around turns. Interestingly, we also found that boundaries of the corridor maze facilitated an increase in segmentation within and across viewings. These results suggest that segmentation can be driven by recognized repeating activity that can become more meaningful over time, highlighting an important link between event segmentation and pattern separation that is relevant to many species in their formation of episodic-(like) memory.

Temporal construal in sentence comprehension depends on linguistically encoded event structure

How events are ordered in time is one of the most fundamental pieces of information guiding our understanding of the world. Linguistically, this order is often not mentioned explicitly. Here, we propose that the mental construal of temporal order in language comprehension is based on event-structural properties. This prediction is based on a central distinction between states and events both in event perception and language: In perception, dynamic events are more salient than static states. In language, stative and eventive predicates also differ, both in their grammatical behavior and how they are processed. Consistent with our predictions, data from seven pre-registered video-sentence matching experiments, each conducted in English and German (total N = 674), show that people draw temporal inferences based on this difference: States precede events. Our findings not only arbitrate between different theories of temporal language comprehension; they also advance theoretical models of how two different cognitive capacities - event cognition and language - integrate to form a mental representation of time.

Towards an ecologically valid naturalistic cognitive neuroscience of memory and event cognition

The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of "naturalistic" event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.

Separated hands further response-response binding effects

Action control is hierarchically organized. Multiple consecutive responses can be integrated into an event representation of higher order and can retrieve each other upon repetition, resulting in so-called response-response binding effects. Previous research indicates that the spatial separation of responses can affect how easily they can be cognitively separated. In this study, we introduced a barrier between the responding hands to investigate whether the spatial separation of two responses also influences response-response binding effects. In line with previous research on stimulus-response binding, we expected an increased separability of responses to result in stronger response-response binding effects when responding hands were separated by a barrier. We indeed found stronger response-response binding effects with separated hands. Results indicate that a more distinct representation of individual actions through increased separability might benefit the control of hierarchical actions.

Modeling human activity comprehension at human scale: prediction, segmentation, and categorization

Humans form sequences of event models-representations of the current situation-to predict how activity will unfold. Multiple mechanisms have been proposed for how the cognitive system determines when to segment the stream of behavior and switch from one active event model to another. Here, we constructed a computational model that learns knowledge about event classes (event schemas), by combining recurrent neural networks for short-term dynamics with Bayesian inference over event classes for event-to-event transitions. This architecture represents event schemas and uses them to construct a series of event models. This architecture was trained on one pass through 18 h of naturalistic human activities. Another 3.5 h of activities were used to test each variant for agreement with human segmentation and categorization. The architecture was able to learn to predict human activity, and it developed segmentation and categorization approaching human-like performance. We then compared two variants of this architecture designed to better emulate human event segmentation: one transitioned when the active event model produced high uncertainty in its prediction and the other transitioned when the active event model produced a large prediction error. The two variants learned to segment and categorize events, and the prediction uncertainty variant provided a somewhat closer match to human segmentation and categorization-despite being given no feedback about segmentation or categorization. These results suggest that event model transitioning based on prediction uncertainty or prediction error can reproduce two important features of human event comprehension.

The influence of event similarity on the detailed recall of autobiographical memories

Memories for life events are thought to be organised based on their relationships with one another, affecting the order in which events are recalled such that similar events tend to be recalled together. However, less is known about how detailed recall for a given event is affected by its associations to other events. Here, we used a cued autobiographical memory recall task where participants verbally recalled events corresponding to personal photographs. Importantly, we characterised the temporal, spatial, and semantic associations between each event to assess how similarity between adjacently cued events affected detailed recall. We found that participants provided more non-episodic details for cued events when the preceding event was both semantically similar and either temporally or spatially dissimilar. However, similarity along time, space, or semantics between adjacent events did not affect the episodic details recalled. We interpret this by considering organisation at the level of a life narrative, rather than individual events. When recalling a stream of personal events, we may feel obligated to justify seeming discrepancies between adjacent events that are semantically similar, yet simultaneously temporally or spatially dissimilar - to do so, we provide additional supplementary detail to help maintain global coherence across the events in our lives.

Bridging a Gap in Coherence: The Coordination of Comprehension Processes When Viewing Visual Narratives

Scene Perception and Event Comprehension Theory (SPECT) posits that understanding picture stories depends upon a coordination of two processes: (1) integrating new information into the current event model that is coherent with it (i.e., mapping) and (2) segmenting experiences into distinct event models (i.e., shifting). In two experiments, we investigated competing hypotheses regarding how viewers coordinate the mapping process of bridging inference generation and the shifting process of event segmentation by manipulating the presence/absence of Bridging Action pictures (i.e., creating coherence gaps) in wordless picture stories. The Computational Effort Hypothesis says that experiencing a coherence gap prompts event segmentation and the additional computational effort to generate bridging inferences. Thus, it predicted a positive relationship between event segmentation and explanations when Bridging Actions were absent. Alternatively, the Coherence Gap Resolution Hypothesis says that experiencing a coherence gap prompt generating a bridging inference to close the gap, which obviates segmentation. Thus, it predicted a negative relationship between event segmentation and the production of explanations. Replicating prior work, viewers were more likely to segment and generate explanations when Bridging Action pictures were absent than when they were present. Crucially, the relationship between explanations and segmentation was negative when Bridging Action pictures were absent, consistent with the Coherence Gap Resolution Hypothesis. Unexpectedly, the relationship was positive when Bridging Actions were present. The results are consistent with SPECT's assumption that mapping and shifting processes are coordinated, but how they are coordinated depends upon the experience of a coherence gap.

People can reliably detect action changes and goal changes during naturalistic perception

As a part of ongoing perception, the human cognitive system segments others' activities into discrete episodes (event segmentation). Although prior research has shown that this process is likely related to changes in an actor's actions and goals, it has not yet been determined whether untrained observers can reliably identify action and goal changes as naturalistic activities unfold, or whether the changes they identify are tied to visual features of the activity (e.g., the beginnings and ends of object interactions). This study addressed these questions by examining untrained participants' identification of action changes, goal changes, and event boundaries while watching videos of everyday activities that were presented in both first-person and third-person perspectives. We found that untrained observers can identify goal changes and action changes consistently, and these changes are not explained by visual change and the onsets or offsets of contact with objects. Moreover, the action and goal changes identified by untrained observers were associated with event boundaries, even after accounting for objective visual features of the videos. These findings suggest that people can identify action and goal changes consistently and with high agreement, that they do so by using sensory information flexibly, and that the action and goal changes they identify may contribute to event segmentation.

Neural connectivity patterns explain why adolescents perceive the world as moving slow

That younger individuals perceive the world as moving slower than adults is a familiar phenomenon. Yet, it remains an open question why that is. Using event segmentation theory, electroencephalogram (EEG) beamforming and nonlinear causal relationship estimation using artificial neural network methods, we studied neural activity while adolescent and adult participants segmented a movie. We show when participants were instructed to segment a movie into meaningful units, adolescents partitioned incoming information into fewer encapsulated segments or episodes of longer duration than adults. Importantly, directed communication between medial frontal and lower-level perceptual areas and between occipito-temporal regions in specific neural oscillation spectrums explained behavioral differences between groups. Overall, the study reveals that a different organization of directed communication between brain regions and inefficient transmission of information between brain regions are key to understand why younger people perceive the world as moving slow.

The relationship between event boundary strength and pattern shifts across the cortical hierarchy during naturalistic movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multi-voxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-subject agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile both in group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual subject level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. Additionally, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas, and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Hierarchical Context-Based Emotion Recognition With Scene Graphs

For a better intention inference, we often try to figure out the emotional states of other people in social communications. Many studies on affective computing have been carried out to infer emotions through perceiving human states, i.e., facial expression and body posture. Such methods are skillful in a controlled environment. However, it often leads to misestimation due to the deficiency of effective inputs in unconstrained circumstances, that is, where context-aware emotion recognition appeared. We take inspiration from the advanced reasoning pattern of humans in perceived emotion recognition and propose the hierarchical context-based emotion recognition method with scene graphs. We propose to extract three contexts from the image, i.e., the entity context, the global context, and the scene context. The scene context contains abstract information about entity labels and their relationships. It is similar to the information processing of the human visual sensing mechanism. After that, these contexts are further fused to perform emotion recognition. We carried out a bunch of experiments on the widely used context-aware emotion datasets, i.e., CAER-S, EMOTIC, and BOdy Language Dataset (BoLD). We demonstrate that the hierarchical contexts can benefit emotion recognition by improving the accuracy of the SOTA score from 84.82% to 90.83% on CAER-S. The ablation experiments show that hierarchical contexts provide complementary information. Our method improves the F1 score of the SOTA result from 29.33% to 30.24% (C-F1) on EMOTIC. We also build the image-based emotion recognition task with BoLD-Img from BoLD and obtain a better emotion recognition score (ERS) score of 0.2153.

Distinctive features of experiential time: Duration, speed and event density

William James's use of "time in passing" and "stream of thoughts" may be two sides of the same coin that emerge from the brain segmenting the continuous flow of information into discrete events. Herein, we investigated how the density of events affects two temporal experiences: the felt duration and speed of time. Using a temporal bisection task, participants classified seconds-long videos of naturalistic scenes as short or long (duration), or slow or fast (passage of time). Videos contained a varying number and type of events. We found that a large number of events lengthened subjective duration and accelerated the felt passage of time. Surprisingly, participants were also faster at estimating their felt passage of time compared to duration. The perception of duration scaled with duration and event density, whereas the felt passage of time scaled with the rate of change. Altogether, our results suggest that distinct mechanisms underlie these two experiential times.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Knowledge-based intervention improves older adult recognition memory for novel activity, but not event segmentation or temporal order memory

Although episodic memory declines with age, older adults are often able to make use of relevant knowledge to support episodic memory. More specifically, prior knowledge may support the perception of meaningful events through the process of event segmentation. We sought to test whether increasing older adults' knowledge for novel activities (i.e., Tai chi, making gyozas) would improve segmentation and memory. We conducted an online, pre-registered intervention in which eighty older adults were recruited based on being novices in each of the targeted activities. Participants completed segmentation and memory tests before and after being randomly assigned to one of two interactive virtual workshops (learning how to practice Tai chi or make gyozas). Each workshop consisted of two one-hour sessions during which an expert provided information about the activity and demonstrated it in a step-by-step fashion. We found that the intervention led to increased learning and recognition memory for the trained activity; however, there were no significant improvements in segmentation behavior, free recall, or memory of sequential information. These findings indicate that either more knowledge training is necessary to affect segmentation, or that segmentation is guided by perceptual features in the environment rather than one's conceptual understanding of the activity.

Impaired prediction of ongoing events in posttraumatic stress disorder

The ability to make accurate predictions about what is going to happen in the near future is critical for comprehension of everyday activity. However, predictive processing may be disrupted in Posttraumatic Stress Disorder (PTSD). Hypervigilance may lead people with PTSD to make inaccurate predictions about the likelihood of future danger. This disruption in predictive processing may occur not only in response to threatening stimuli, but also during processing of neutral stimuli. Therefore, the current study investigated whether PTSD was associated with difficulty making predictions about near-future neutral activity. Sixty-three participants with PTSD and 63 trauma controls completed two tasks, one testing explicit prediction and the other testing implicit prediction. Higher PTSD severity was associated with greater difficulty with predictive processing on both of these tasks. These results suggest that effective treatments to improve functional outcomes for people with PTSD may work, in part, by improving predictive processing.

Event characteristics help to explain the distribution of autobiographical memories over the first decade of life

The shape of the distribution of autobiographical memories over the first decade of life is characterised by a paucity of memories from the early years followed by a gradual increase in the number of surviving memories. Though many events and experiences from this period are forgotten, some are well remembered. To better understand why certain memories survive, we examined characteristics of events recalled by young adolescents (12 - to 14-year-olds), sampled over their first decade of life, and whether they predict consistency in recall. Characteristics were assessed via third-party observer ratings of event narratives. Events with more negative emotional valence, lower frequency of occurrence, and that were culturally shared were more likely to be recalled. The details of events with less positive emotional valence, shorter duration, fewer changes in location, and less predictability were more consistently recalled. The characteristics of reported events were largely similar across the decade, with significant differences in the representation of event characteristics only between earliest memories (1-5 years) and later periods (6-10 years and the previous year). The findings suggest that event characteristics play a role in how consistently events are remembered and how memories are distributed over the first decade of life.

Perception and action as viewed from the Theory of Event Coding: a multi-lab replication and effect size estimation of common experimental designs

The Theory of Event Coding (TEC) has influenced research on action and perception across the past two decades. It integrates several seminal empirical phenomena and it has continued to stimulate novel experimental approaches on the representational foundations of action control and perceptual experience. Yet, many of the most notable results surrounding TEC originate from an era of psychological research that relied on rather small sample sizes as judged by today's standards. This state hampers future research aiming to build on previous phenomena. We, therefore, provide a multi-lab re-assessment of the following six classical observations: response-effect compatibility, action-induced blindness, response-effect learning, stimulus-response binding, code occupation, and short-term response-effect binding. Our major goal is to provide precise estimates of corresponding effect sizes to facilitate future scientific endeavors. These effect sizes turned out to be considerably smaller than in the original reports, thus allowing for informed decisions on how to address each phenomenon in future work. Of note, the most relevant results of the original observations were consistently obtained in the present experiments as well.

Unsupervised Learning of Temporal Abstractions With Slot-Based Transformers

The discovery of reusable subroutines simplifies decision making and planning in complex reinforcement learning problems. Previous approaches propose to learn such temporal abstractions in an unsupervised fashion through observing state-action trajectories gathered from executing a policy. However, a current limitation is that they process each trajectory in an entirely sequential manner, which prevents them from revising earlier decisions about subroutine boundary points in light of new incoming information. In this work, we propose slot-based transformer for temporal abstraction (SloTTAr), a fully parallel approach that integrates sequence processing transformers with a slot attention module to discover subroutines in an unsupervised fashion while leveraging adaptive computation for learning about the number of such subroutines solely based on their empirical distribution. We demonstrate how SloTTAr is capable of outperforming strong baselines in terms of boundary point discovery, even for sequences containing variable amounts of subroutines, while being up to seven times faster to train on existing benchmarks.

Multi-view manifold learning of human brain-state trajectories

The complexity of the human brain gives the illusion that brain activity is intrinsically high-dimensional. Nonlinear dimensionality-reduction methods such as uniform manifold approximation and t-distributed stochastic neighbor embedding have been used for high-throughput biomedical data. However, they have not been used extensively for brain activity data such as those from functional magnetic resonance imaging (fMRI), primarily due to their inability to maintain dynamic structure. Here we introduce a nonlinear manifold learning method for time-series data-including those from fMRI-called temporal potential of heat-diffusion for affinity-based transition embedding (T-PHATE). In addition to recovering a low-dimensional intrinsic manifold geometry from time-series data, T-PHATE exploits the data's autocorrelative structure to faithfully denoise and unveil dynamic trajectories. We empirically validate T-PHATE on three fMRI datasets, showing that it greatly improves data visualization, classification, and segmentation of the data relative to several other state-of-the-art dimensionality-reduction benchmarks. These improvements suggest many potential applications of T-PHATE to other high-dimensional datasets of temporally diffuse processes.

Examining the Effect of Interruptions at Different Breakpoints and Frequencies Within a Task

OBJECTIVE

The purpose was to explore how event segmentation theory (EST) can be used to determine optimal moments for an interruption relying on hierarchical task analysis (HTA) to identify coarse and fine event boundaries.

BACKGROUND

Research on the effects of interruptions shows that they can be either disruptive or beneficial, depending on which aspects of an interruption are manipulated. Two important aspects that contribute to these conflicting results concern when and how often interruptions occur.

METHOD

Undergraduates completed a trip planning task divided into three subtasks. The within-subjects factor was interruption timing with three levels: none, coarse breakpoints, and fine breakpoints. The between-subjects factor was interruption frequency with two levels: one and three. The dependent measures included resumption lag, number of errors, mental workload, and frustration.

RESULTS

Participants took longer to resume the primary task and reported higher mental workload when interruptions occurred at fine breakpoints. The effect of interruptions at coarse breakpoints was similar to completing the task without interruption. Interruption frequency had no effect on performance; however, participants spent significantly longer attending to interruptions in the initial task, and within a task, the first and second interruptions were attended to significantly longer than the third interruption.

CONCLUSION

The disruptiveness of an interruption is tied to the point within the task hierarchy where it occurs.

APPLICATION

The performance cost associated with interruptions must be considered within the task structure. Interruptions occurring at coarse breakpoints may not be disruptive or have a negative effect on mental workload.

Representations of Complex Contexts: A Role for Hippocampus

The hippocampus plays a critical role in supporting episodic memory, in large part by binding together experiences and items with surrounding contextual information. At present, however, little is known about the roles of different hippocampal subfields in supporting this item-context binding. To address this question, we constructed a task in which items were affiliated with differing types of context-cognitive associations that vary at the local, item level and membership in temporally organized lists that linked items together at a global level. Participants made item recognition judgments while undergoing high-resolution fMRI. We performed voxel pattern similarity analyses to answer the question of how human hippocampal subfields represent retrieved information about cognitive states and the time at which a past event took place. As participants recollected previously presented items, activity patterns in the CA23DG subregion carried information about prior cognitive states associated with these items. We found no evidence to suggest reinstatement of information about temporal context at the level of list membership, but exploratory analyses revealed representations of temporal context at a coarse level in conjunction with representations of cognitive contexts. Results are consistent with characterizations of CA23DG as a critical site for binding together items and contexts in the service of memory retrieval.

The effects of domain knowledge and event structure on event processing

Research suggests that domain knowledge facilitates memory for domain-specific information through two mechanisms: differentiation, which involves the ability to identify meaningful, fine-grained details within a sequence, and unitization, which involves binding individual components from a sequence into functional wholes. This study investigated the extent to which individuals engaged in differentiation and unitization when parsing continuous events into discrete, meaningful units (i.e., event segmentation) and recalling them. Participants watched and segmented basketball videos. They then rewatched the videos and provided descriptions afterward. Videos were coded for the presence of higher order goals (A2 actions) and the individual sub-actions that comprised them (A1 actions). Results suggested that event segmentation behavior for participants with less knowledge was more aligned with changes in basic actions (A1 actions) than for participants with greater knowledge. When describing events, participants with greater knowledge were more likely than participants with less knowledge to use statements that reflected unitization.

Measuring event segmentation: An investigation into the stability of event boundary agreement across groups

People spontaneously divide everyday experience into smaller units (event segmentation). To measure event segmentation, studies typically ask participants to explicitly mark the boundaries between events as they watch a movie (segmentation task). Their data may then be used to infer how others are likely to segment the same movie. However, significant variability in performance across individuals could undermine the ability to generalize across groups, especially as more research moves online. To address this concern, we used several widely employed and novel measures to quantify segmentation agreement across different sized groups (n = 2-32) using data collected on different platforms and movie types (in-lab & commercial film vs. online & everyday activities). All measures captured nonrandom and video-specific boundaries, but with notable between-sample variability. Samples of 6-18 participants were required to reliably detect video-driven segmentation behavior within a single sample. As sample size increased, agreement values improved and eventually stabilized at comparable sample sizes for in-lab & commercial film data and online & everyday activities data. Stabilization occurred at smaller sample sizes when measures reflected (1) agreement between two groups versus agreement between an individual and group, and (2) boundary identification between small (fine-grained) rather than large (coarse-grained) events. These analyses inform the tailoring of sample sizes based on the comparison of interest, materials, and data collection platform. In addition to demonstrating the reliability of online and in-lab segmentation performance at moderate sample sizes, this study supports the use of segmentation data to infer when events are likely to be segmented.

A partially nested cortical hierarchy of neural states underlies event segmentation in the human brain

A fundamental aspect of human experience is that it is segmented into discrete events. This may be underpinned by transitions between distinct neural states. Using an innovative data-driven state segmentation method, we investigate how neural states are organized across the cortical hierarchy and where in the cortex neural state boundaries and perceived event boundaries overlap. Our results show that neural state boundaries are organized in a temporal cortical hierarchy, with short states in primary sensory regions, and long states in lateral and medial prefrontal cortex. State boundaries are shared within and between groups of brain regions that resemble well-known functional networks. Perceived event boundaries overlap with neural state boundaries across large parts of the cortical hierarchy, particularly when those state boundaries demarcate a strong transition or are shared between brain regions. Taken together, these findings suggest that a partially nested cortical hierarchy of neural states forms the basis of event segmentation.

The neural bases for timing of durations

Durations are defined by a beginning and an end, and a major distinction is drawn between durations that start in the present and end in the future ('prospective timing') and durations that start in the past and end either in the past or the present ('retrospective timing'). Different psychological processes are thought to be engaged in each of these cases. The former is thought to engage a clock-like mechanism that accurately tracks the continuing passage of time, whereas the latter is thought to engage a reconstructive process that utilizes both temporal and non-temporal information from the memory of past events. We propose that, from a biological perspective, these two forms of duration 'estimation' are supported by computational processes that are both reliant on population state dynamics but are nevertheless distinct. Prospective timing is effectively carried out in a single step where the ongoing dynamics of population activity directly serve as the computation of duration, whereas retrospective timing is carried out in two steps: the initial generation of population state dynamics through the process of event segmentation and the subsequent computation of duration utilizing the memory of those dynamics.

Retrieval context determines whether event boundaries impair or enhance temporal order memory

Meaningful changes in context create "event boundaries", segmenting continuous experience into distinct episodes in memory. A foundational finding in this literature is that event boundaries impair memory for the temporal order of stimuli spanning a boundary compared to equally spaced stimuli within an event. This seems surprising in light of intuitions about memory in everyday life, where the order of within-event experiences (did I have coffee before the first bite of bagel?) often seems more difficult to recall than the order of events per se (did I have breakfast or do the dishes first?). Here, we aimed to resolve this discrepancy by manipulating whether stimuli carried information about their encoding context during retrieval, as they often do in everyday life (e.g., bagel-breakfast). In Experiments 1 and 2, we show that stimuli inherently associated with a unique encoding context produce a "flipped" order memory effect, whereby temporal memory was superior for cross-boundary than within-event item pairs. In Experiments 3 and 4, we added context information at retrieval to a standard laboratory event memory protocol where stimuli were encoded in the presence of arbitrary context cues (colored frames). We found that whether temporal order memory for cross-boundary stimuli was enhanced or impaired relative to within-event items depended on whether the context was present or absent during the memory test. Taken together, we demonstrate that the effect of event boundaries on temporal memory is malleable, and determined by the availability of context information at retrieval.

Event Perception and Social Skills in Undergraduates with ADHD Symptoms

BACKGROUND

Event perception provides a promising, novel approach for investigating underlying cognitive mechanisms of the social impairment associated with symptoms of ADHD.

AIMS

The goal of this study was to establish the relationship among event perception, symptoms of ADHD, and social skills.

METHODOLOGY

Eighty-three undergraduates were recruited from the University of Ottawa first year psychology courses (38 with ADHD, 45 without ADHD). They performed an event perception task and completed self-report questionnaires assessing social functioning and symptoms of ADHD (The Social Skills-Improvement System and the Conner's CBRS-SR).

RESULTS

Bootstrapping mediation analyses revealed that symptoms of inattention and hyperactivity mediated the relationship between event perception and social skills. A model with predictor and mediator reversed was also tested, and was not significant, providing strength to the directionality of the relationships. Results highlight the applicability of event perception to understanding the association between social impairment and symptoms of ADHD.

Shared Representations in Athletes: Segmenting Action Sequences From Taekwondo Reveals Implicit Agreement

How do athletes represent actions from their sport? How are these representations structured and which knowledge is shared among experts in the same discipline? To address these questions, the event segmentation task was used. Experts in Taekwondo and novices indicated how they would subjectively split videos of Taekwondo form sequences into meaningful units. In previous research, this procedure was shown to unveil the structure of internal action representations and to be affected by sensorimotor knowledge. Without specific instructions on the grain size of segmentation, experts tended to integrate over longer episodes which resulted in a lower number of single units. Moreover, in accordance with studies in figure-skating and basketball, we expected higher agreement among experts on where to place segmentation marks, i.e., boundaries. In line with this hypothesis, significantly more overlap of boundaries was found within the expert group as compared to the control group. This was observed even though the interindividual differences in the selected grain size were huge and expertise had no systematic influence here. The absence of obvious goals or objects to structure Taekwondo forms underlines the importance of shared expert knowledge. Further, experts might have benefited from sensorimotor skills which allowed to simulate the observed actions more precisely. Both aspects may explain stronger agreement among experts even in unfamiliar Taekwondo forms. These interpretations are descriptively supported by the participants’ statements about features which guided segmentation and by an overlap of the group’s agreed boundaries with those of an experienced referee. The study shows that action segmentation can be used to provide insights into structure and content of action representations specific to experts. The mechanisms underlying shared knowledge among Taekwondoists and among experts in general are discussed on the background of current theoretic frameworks.

The role of social signals in segmenting observed actions in 18-month-old children

Learning about actions requires children to identify the boundaries of an action and its units. Whereas some action units are easily identified, parents can support children's action learning by adjusting the presentation and using social signals. However, currently, little is understood regarding how children use these signals to learn actions. In the current study, we investigate the possibility that communicative signals are a particularly suitable cue for segmenting events. We investigated this hypothesis by presenting 18-month-old children (N = 60) with short action sequences consisting of toy animals either hopping or sliding across a board into a house, but interrupting this two-step sequence either (a) using an ostensive signal as a segmentation cue, (b) using a non-ostensive segmentation cue and (c) without additional segmentation information between the actions. Marking the boundary using communicative signals increased children's imitation of the less salient sliding action. Imitation of the hopping action remained unaffected. Crucially, marking the boundary of both actions using a non-communicative control condition did not increase imitation of either action. Communicative signals might be particularly suitable in segmenting non-salient actions that would otherwise be perceived as part of another action or as non-intentional. These results provide evidence of the importance of ostensive signals at event boundaries in scaffolding children's learning.

Can the early visual processing of others' actions be related to social power and dominance?

Although goals often drive action understanding, this ability is also prone to important variability among individuals, which may have its origin in individual social characteristics. The present study aimed at evaluating the relationship between the tendency to prioritize goal information over grip information during early visual processing of action and several social dimensions. Visual processing of grip and goal information during action recognition was evaluated in 64 participants using the priming protocol developed by Decroix and Kalénine (Exp Brain Res 236(8):2411-2426, 2018). Object-directed action photographs were primed by photographs sharing the same goal and/or the same grip. The effects of goal and grip priming on action recognition were evaluated for different prime durations. The same participants further fulfilled questionnaires characterizing the way individuals deal with their social environment, namely their sense of social power, dominance, perspective taking, and construal level. At the group level, results confirmed greater goal than grip priming effects on action recognition for the shortest prime duration. Regression analyses between the pattern of response times in the action priming protocol and scores at the questionnaires further showed that the advantage of goal over grip priming was associated with higher sense of social power, and possibly to lower dominance. Overall, data confirm that observers tend to prioritize goal-related information when processing visual actions but further indicate that this tendency is sensitive to individual social characteristics. Results suggest that goal information may not always drive action understanding and point out the connection between low-level processing of observed actions and more general individual characteristics.

Constructing Expertise: Surmounting Performance Plateaus by Tasks, by Tools, and by Techniques

Acquiring expertise in a task is often thought of as an automatic process that follows inevitably with practice according to the log-log law (aka: power law) of learning. However, as Ericsson, Chase, and Faloon (1980) showed, this is not true for digit-span experts and, as we show, it is certainly not true for Tetris players at any level of expertise. Although some people may simply "twitch" faster than others, the limit to Tetris expertise is not raw keypress time but the techniques acquired by players that allow them to use the tools provided by the hardware and software to compensate for the game's relentlessly increasing drop speed. Unfortunately, these increases in drop speed between Tetris levels make performance plateaus very short and quickly followed by game death. Hence, a player's success at discovering, exploring, and practicing new techniques for the tasks of board preparation, board maintenance, optimal placement discovery, zoid rotation, lateral movement of zoids, and other tasks important to expertise in Tetris is limited. In this paper, we analyze data collected from 492 Tetris players to reveal the challenges they confronted while constructing expertise via the discovery of new techniques for gameplay at increasingly difficult levels of Tetris.

The extended present: an informational context for perception

Several previous authors have proposed a kind of specious or subjective present moment that covers a few seconds of recent information. This article proposes a new hypothesis about the subjective present, renamed the extended present, defined not in terms of time covered but as a thematically connected information structure held in working memory and in transiently accessible form in long-term memory. The three key features of the extended present are that information in it is thematically connected, both internally and to current attended perceptual input, it is organised in a hierarchical structure, and all information in it is marked with temporal information, specifically ordinal and duration information. Temporal boundaries to the information structure are determined by hierarchical structure processing and by limits on processing and storage capacity. Supporting evidence for the importance of hierarchical structure analysis is found in the domains of music perception, speech and language processing, perception and production of goal-directed action, and exact arithmetical calculation. Temporal information marking is also discussed and a possible mechanism for representing ordinal and duration information on the time scale of the extended present is proposed. It is hypothesised that the extended present functions primarily as an informational context for making sense of current perceptual input, and as an enabler for perception and generation of complex structures and operations in language, action, music, exact calculation, and other domains.

Events structure information accessibility less in children than adults

To manage the onslaught of continuously unfolding information in our complex environments, we adults are known to carve up our continuous experience into meaningful events, a process referred to as event segmentation. This segmentation directly shapes how our everyday experiences are construed: content experienced within an event is held mentally in an accessible state, which is then dropped after an event boundary. The greater accessibility of event-specific information has been shown to influence-at its most basic level-how information is processed and remembered. However, it is as yet unknown if accessibility is similarly influenced by event boundaries in children, who are still developing the working memory capacity and semantic knowledge thought to support event segmentation. Here, we tested seven- to nine-year-old children's and adults' recognition of objects experienced either within or across event boundaries of two cartoons. We found that children and adults were both more accurate and faster to correctly recognize objects that last occurred within events versus across event boundaries. We, however, additionally observed an interaction such that children's access to recent experience was less influenced by event boundaries than adults'. Thus, while the spontaneous segmentation of complex events emerges by middle childhood, event structure shapes the active contents of children's minds less reliably than adults'.

Predictive Uncertainty Underlies Auditory Boundary Perception

Anticipating the future is essential for efficient perception and action planning. Yet the role of anticipation in event segmentation is understudied because empirical research has focused on retrospective cues such as surprise. We address this concern in the context of perception of musical-phrase boundaries. A computational model of cognitive sequence processing was used to control the information-dynamic properties of tone sequences. In an implicit, self-paced listening task (N = 38), undergraduates dwelled longer on tones generating high entropy (i.e., high uncertainty) than on those generating low entropy (i.e., low uncertainty). Similarly, sequences that ended on tones generating high entropy were rated as sounding more complete (N = 31 undergraduates). These entropy effects were independent of both the surprise (i.e., information content) and phrase position of target tones in the original musical stimuli. Our results indicate that events generating high entropy prospectively contribute to segmentation processes in auditory sequence perception, independently of the properties of the subsequent event.

Hippocampal-Medial Prefrontal Event Segmentation and Integration Contribute to Episodic Memory Formation

How do we encode our continuous life experiences for later retrieval? Theories of event segmentation and integration suggest that the hippocampus binds separately represented events into an ordered narrative. Using a functional Magnetic Resonance Imaging (fMRI) movie watching-recall dataset, we quantified two types of neural similarities (i.e., "activation pattern" similarity and within-region voxel-based "connectivity pattern" similarity) between separate events during movie watching and related them to subsequent retrieval of events as well as retrieval of sequential order. We demonstrated that compared with forgotten events, successfully remembered events were associated with distinct "activation patterns" in the hippocampus and medial prefrontal cortex. In contrast, similar "connectivity pattern" between events were associated with memory formation and were also relevant for retaining events in the correct order. We applied the same approaches to an independent movie watching fMRI dataset as validation and highlighted again the role of hippocampal activation pattern and connectivity pattern in memory formation. We propose that distinct activation patterns represent neural segmentation of events, while similar connectivity patterns encode context information and, therefore, integrate events into a narrative. Our results provide novel evidence for the role of hippocampal-medial prefrontal event segmentation and integration in episodic memory formation of real-life experience.

Watching Movies Unfold, a Frame-by-Frame Analysis of the Associated Neural Dynamics

Our lives unfold as sequences of events. We experience these events as seamless, although they are composed of individual images captured in between the interruptions imposed by eye blinks and saccades. Events typically involve visual imagery from the real world (scenes), and the hippocampus is frequently engaged in this context. It is unclear, however, whether the hippocampus would be similarly responsive to unfolding events that involve abstract imagery. Addressing this issue could provide insights into the nature of its contribution to event processing, with relevance for theories of hippocampal function. Consequently, during magnetoencephalography (MEG), we had female and male humans watch highly matched unfolding movie events composed of either scene image frames that reflected the real world, or frames depicting abstract patterns. We examined the evoked neuronal responses to each image frame along the time course of the movie events. Only one difference between the two conditions was evident, and that was during the viewing of the first image frame of events, detectable across frontotemporal sensors. Further probing of this difference using source reconstruction revealed greater engagement of a set of brain regions across parietal, frontal, premotor, and cerebellar cortices, with the largest change in broadband (1–30 Hz) power in the hippocampus during scene-based movie events. Hippocampal engagement during the first image frame of scene-based events could reflect its role in registering a recognizable context perhaps based on templates or schemas. The hippocampus, therefore, may help to set the scene for events very early on.

Hunters and Gatherers of Pictures: Why Photography Has Become a Human Universal

Photography is ubiquitous worldwide. We analyzed why people take, share, and use personal photographs, independent of their specific cultural background. These behaviors are still poorly understood. Experimental research on them is scarce. Smartphone technology and social media have pushed the success of photography, but cannot explain it, as not all smartphone features are widely used just because they are available. We analyzed properties of human nature that have made taking and using photographs functional behaviors. We did this based on the four levels, which Nikolaas Tinbergen suggested for analyzing why animals behave in a particular way. Including findings from multiple disciplines, we developed a novel conceptual framework-the "Mental Utilization Hypothesis of Photography." It suggests that people adopt photography because it matches with core human mental mechanisms mainly from the social domain, and people use photography as a cognitive, primarily social coping strategy. Our framework comprises a range of testable predictions, provides a new theoretical basis for future empirical investigations into photography, and has practical implications. We conclude that photography has become a human universal, which is based on context-sensitive mental predispositions and differentiates itself in the social and societal environment.

Using the phenomenology of memory for recent events to bridge the gap between episodic and semantic memory

Public events such as celebrity news, tragedies, and political events are widely experienced. Initially at least, memories of these events are "episodic" in nature; however, these events are also stored in associative networks similar to the semantic organization of knowledge (N. R. Brown, 1990, Journal of Experimental Psychology: General, 119[3], 297-314). Thus, these memories provide a novel way of examining how episodically experienced events might become semanticized and integrated into the knowledge base. Younger and older adults rated their subjective memory strength for and answered questions about details of events occurring over the previous 12 years. Participants also rated their phenomenological experience using a modified remember/know paradigm, in which no instructions about usage of the terms were provided. Interestingly, remembered and known items were equal in terms of subjective strength. Know responses were highly accurate, and more so than remember responses. Older and younger adults performed similarly. Participants' own definitions of remember, know, and just familiar revealed that knowing is associated with retrieval from semantic memory, whereas remembering and just familiarity are more associated with event/episodic memory. These results suggest that memory for public events shares phenomenological features with both episodic/event memory and semantic memory. Public events thus allow researchers to examine the complex ways in which storage of novel information can be jointly maintained in both episodic and semantic memory.

Detecting neural state transitions underlying event segmentation

Segmenting perceptual experience into meaningful events is a key cognitive process that helps us make sense of what is happening around us in the moment, as well as helping us recall past events. Nevertheless, little is known about the underlying neural mechanisms of the event segmentation process. Recent work has suggested that event segmentation can be linked to regional changes in neural activity patterns. Accurate methods for identifying such activity changes are important to allow further investigation of the neural basis of event segmentation and its link to the temporal processing hierarchy of the brain. In this study, we introduce a new set of elegant and simple methods to study these mechanisms. We introduce a method for identifying the boundaries between neural states in a brain area and a complementary one for identifying the number of neural states. Furthermore, we present the results of a comprehensive set of simulations and analyses of empirical fMRI data to provide guidelines for reliable estimation of neural states and show that our proposed methods outperform the current state-of-the-art in the literature. This methodological innovation will allow researchers to make headway in investigating the neural basis of event segmentation and information processing during naturalistic stimulation.