Bound to remember: Infants show superior memory for objects presented at event boundaries

Temporal construal in sentence comprehension depends on linguistically encoded event structure

How events are ordered in time is one of the most fundamental pieces of information guiding our understanding of the world. Linguistically, this order is often not mentioned explicitly. Here, we propose that the mental construal of temporal order in language comprehension is based on event-structural properties. This prediction is based on a central distinction between states and events both in event perception and language: In perception, dynamic events are more salient than static states. In language, stative and eventive predicates also differ, both in their grammatical behavior and how they are processed. Consistent with our predictions, data from seven pre-registered video-sentence matching experiments, each conducted in English and German (total N = 674), show that people draw temporal inferences based on this difference: States precede events. Our findings not only arbitrate between different theories of temporal language comprehension; they also advance theoretical models of how two different cognitive capacities - event cognition and language - integrate to form a mental representation of time.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

Target detection does not influence temporal memory

Target detection has been found to enhance memory for concurrently presented stimuli under dual-task conditions. This "attentional boost effect" is reminiscent of findings in the event memory literature, where conditions giving rise to event boundaries have been shown to enhance memory for boundary items. Target detection commonly requires a working memory update (e.g., adding to a covert mental target count), which is also thought to be a key contributor to creating event boundaries. However, whether target detection impacts temporal memory in similar ways as event boundaries remains unknown, because these two parallel literatures have used different types of memory tests, making direct comparisons difficult. In a preregistered experiment with sequential Bayes factor design, we examined whether target detection influences temporal binding between items by inserting target and nontarget stimuli during encoding of trial-unique object images, and then comparing subsequent temporal order and distance memory for image pairs that span a target or nontarget. We found that target detection enhanced recognition memory for target trial images but had no effect on temporal binding between items. In a follow-up experiment, we showed that when the encoding task required updating of task set rather than target count, event segmentation-related temporal memory effects were observed. These results document that target detection as such does not disrupt inter-item associations in memory, and that attention orienting in the absence of updating task sets does not create event boundaries. This suggests a key distinction between declarative and procedural working memory updates in segmenting events in memory.

6-, 10-, and 12-month-olds remember complex dynamic events across 2 weeks

Whereas infants' ability to remember simple static material (e.g., pictures) has been documented extensively, we know surprisingly little about infants' memory of dynamic events (i.e., events unfolding in time) in the first year after birth. Although there is evidence to suggest that infants show some kind of sensitivity toward complex dynamic events (i.e., events involving agents and a storyline) as indicated by visual engagement in the first year after birth, 16- to 18-month-olds are hitherto the youngest infants documented to remember such material. Using a visual paired-comparison (VPC) task, in Experiment 1 we examined 6-, 10-, and 12-month-olds' (N = 108) ability to encode and remember cartoons involving complex dynamic events across 2 weeks. Results showed that all age groups remembered these cartoons. To investigate further the role of a complex storyline, in Experiment 2 we assessed the memory of 107 infants of the same age groups for similar cartoons but without coherent storyline information by scrambling the temporal presentation of the information in the cartoons. The results showed that the two youngest age groups did not remember this version. To our knowledge, this is the first experiment to document memory for such complex material in young infants using VPC.

Tracking what went where across toddlerhood: Feature-location bound object representations in 2- to 3-year-olds' working memory

Two experiments examined the development of the ability to encode, maintain, and update integrated representations of occluded objects' locations and featural identities in working memory across toddlerhood. Sixty-eight 28- to 40-month-old US toddlers (13 Asian or Pacific Islander, 6 Black, 48 White, 1 multiracial; 40 girls; tested between February 2015 and July 2017) tracked the locations of different color beads that were hidden simultaneously (Experiment 1) or sequentially (Experiment 2). Toddlers' ability to reliably store feature-location bound object representations in working memory varied as a function of age, memory load, and task demands. These results bridge a developmental gap between infancy and early childhood and provide new insights into sources of limitation and developmental change in children's early object representational capacities.

Neural event segmentation of continuous experience in human infants

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

Effects of spatial boundaries on episodic memory development

Children's spatial mapping starts out particularly sensitive to 3D wall-like boundaries and develops over early childhood to flexibly include other boundary types. This study investigated whether spatial boundaries influence children's episodic memory, as in adults, and whether this effect is modulated by boundary type. Eighty-one Korean children (34 girls, 36-84 months old) re-enacted a sequence of three discrete hiding events within a space containing one of three boundaries: 3D wall, aligned objects, or 2D line. Children's memory of events occurring on one side of the boundary developed earlier than those that crossed the boundary. At first, this interaction only applied to the 3D wall and extended to other boundary types with age, suggesting that children's changing spatial representations influence their episodic memory development.

Event boundaries shape temporal organization of memory by resetting temporal context

In memory, our continuous experiences are broken up into discrete events. Boundaries between events are known to influence the temporal organization of memory. However, how and through which mechanism event boundaries shape temporal order memory (TOM) remains unknown. Across four experiments, we show that event boundaries exert a dual role: improving TOM for items within an event and impairing TOM for items across events. Decreasing event length in a list enhances TOM, but only for items at earlier local event positions, an effect we term the local primacy effect. A computational model, in which items are associated to a temporal context signal that drifts over time but resets at boundaries captures all behavioural results. Our findings provide a unified algorithmic mechanism for understanding how and why event boundaries affect TOM, reconciling a long-standing paradox of why both contextual similarity and dissimilarity promote TOM.

Our memory is temporally organized, but our internal clock can be distorted. The authors demonstrate how environmental changes (termed event boundaries) affect memory for event order, and provide a computational model to explain these effects.

Predicting explicit memory for meaningful cartoons from visual paired comparison in infants and toddlers

We tested the memory of 18-, 33-, and 39-month-olds (N = 120) for dynamic stimulus material (simple cartoons) after 6 months in a visual paired comparison (VPC) task. We also tested the explicit recognition memory (ERM) for the same material. Only the oldest age group (39-month-olds) showed a significant visual (familiarity) preference at the test. Similarly, only the oldest group reliably chose the correct cartoon in the ERM test. Data from the VPC and ERM tasks did not correlate in any age group. However, we suggested a novel score (coined ΔVPC) measuring how much visual preference changes during the test phase in the VPC task. We found that this ΔVPC score (and vocabulary) predicted children's performance in the ERM task, whereas other potential predictors such as age and conventional novelty preference did not. We discuss the impact of these findings in relation to the development of implicit and explicit memory. Furthermore, we propose that VPC measures are associated with explicit memory only when the participants processed the stimuli conceptually. In such cases, we suggest that the ΔVPC score is an approximation of how demanding it is to construct the mental representation of the familiar stimulus during the test phase.

Events structure information accessibility less in children than adults

To manage the onslaught of continuously unfolding information in our complex environments, we adults are known to carve up our continuous experience into meaningful events, a process referred to as event segmentation. This segmentation directly shapes how our everyday experiences are construed: content experienced within an event is held mentally in an accessible state, which is then dropped after an event boundary. The greater accessibility of event-specific information has been shown to influence-at its most basic level-how information is processed and remembered. However, it is as yet unknown if accessibility is similarly influenced by event boundaries in children, who are still developing the working memory capacity and semantic knowledge thought to support event segmentation. Here, we tested seven- to nine-year-old children's and adults' recognition of objects experienced either within or across event boundaries of two cartoons. We found that children and adults were both more accurate and faster to correctly recognize objects that last occurred within events versus across event boundaries. We, however, additionally observed an interaction such that children's access to recent experience was less influenced by event boundaries than adults'. Thus, while the spontaneous segmentation of complex events emerges by middle childhood, event structure shapes the active contents of children's minds less reliably than adults'.

Putting effort into infant cognition

Working memory allows for the manipulation of information in support of ongoing tasks, providing a workspace for cognitive processes such as learning, reasoning, and decision making. How well working memory works depends, in part, on effort. Someone who pays attention at the right time and place will have better memory, and performance. In adult cognitive research studies, participants' devotion of maximal task-focused effort is often taken for granted, but in infant studies researchers cannot make that assumption. Here we showcase how pupillometry can provide an easy-to-obtain physiological measure of cognitive effort, allowing us to better understand infants' emerging abilities. In our work, we used pupillometry to measure trial-by-trial fluctuations of effort, establishing that, just as in adults, it influences how well infants could encode information in visual working memory. We hope that by using physiological measures such as pupil dilation, there will be a renewed effort to investigate the interaction between infants' attentive states and cognition.

Event Perception and Memory

Events make up much of our lived experience, and the perceptual mechanisms that represent events in experience have pervasive effects on action control, language use, and remembering. Event representations in both perception and memory have rich internal structure and connections one to another, and both are heavily informed by knowledge accumulated from previous experiences. Event perception and memory have been identified with specific computational and neural mechanisms, which show protracted development in childhood and are affected by language use, expertise, and brain disorders and injuries. Current theoretical approaches focus on the mechanisms by which events are segmented from ongoing experience, and emphasize the common coding of events for perception, action, and memory. Abetted by developments in eye-tracking, neuroimaging, and computer science, research on event perception and memory is moving from small-scale laboratory analogs to the complexity of events in the wild.

Transcending time in the brain: How event memories are constructed from experience

Our daily lives unfold continuously, yet when we reflect on the past, we remember those experiences as distinct and cohesive events. To understand this phenomenon, early investigations focused on how and when individuals perceive natural breakpoints, or boundaries, in ongoing experience. More recent research has examined how these boundaries modulate brain mechanisms that support long-term episodic memory. This work has revealed that a complex interplay between hippocampus and prefrontal cortex promotes the integration and separation of sequential information to help organize our experiences into mnemonic events. Here, we discuss how both temporal stability and change in one's thoughts, goals, and surroundings may provide scaffolding for these neural processes to link and separate memories across time. When learning novel or familiar sequences of information, dynamic hippocampal processes may work both independently from and in concert with other brain regions to bind sequential representations together in memory. The formation and storage of discrete episodic memories may occur both proactively as an experience unfolds. They may also occur retroactively, either during a context shift or when reactivation mechanisms bring the past into the present to allow integration. We also describe conditions and factors that shape the construction and integration of event memories across different timescales. Together these findings shed new light on how the brain transcends time to transform everyday experiences into meaningful memory representations.

Meaningful Memory? Eighteen-Month-Olds Only Remember Cartoons With a Meaningful Storyline

In two studies we investigated the importance of a storyline for remembering cartoons across a delay of 2 weeks in 18-month-old infants by means of the visual paired-comparison (VPC) paradigm. In Study 1 seventy-one 18-month-olds were tested using similar cartoons as in a recent study from our lab while varying the richness of the storyline information. In a VPC task half of the infants watched uncompromised versions of the cartoons used in the recent study (Storyline Condition), whereas the other half watched Pixelized versions of the cartoons (number of pixels reduced by 98% covering up the narrative, but leaving perceptual details, e.g., colors, movements, the same, and Pixelized Condition). Two weeks later they were presented with the familiar cartoon and a novel cartoon from the same version (Storyline or Pixelized) simultaneously, while being eye-tracked. Results showed that only the infants in the Storyline Condition remembered the target cartoon, thus suggesting that the storyline is important for memory. However, an alternative interpretation of the results could be that what made the infants in the Storyline Condition remember the target cartoon was not the storyline, but the static conceptual information of the objects and agents present in the cartoon (which was not visible in the Pixelized version). To test this possibility, a control study was created. In Study 2 thirty-six infants were therefore presented with a version of the cartoon in which we broke down the temporal presentation into 1 s segments and presented these out of order. This was done to preserve the static conceptual information (e.g., objects and agents) while still disturbing the storyline. Results showed that the infants in this condition still did not remember the target cartoon, suggesting that the meaningfulness of the storyline - and not only static conceptual information - is important for later memory.