Neural representations of naturalistic events are updated as our understanding of the past changes

Memory updating and the structure of event representations

People form memories of specific events and use those memories to make predictions about similar new experiences. Living in a dynamic environment presents a challenge: How does one represent valid prior events in memory while encoding new experiences when things change? There is evidence for two seemingly contradictory classes of mechanism: One differentiates outdated event features by making them less similar or less accessible than updated event features. The other integrates updated features of new events with outdated memories, and the relationship between them, into a structured representation. Integrative encoding may occur when changed events trigger inaccurate predictions based on remembered prior events. We propose that this promotes subsequent recollection of events and their order, enabling adaptation to environmental changes.

Unraveling the neural dichotomy of consensus and idiosyncratic experiences in short video viewing

Human experiences are inherently shaped by individual perspectives, leading to diverse interpretations of the same events. However, shared activities, such as communal film watching or sports viewing, underscore the dual nature of these experiences: collective joy arises through social interactions, while individual emotional responses are influenced by personal preferences. The neural mechanisms underlying this interplay between shared and idiosyncratic experiences, particularly in the context of reward processing, remain insufficiently explored. In this study, we investigated the neural basis of both communal enjoyment and individual anticipatory responses during short video viewing. Using functional MRI, we measured brain activity in participants as they watched 90 short videos and provided ratings of their reward expectations and experienced pleasure. By integrating intersubject correlation (ISC) and individual-specific analyses, we identified shared and unique neural activity patterns. Our findings reveal that synchronized activity within the default mode network (DMN) and reward-related regions underlies shared experiences of collective pleasure. In contrast, distinct activations in the frontal cortex and caudate nucleus were associated with personal preferences and reward anticipation, highlighting a neural dichotomy between communal enjoyment and individualized reward processing during digital media engagement.

Linguistic coupling between neural systems for speech production and comprehension during real-time dyadic conversations

The core use of human language is communicating complex ideas from one mind to another in everyday conversations. In conversations, comprehension and production processes are intertwined, as speakers soon become listeners, and listeners become speakers. Nonetheless, the neural systems underlying these faculties are typically studied in isolation using paradigms that cannot fully engage our capacity for interactive communication. Here, we used an fMRI hyperscanning paradigm to measure neural activity simultaneously in pairs of subjects engaged in real-time, interactive conversations. We used contextual word embeddings from a large language model to quantify the linguistic coupling between production and comprehension systems within and across individual brains. We found a highly overlapping network of regions involved in both production and comprehension spanning much of the cortical language network. Our findings reveal that shared representations for both processes extend beyond the language network into areas associated with social cognition. Together, these results suggest that the specialized neural systems for speech perception and production align on a common set of linguistic features encoded in a broad cortical network for language and communication.

Fast-timescale hippocampal processes bridge between slowly unfurling neocortical states during memory search

Prior behavioral work showed that event structure plays a key role in our ability to mentally search through memories of continuous naturalistic experience. We hypothesized that, neurally, this memory search process involves a division of labor between slowly unfurling neocortical states representing event knowledge and fast hippocampal-neocortical communication that supports retrieval of new information at transitions between events. To test this, we tracked slow neural state-patterns in a sample of ten patients undergoing intracranial electroencephalography as they viewed a movie and then searched their memories in a structured naturalistic interview. As patients answered questions ("after X, when does Y happen next?"), state-patterns from movie-viewing were reinstated in neocortex; during memory-search, states unfurled in a forward direction. Moments of state-transition were marked by low-frequency power decreases in cortex and preceded by power decreases in hippocampus that correlated with reinstatement. Connectivity-analysis revealed information-flow from hippocampus to cortex underpinning state-transitions. Together, these results support our hypothesis that fast hippocampal processes bridge between slow neocortical states during memory search.

A core set of neural states underlying naturalistic memory reactivation in the posterior medial cortex

In the posterior midline default mode network, spatial activity patterns similar to those during the initial experience are reactivated during the successful recall of past events. Prior studies have shown that these event-specific activity patterns are consistent across individuals recalling a shared experience, suggesting that common functional responses underlying episodic recall do exist. However, the spatial organization of function during episodic encoding and subsequent recall, especially in the absence of external stimuli, remains poorly understood. To address this, we leverage fMRI data collected during the encoding and recall of naturalistic movies to identify a core set of neural states in the posterior medial cortex. These states are stimulus-locked, reactivated during recall, and have a shared spatial organization across brains (i.e., individuals). We show that a surprisingly small number of these states (16 states across hemispheres) is sufficient to achieve the same levels of reactivation in the posterior medial cortex as when using the standard methods of the field. Additionally, these states are linked to actions and social-affective features of events in the movies. Our findings elucidate the properties of a common, spatially organized code within the posterior default mode network which appears during natural recollection of memories.

Coordinated representations for naturalistic memory encoding and retrieval in hippocampal neural subspaces

Our naturalistic experiences are organized into memories through multiple processes, including novelty encoding, memory formation, and retrieval. However, the neural mechanisms coordinating these processes remain elusive. Using fMRI data acquired during movie viewing and subsequent narrative recall, we examine hippocampal neural subspaces associated with distinct memory processes and characterized their relationships. We quantify novelty in character co-occurrences and the valence of relationships and estimate event memorability. Within the hippocampus, the novelty subspaces encoding each type exhibit partial overlap, and these overlapping novelty subspaces align with the subspace involved in memorability. Notably, following event boundaries, hippocampal states within these subspaces align inversely along a shared coding axis, predicting subsequent recall performance. This novelty-memorability alignment is selectively observed during encoding but not during retrieval. Finally, the identified functional subspaces reflect the intrinsic functional organization of the hippocampus. Our findings offer insights into how the hippocampus dynamically coordinates representations underlying memory encoding and retrieval at the population level to transform ongoing experiences into enduring memories.

Narrative 'twist' shifts within-individual neural representations of dissociable story features

Given the same external input, one's understanding of that input can differ based on internal contextual knowledge. Where and how does the brain represent latent belief frameworks that interact with incoming sensory information to shape subjective interpretations? In this study, participants listened to the same auditory narrative twice, with a plot twist in the middle that dramatically shifted their interpretations of the story. Using a robust within-subject whole-brain approach, we leveraged shifts in neural activity between the two listens to identify where latent interpretations are represented in the brain. We considered the narrative in terms of its hierarchical structure, examining how global situation models and their subcomponents-namely, episodes and characters-are represented, finding that they rely on partially distinct sets of brain regions. Results suggest that our brains represent narratives hierarchically, with individual narrative elements being distinct and dynamically updated as a part of changing interpretations of incoming information.

COMMON AND DISTINCT NEURAL CORRELATES OF SOCIAL INTERACTION PERCEPTION AND THEORY OF MIND

Social cognition spans from perceiving agents and their interactions to making inferences based on theory of mind (ToM). Despite their frequent co-occurrence in real life, the commonality and distinction between social interaction perception and ToM at behavioral and neural levels remain unclear. Here, participants (N = 231) provided moment-by-moment ratings of four text and four audio narratives on social interactions and ToM engagement. Social interaction and ToM ratings were reliable (split-half r = .98 and .92, respectively) but only modestly correlated across time (r = .32). In a second sample (N = 90), we analyzed co-variation between normative social interaction and ToM ratings and functional magnetic resonance (fMRI) activity during narrative reading (text) and listening (audio). Social interaction perception and ToM activity maps generalized across text and audio presentation (r = .83 and .57 between unthresholded t maps, respectively). When ToM was held constant, merely perceiving social interactions activated all regions canonically associated with ToM under both modalities (FDR q < .01), including temporoparietal junction, superior temporal sulcus, medial prefrontal cortex, and precuneus. ToM activated these regions as well, indicating a shared, modality-general system for social interaction perception and ToM. Furthermore, ToM uniquely engaged lateral occipitotemporal cortex, left anterior intraparietal sulcus, and right premotor cortex. These results imply that perceiving social interactions automatically engages regions implicated in mental state inferences. In addition, ToM is distinct from social interaction perception in its recruitment of regions associated with higher-level cognitive processes, including action understanding and executive functions.

The "Naturalistic Free Recall" dataset: four stories, hundreds of participants, and high-fidelity transcriptions

The "Naturalistic Free Recall" dataset provides transcribed verbal recollections of four spoken narratives collected from 229 participants. Each participant listened to two stories, varying in duration from approximately 8 to 13 minutes, recorded by different speakers. Subsequently, participants were tasked with verbally recalling the narrative content in as much detail as possible and in the correct order. The dataset includes high-fidelity, time-stamped text transcripts of both the original narratives and participants' recollections. To validate the dataset, we apply a previously published automated method to score memory performance for narrative content. Using this approach, we extend effects traditionally observed in classic list-learning paradigms. The analysis of narrative contents and its verbal recollection presents unique challenges compared to controlled list-learning experiments. To facilitate the use of these rich data by the community, we offer an overview of recent computational methods that can be used to annotate and evaluate key properties of narratives and their recollections. Using advancements in machine learning and natural language processing, these methods can help the community understand the role of event structure, discourse properties, prediction error, high-level semantic features (e.g., idioms, humor), and more. All experimental materials, code, and data are publicly available to facilitate new advances in understanding human memory.

Top-down attention shifts behavioral and neural event boundaries in narratives with overlapping event scripts

Understanding and remembering the complex experiences of everyday life relies critically on prior schematic knowledge about how events in our world unfold over time. How does the brain construct event representations from a library of schematic scripts, and how does activating a specific script impact the way that events are segmented in time? We developed a novel set of 16 audio narratives, each of which combines one of four location-relevant event scripts (restaurant, airport, grocery store, and lecture hall) with one of four socially relevant event scripts (breakup, proposal, business deal, and meet cute), and presented them to participants in an fMRI study and a separate online study. Responses in the angular gyrus, parahippocampal gyrus, and subregions of the medial prefrontal cortex (mPFC) were driven by scripts related to both location and social information, showing that these regions can track schematic sequences from multiple domains. For some stories, participants were primed to attend to one of the two scripts by training them to listen for and remember specific script-relevant episodic details. Activating a location-related event script shifted the timing of subjective event boundaries to align with script-relevant changes in the narratives, and this behavioral shift was mirrored in the timing of neural responses, with mPFC event boundaries (identified using a hidden Markov model) aligning to location-relevant rather than socially relevant boundaries when participants were location primed. Our findings demonstrate that neural event dynamics are actively modulated by top-down goals and provide new insight into how narrative event representations are constructed through the activation of temporally structured prior knowledge.

The causal structure and computational value of narratives

Many human behavioral and brain imaging studies have used narratively structured stimuli (e.g., written, audio, or audiovisual stories) to better emulate real-world experience in the laboratory. However, narratives are a special class of real-world experience, largely defined by their causal connections across time. Much contemporary neuroscience research does not consider this key property. We review behavioral and neuroscientific work that speaks to how causal structure shapes comprehension of and memory for narratives. We further draw connections between this work and reinforcement learning, highlighting how narratives help link causes to outcomes in complex environments. By incorporating the plausibility of causal connections between classes of actions and outcomes, reinforcement learning models may become more ecologically valid, while simultaneously elucidating the value of narratives.

The language network as a natural kind within the broader landscape of the human brain

Language behaviour is complex, but neuroscientific evidence disentangles it into distinct components supported by dedicated brain areas or networks. In this Review, we describe the 'core' language network, which includes left-hemisphere frontal and temporal areas, and show that it is strongly interconnected, independent of input and output modalities, causally important for language and language-selective. We discuss evidence that this language network plausibly stores language knowledge and supports core linguistic computations related to accessing words and constructions from memory and combining them to interpret (decode) or generate (encode) linguistic messages. We emphasize that the language network works closely with, but is distinct from, both lower-level - perceptual and motor - mechanisms and higher-level systems of knowledge and reasoning. The perceptual and motor mechanisms process linguistic signals, but, in contrast to the language network, are sensitive only to these signals' surface properties, not their meanings; the systems of knowledge and reasoning (such as the system that supports social reasoning) are sometimes engaged during language use but are not language-selective. This Review lays a foundation both for in-depth investigations of these different components of the language processing pipeline and for probing inter-component interactions.

The memory trace of an intrusive trauma-analog episode

Intrusive memories are a core symptom of posttraumatic stress disorder. Compared with memories of everyday events, they are characterized by several seemingly contradictory features: intrusive memories contain distinct sensory and emotional details of the traumatic event and can be triggered by various perceptually similar cues, but they are poorly integrated into conceptual memory. Here, we conduct exploratory whole-brain analyses to investigate the neural representations of trauma-analog experiences and how they are reactivated during memory intrusions. We show that trauma-analog movies induce excessive processing and generalized representations in sensory areas but decreased blood-oxygen-level-dependent (BOLD) responses and highly distinct representations in conceptual/semantic areas. Intrusive memories activate generalized representations in sensory areas and reactivate memory traces specific to trauma-analog events in the anterior cingulate cortex. These findings provide the first evidence of how traumatic events could distort memory representations in the human brain, which may form the basis for future confirmatory research on the neural representations of traumatic experiences.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

Cortical gradients during naturalistic processing are hierarchical and modality-specific

Understanding cortical topographic organization and how it supports complex perceptual and cognitive processes is a fundamental question in neuroscience. Previous work has characterized functional gradients that demonstrate large-scale principles of cortical organization. How these gradients are modulated by rich ecological stimuli remains unknown. Here, we utilize naturalistic stimuli via movie-fMRI to assess macroscale functional organization. We identify principal movie gradients that delineate separate hierarchies anchored in sensorimotor, visual, and auditory/language areas. At the opposite/heteromodal end of these perception-to-cognition axes, we find a more central role for the frontoparietal network along with the default network. Even across different movie stimuli, movie gradients demonstrated good reliability, suggesting that these hierarchies reflect a brain state common across different naturalistic conditions. The relative position of brain areas within movie gradients showed stronger and more numerous correlations with cognitive behavioral scores compared to resting state gradients. Together, these findings provide an ecologically valid representation of the principles underlying cortical organization while the brain is active and engaged in multimodal, dynamic perceptual and cognitive processing.