Replay in minds and machines

Awake replay: off the clock but on the job

Hippocampal replay is widely thought to support two key cognitive functions: online decision-making and offline memory consolidation. In this review, we take a closer look at the hypothesized link between awake replay and online decision-making in rodents, and find only marginal evidence in support of this role. By contrast, the consolidation view is bolstered by new computational ideas and recent data, suggesting that (i) replay performs offline fictive learning for later goal-oriented behavior; and (ii) replay tags memories prior to sleep, prioritizing them for consolidation. Based on these recent advances, we favor an updated and refined role for awake replay - that is, supporting prioritized offline learning and tagging outside the hippocampus - rather than a direct, online role in guiding behavior.

Topography of putative bi-directional interaction between hippocampal sharp-wave ripples and neocortical slow oscillations

Systems consolidation relies on coordination between hippocampal sharp-wave ripples (SWRs) and neocortical UP/DOWN states during sleep. However, whether this coupling exists across the neocortex and the mechanisms enabling it remains unknown. By combining electrophysiology in mouse hippocampus (HPC) and retrosplenial cortex (RSC) with wide-field imaging of the dorsal neocortex, we found spatially and temporally precise bi-directional hippocampo-neocortical interaction. HPC multi-unit activity and SWR probability were correlated with UP/DOWN states in the default mode network (DMN), with the highest modulation by the RSC in deep sleep. Further, some SWRs were preceded by the high rebound excitation accompanying DMN DOWN → UP transitions, whereas large-amplitude SWRs were often followed by DOWN states originating in the RSC. We explain these electrophysiological results with a model in which the HPC and RSC are weakly coupled excitable systems capable of bi-directional perturbation and suggest that the RSC may act as a gateway through which SWRs can perturb downstream cortical regions via cortico-cortical propagation of DOWN states.

The time course and organization of hippocampal replay

The mechanisms by which the brain replays neural activity sequences remain unknown. Recording from large ensembles of hippocampal place cells in freely behaving rats, we observed that replay content is strictly organized over multiple timescales and governed by self-avoidance. After movement cessation, replays avoided the animal's previous path for 3 seconds. Chains of replays avoided self-repetition over a shorter timescale. We used a continuous attractor model of neural activity to demonstrate that neuronal fatigue both generates replay sequences and produces self-avoidance over the observed timescales. In addition, replay of past experience became predominant later into the stopping period, in a manner requiring cortical input. These results indicate a mechanism for replay generation that unexpectedly constrains which sequences can be produced across time.

Investigating the intrinsic top-down dynamics of deep generative models

Hierarchical generative models can produce data samples based on the statistical structure of their training distribution. This capability can be linked to current theories in computational neuroscience, which propose that spontaneous brain activity at rest is the manifestation of top-down dynamics of generative models detached from action-perception cycles. A popular class of hierarchical generative models is that of Deep Belief Networks (DBNs), which are energy-based deep learning architectures that can learn multiple levels of representations in a completely unsupervised way exploiting Hebbian-like learning mechanisms. In this work, we study the generative dynamics of a recent extension of the DBN, the iterative DBN (iDBN), which more faithfully simulates neurocognitive development by jointly tuning the connection weights across all layers of the hierarchy. We characterize the number of states visited during top-down sampling and investigate whether the heterogeneity of visited attractors could be increased by initiating the generation process from biased hidden states. To this end, we train iDBN models on well-known datasets containing handwritten digits and pictures of human faces, and show that the ability to generate diverse data prototypes can be enhanced by initializing top-down sampling from "chimera states", which represent high-level features combining multiple abstract representations of the sensory data. Although the models are not always able to transition between all potential target states within a single-generation trajectory, the iDBN shows richer top-down dynamics in comparison to a shallow generative model (a single-layer Restricted Bolzamann Machine). We further show that the generated samples can be used to support continual learning through generative replay mechanisms. Our findings suggest that the top-down dynamics of hierarchical generative models is significantly influenced by the shape of the energy function, which depends both on the depth of the processing architecture and on the statistical structure of the sensory data.

Representational spaces in orbitofrontal and ventromedial prefrontal cortex: task states, values, and beyond

The orbitofrontal cortex (OFC) and ventromedial-prefrontal cortex (vmPFC) play a key role in decision-making and encode task states in addition to expected value. We review evidence suggesting a connection between value and state representations and argue that OFC / vmPFC integrate stimulus, context, and outcome information. Comparable encoding principles emerge in late layers of deep reinforcement learning (RL) models, where single nodes exhibit similar forms of mixed-selectivity, which enables flexible readout of relevant variables by downstream neurons. Based on these lines of evidence, we suggest that outcome-maximization leads to complex representational spaces that are insufficiently characterized by linear value signals that have been the focus of most prior research on the topic. Major outstanding questions concern the role of OFC/ vmPFC in learning across tasks, in encoding of task-irrelevant aspects, and the role of hippocampus-PFC interactions.

A hierarchical active inference model of spatial alternation tasks and the hippocampal-prefrontal circuit

Cognitive problem-solving benefits from cognitive maps aiding navigation and planning. Physical space navigation involves hippocampal (HC) allocentric codes, while abstract task space engages medial prefrontal cortex (mPFC) task-specific codes. Previous studies show that challenging tasks, like spatial alternation, require integrating these two types of maps. The disruption of the HC-mPFC circuit impairs performance. We propose a hierarchical active inference model clarifying how this circuit solves spatial interaction tasks by bridging physical and task-space maps. Simulations demonstrate that the model's dual layers develop effective cognitive maps for physical and task space. The model solves spatial alternation tasks through reciprocal interactions between the two layers. Disrupting its communication impairs decision-making, which is consistent with empirical evidence. Additionally, the model adapts to switching between multiple alternation rules, providing a mechanistic explanation of how the HC-mPFC circuit supports spatial alternation tasks and the effects of disruption.

An inductive bias for slowly changing features in human reinforcement learning

Identifying goal-relevant features in novel environments is a central challenge for efficient behaviour. We asked whether humans address this challenge by relying on prior knowledge about common properties of reward-predicting features. One such property is the rate of change of features, given that behaviourally relevant processes tend to change on a slower timescale than noise. Hence, we asked whether humans are biased to learn more when task-relevant features are slow rather than fast. To test this idea, 295 human participants were asked to learn the rewards of two-dimensional bandits when either a slowly or quickly changing feature of the bandit predicted reward. Across two experiments and one preregistered replication, participants accrued more reward when a bandit's relevant feature changed slowly, and its irrelevant feature quickly, as compared to the opposite. We did not find a difference in the ability to generalise to unseen feature values between conditions. Testing how feature speed could affect learning with a set of four function approximation Kalman filter models revealed that participants had a higher learning rate for the slow feature, and adjusted their learning to both the relevance and the speed of feature changes. The larger the improvement in participants' performance for slow compared to fast bandits, the more strongly they adjusted their learning rates. These results provide evidence that human reinforcement learning favours slower features, suggesting a bias in how humans approach reward learning.

Topography of putative bidirectional interaction between hippocampal sharp wave ripples and neocortical slow oscillations

Systems consolidation relies on coordination between hippocampal sharp-wave ripples (SWRs) and neocortical UP/DOWN states during sleep. However, whether this coupling exists across neocortex and the mechanisms enabling it remain unknown. By combining electrophysiology in mouse hippocampus (HPC) and retrosplenial cortex (RSC) with widefield imaging of dorsal neocortex, we found spatially and temporally precise bidirectional hippocampo-neocortical interaction. HPC multi-unit activity and SWR probability was correlated with UP/DOWN states in mouse default mode network, with highest modulation by RSC in deep sleep. Further, some SWRs were preceded by the high rebound excitation accompanying DMN DOWN→UP transitions, while large-amplitude SWRs were often followed by DOWN states originating in RSC. We explain these electrophysiological results with a model in which HPC and RSC are weakly coupled excitable systems capable of bi-directional perturbation and suggest RSC may act as a gateway through which SWRs can perturb downstream cortical regions via cortico-cortical propagation of DOWN states.

Learning, Fast and Slow: Single- and Many-Shot Learning in the Hippocampus

The hippocampus is critical for memory and spatial navigation. The ability to map novel environments, as well as more abstract conceptual relationships, is fundamental to the cognitive flexibility that humans and other animals require to survive in a dynamic world. In this review, we survey recent advances in our understanding of how this flexibility is implemented anatomically and functionally by hippocampal circuitry, during both active exploration (online) and rest (offline). We discuss the advantages and limitations of spike timing-dependent plasticity and the more recently discovered behavioral timescale synaptic plasticity in supporting distinct learning modes in the hippocampus. Finally, we suggest complementary roles for these plasticity types in explaining many-shot and single-shot learning in the hippocampus and discuss how these rules could work together to support the learning of cognitive maps.

Self-avoidance dominates the selection of hippocampal replay

Spontaneous neural activity sequences are generated by the brain in the absence of external input 1-12 , yet how they are produced remains unknown. During immobility, hippocampal replay sequences depict spatial paths related to the animal's past experience or predicted future 13 . By recording from large ensembles of hippocampal place cells 14 in combination with optogenetic manipulation of cortical input in freely behaving rats, we show here that the selection of hippocampal replay is governed by a novel self-avoidance principle. Following movement cessation, replay of the animal's past path is strongly avoided, while replay of the future path predominates. Moreover, when the past and future paths overlap, early replays avoid both and depict entirely different trajectories. Further, replays avoid self-repetition, on a shorter timescale compared to the avoidance of previous behavioral trajectories. Eventually, several seconds into the stopping period, replay of the past trajectory dominates. This temporal organization contrasts with established and recent predictions 9,10,15,16 but is well-recapitulated by a symmetry-breaking attractor model of sequence generation in which individual neurons adapt their firing rates over time 26-35 . However, while the model is sufficient to produce avoidance of recently traversed or reactivated paths, it requires an additional excitatory input into recently activated cells to produce the later window of past-dominance. We performed optogenetic perturbations to demonstrate that this input is provided by medial entorhinal cortex, revealing its role in maintaining a memory of past experience that biases hippocampal replay. Together, these data provide specific evidence for how hippocampal replays are generated.

Efficient, continual, and generalized learning in the brain - neural mechanism of Mental Schema 2.0

There has been tremendous progress in artificial neural networks (ANNs) over the past decade; however, the gap between ANNs and the biological brain as a learning device remains large. With the goal of closing this gap, this paper reviews learning mechanisms in the brain by focusing on three important issues in ANN research: efficiency, continuity, and generalization. We first discuss the method by which the brain utilizes a variety of self-organizing mechanisms to maximize learning efficiency, with a focus on the role of spontaneous activity of the brain in shaping synaptic connections to facilitate spatiotemporal learning and numerical processing. Then, we examined the neuronal mechanisms that enable lifelong continual learning, with a focus on memory replay during sleep and its implementation in brain-inspired ANNs. Finally, we explored the method by which the brain generalizes learned knowledge in new situations, particularly from the mathematical generalization perspective of topology. Besides a systematic comparison in learning mechanisms between the brain and ANNs, we propose "Mental Schema 2.0," a new computational property underlying the brain's unique learning ability that can be implemented in ANNs.

Psychological Momentum

Our mental experience is largely continuous on the scale of seconds and minutes. However, this continuity does not always arise from a volitional carrying forward of ideas. Instead, recent actions, thoughts, dispositions, and emotions can persist in mind, continually shaping our later experience. Aspects of this fundamental property of human cognition - psychological momentum - have been studied under the rubrics of memory, task set, mood, mind-wandering, and mindset. Reviewing these largely independent threads of research, we argue that psychological momentum is best understood from an integrated perspective, as an adaptation that helps us meet the current demands of our environment and to form lasting memories.

Hippocampal spatio-predictive cognitive maps adaptively guide reward generalization

The brain forms cognitive maps of relational knowledge-an organizing principle thought to underlie our ability to generalize and make inferences. However, how can a relevant map be selected in situations where a stimulus is embedded in multiple relational structures? Here, we find that both spatial and predictive cognitive maps influence generalization in a choice task, where spatial location determines reward magnitude. Mirroring behavior, the hippocampus not only builds a map of spatial relationships but also encodes the experienced transition structure. As the task progresses, participants' choices become more influenced by spatial relationships, reflected in a strengthening of the spatial map and a weakening of the predictive map. This change is driven by orbitofrontal cortex, which represents the degree to which an outcome is consistent with the spatial rather than the predictive map and updates hippocampal representations accordingly. Taken together, this demonstrates how hippocampal cognitive maps are used and updated flexibly for inference.

How our understanding of memory replay evolves

Memory reactivations and replay, widely reported in the hippocampus and cortex across species, have been implicated in memory consolidation, planning, and spatial and skill learning. Technological advances in electrophysiology, calcium imaging, and human neuroimaging techniques have enabled neuroscientists to measure large-scale neural activity with increasing spatiotemporal resolution and have provided opportunities for developing robust analytic methods to identify memory replay. In this article, we first review a large body of historically important and representative memory replay studies from the animal and human literature. We then discuss our current understanding of memory replay functions in learning, planning, and memory consolidation and further discuss the progress in computational modeling that has contributed to these improvements. Next, we review past and present analytic methods for replay analyses and discuss their limitations and challenges. Finally, looking ahead, we discuss some promising analytic methods for detecting nonstereotypical, behaviorally nondecodable structures from large-scale neural recordings. We argue that seamless integration of multisite recordings, real-time replay decoding, and closed-loop manipulation experiments will be essential for delineating the role of memory replay in a wide range of cognitive and motor functions.

Replay and compositional computation

Replay in the brain has been viewed as rehearsal or, more recently, as sampling from a transition model. Here, we propose a new hypothesis: that replay is able to implement a form of compositional computation where entities are assembled into relationally bound structures to derive qualitatively new knowledge. This idea builds on recent advances in neuroscience, which indicate that the hippocampus flexibly binds objects to generalizable roles and that replay strings these role-bound objects into compound statements. We suggest experiments to test our hypothesis, and we end by noting the implications for AI systems which lack the human ability to radically generalize past experience to solve new problems.

Mesoscopic description of hippocampal replay and metastability in spiking neural networks with short-term plasticity

Bottom-up models of functionally relevant patterns of neural activity provide an explicit link between neuronal dynamics and computation. A prime example of functional activity patterns are propagating bursts of place-cell activities called hippocampal replay, which is critical for memory consolidation. The sudden and repeated occurrences of these burst states during ongoing neural activity suggest metastable neural circuit dynamics. As metastability has been attributed to noise and/or slow fatigue mechanisms, we propose a concise mesoscopic model which accounts for both. Crucially, our model is bottom-up: it is analytically derived from the dynamics of finite-size networks of Linear-Nonlinear Poisson neurons with short-term synaptic depression. As such, noise is explicitly linked to stochastic spiking and network size, and fatigue is explicitly linked to synaptic dynamics. To derive the mesoscopic model, we first consider a homogeneous spiking neural network and follow the temporal coarse-graining approach of Gillespie to obtain a "chemical Langevin equation", which can be naturally interpreted as a stochastic neural mass model. The Langevin equation is computationally inexpensive to simulate and enables a thorough study of metastable dynamics in classical setups (population spikes and Up-Down-states dynamics) by means of phase-plane analysis. An extension of the Langevin equation for small network sizes is also presented. The stochastic neural mass model constitutes the basic component of our mesoscopic model for replay. We show that the mesoscopic model faithfully captures the statistical structure of individual replayed trajectories in microscopic simulations and in previously reported experimental data. Moreover, compared to the deterministic Romani-Tsodyks model of place-cell dynamics, it exhibits a higher level of variability regarding order, direction and timing of replayed trajectories, which seems biologically more plausible and could be functionally desirable. This variability is the product of a new dynamical regime where metastability emerges from a complex interplay between finite-size fluctuations and local fatigue.

The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning

We advance a novel computational theory of the hippocampal formation as a hierarchical generative model that organizes sequential experiences, such as rodent trajectories during spatial navigation, into coherent spatiotemporal contexts. We propose that the hippocampal generative model is endowed with inductive biases to identify individual items of experience (first hierarchical layer), organize them into sequences (second layer) and cluster them into maps (third layer). This theory entails a novel characterization of hippocampal reactivations as generative replay: the offline resampling of fictive sequences from the generative model, which supports the continual learning of multiple sequential experiences. We show that the model learns and efficiently retains multiple spatial navigation trajectories, by organizing them into spatial maps. Furthermore, the model reproduces flexible and prospective aspects of hippocampal dynamics that are challenging to explain within existing frameworks. This theory reconciles multiple roles of the hippocampal formation in map-based navigation, episodic memory and imagination.

Decoding cognition from spontaneous neural activity

In human neuroscience, studies of cognition are rarely grounded in non-task-evoked, 'spontaneous' neural activity. Indeed, studies of spontaneous activity tend to focus predominantly on intrinsic neural patterns (for example, resting-state networks). Taking a 'representation-rich' approach bridges the gap between cognition and resting-state communities: this approach relies on decoding task-related representations from spontaneous neural activity, allowing quantification of the representational content and rich dynamics of such activity. For example, if we know the neural representation of an episodic memory, we can decode its subsequent replay during rest. We argue that such an approach advances cognitive research beyond a focus on immediate task demand and provides insight into the functional relevance of the intrinsic neural pattern (for example, the default mode network). This in turn enables a greater integration between human and animal neuroscience, facilitating experimental testing of theoretical accounts of intrinsic activity, and opening new avenues of research in psychiatry.

The Neural Representation of Events Is Dominated by Elements that Are Most Reliably Present

An episodic memory is specific to an event that occurred at a particular time and place. However, the elements that comprise the event-the location, the people present, and their actions and goals-might be shared with numerous other similar events. Does the brain preferentially represent certain elements of a remembered event? If so, which elements dominate its neural representation: those that are shared across similar events, or the novel elements that define a specific event? We addressed these questions by using a novel experimental paradigm combined with fMRI. Multiple events were created involving conversations between two individuals using the format of a television chat show. Chat show "hosts" occurred repeatedly across multiple events, whereas the "guests" were unique to only one event. Before learning the conversations, participants were scanned while viewing images or names of the (famous) individuals to be used in the study to obtain person-specific activity patterns. After learning all the conversations over a week, participants were scanned for a second time while they recalled each event multiple times. We found that during recall, person-specific activity patterns within the posterior midline network were reinstated for the hosts of the shows but not the guests, and that reinstatement of the hosts was significantly stronger than the reinstatement of the guests. These findings demonstrate that it is the more generic, familiar, and predictable elements of an event that dominate its neural representation compared with the more idiosyncratic, event-defining, elements.