Prioritized experience replays on a hippocampal predictive map for learning

Non-feature-specific elevated responses and feature-specific backward replay in human brain induced by visual sequence exposure

The ability of cortical circuits to adapt in response to experience is a fundamental property of the brain. After exposure to a moving dot sequence, flashing a dot as a cue at the starting point of the sequence can elicit successive elevated responses even in the absence of the sequence. These cue-triggered elevated responses have been shown to play a crucial role in predicting future events in dynamic environments. However, temporal sequences we are exposed to typically contain rich feature information. It remains unknown whether the elevated responses are feature-specific and, more crucially, how the brain organizes sequence information after exposure. To address these questions, participants were exposed to a predefined sequence of four motion directions for about 30 min, followed by the presentation of the start or end motion direction of the sequence as a cue. Surprisingly, we found that cue-triggered elevated responses were not specific to any motion direction. Interestingly, motion direction information was spontaneously reactivated, and the motion sequence was backward replayed in a time-compressed manner. These effects were observed even after brief exposure. Notably, no replay events were observed when the second or third motion direction of the sequence served as a cue. Further analyses revealed that activity in the medial temporal lobe (MTL) preceded the ripple power increase in visual cortex at the onset of replay, implying a coordinated relationship between the activities in the MTL and visual cortex. Together, these findings demonstrate that visual sequence exposure induces twofold brain plasticity that may simultaneously serve for different functional purposes. The non-feature-specific elevated responses may facilitate general processing of upcoming stimuli, whereas the feature-specific backward replay may underpin passive learning of visual sequences.

Hippocampal Place Cell Sequences Are Impaired in a Rat Model of Fragile X Syndrome

Fragile X syndrome (FXS) is a neurodevelopmental disorder that can cause impairments in spatial cognition and memory. The hippocampus is thought to support spatial cognition through the activity of place cells, neurons with spatial receptive fields. Coordinated firing of place cell populations is organized by different oscillatory patterns in the hippocampus during specific behavioral states. Theta rhythms organize place cell populations during awake exploration. Sharp wave-ripples organize place cell population reactivation during waking rest. Here, we examined the coordination of CA1 place cell populations during active behavior and subsequent rest in a rat model of FXS (Fmr1 knock-out rats). While the organization of individual place cells by the theta rhythm was normal, the coordinated activation of sequences of place cells during individual theta cycles was impaired in Fmr1 knock-out rats. Furthermore, the subsequent replay of place cell sequences was impaired during waking rest following active exploration. Together, these results expand our understanding of how genetic modifications that model those observed in FXS affect hippocampal physiology and suggest a potential mechanism underlying impaired spatial cognition in FXS.

Awake replay: off the clock but on the job

Hippocampal replay is widely thought to support two key cognitive functions: online decision-making and offline memory consolidation. In this review, we take a closer look at the hypothesized link between awake replay and online decision-making in rodents, and find only marginal evidence in support of this role. By contrast, the consolidation view is bolstered by new computational ideas and recent data, suggesting that (i) replay performs offline fictive learning for later goal-oriented behavior; and (ii) replay tags memories prior to sleep, prioritizing them for consolidation. Based on these recent advances, we favor an updated and refined role for awake replay - that is, supporting prioritized offline learning and tagging outside the hippocampus - rather than a direct, online role in guiding behavior.

The time course and organization of hippocampal replay

The mechanisms by which the brain replays neural activity sequences remain unknown. Recording from large ensembles of hippocampal place cells in freely behaving rats, we observed that replay content is strictly organized over multiple timescales and governed by self-avoidance. After movement cessation, replays avoided the animal's previous path for 3 seconds. Chains of replays avoided self-repetition over a shorter timescale. We used a continuous attractor model of neural activity to demonstrate that neuronal fatigue both generates replay sequences and produces self-avoidance over the observed timescales. In addition, replay of past experience became predominant later into the stopping period, in a manner requiring cortical input. These results indicate a mechanism for replay generation that unexpectedly constrains which sequences can be produced across time.

Interictal spikes during spatial working memory carry helpful or distracting representations of space and have opposing impacts on performance

In temporal lobe epilepsy, interictal spikes (IS) - hypersynchronous bursts of network activity - occur at high rates in between seizures. We sought to understand the influence of IS on working memory by recording hippocampal local field potentials from epileptic mice while they performed a delayed alternation task. We found that IS disrupted performance when they were spatially non-restricted and occurred during running. In contrast, when IS were clustered at reward locations, animals performed well. A machine learning decoding approach revealed that IS at reward sites were larger than IS elsewhere on the maze, and could be classified as occurring at specific reward locations - suggesting they carry informative content for the memory task. Finally, a spiking model revealed that spatially clustered IS preserved hippocampal replay, while spatially dispersed IS disrupted replay by causing over-generalization. Together, these results show that IS can have opposing outcomes on memory.

Replay shapes abstract cognitive maps for efficient social navigation

To make adaptive social decisions, people must anticipate how information flows through their social network. While this requires knowledge of how people are connected, networks are too large to have first-hand experience with every possible route between individuals. How, then, are people able to accurately track information flow through social networks? Here we find that people immediately cache abstract knowledge about social network structure as they learn who is friends with whom, which enables the identification of efficient routes between remotely connected individuals. These cognitive maps of social networks, which are built while learning, are then reshaped through overnight rest. During these extended periods of rest, a replay-like mechanism helps to make these maps increasingly abstract, which privileges improvements in social navigation accuracy for the longest communication paths that span distinct communities within the network. Together, these findings provide mechanistic insight into the sophisticated mental representations humans use for social navigation.

Selective reactivation of value- and place-dependent information during sharp-wave ripples in the intermediate and dorsal hippocampus

Reactivating place cells during sharp-wave ripples in the hippocampus is important for memory consolidation. However, whether hippocampal reactivation is affected by the values of events experienced by the animal is largely unknown. Here, we investigated whether place cells in the dorsal (dHP) and intermediate hippocampus (iHP) of rats are differentially reactivated depending on the value associated with a place during the learning of places associated with higher-value rewards in a T-maze. Place cells in the iHP representing the high-value location were reactivated significantly more frequently than those representing the low-value location, characteristics not observed in the dHP. In contrast, the activities of place cells in the dHP coding the routes leading to high-value locations were replayed more than those in the iHP. Our findings suggest that value-based differential reactivation patterns along the septotemporal axis of the hippocampus may play essential roles in optimizing goal-directed spatial learning for maximal reward.

Selection of experience for memory by hippocampal sharp wave ripples

Experiences need to be tagged during learning for further consolidation. However, neurophysiological mechanisms that select experiences for lasting memory are not known. By combining large-scale neural recordings in mice with dimensionality reduction techniques, we observed that successive maze traversals were tracked by continuously drifting populations of neurons, providing neuronal signatures of both places visited and events encountered. When the brain state changed during reward consumption, sharp wave ripples (SPW-Rs) occurred on some trials, and their specific spike content decoded the trial blocks that surrounded them. During postexperience sleep, SPW-Rs continued to replay those trial blocks that were reactivated most frequently during waking SPW-Rs. Replay content of awake SPW-Rs may thus provide a neurophysiological tagging mechanism to select aspects of experience that are preserved and consolidated for future use.

Disruption of awake sharp-wave ripples does not affect memorization of locations in repeated-acquisition spatial memory tasks

Storing and accessing memories is required to successfully perform day-to-day tasks, for example for engaging in a meaningful conversation. Previous studies in both rodents and primates have correlated hippocampal cellular activity with behavioral expression of memory. A key role has been attributed to awake hippocampal replay - a sequential reactivation of neurons representing a trajectory through space. However, it is unclear if awake replay impacts immediate future behavior, gradually creates and stabilizes long-term memories over a long period of time (hours and longer), or enables the temporary memorization of relevant events at an intermediate time scale (seconds to minutes). In this study, we aimed to address the uncertainty around the timeframe of impact of awake replay by collecting causal evidence from behaving rats. We detected and disrupted sharp wave ripples (SWRs) - signatures of putative replay events - using electrical stimulation of the ventral hippocampal commissure in rats that were trained on three different spatial memory tasks. In each task, rats were required to memorize a new set of locations in each trial or each daily session. Interestingly, the rats performed equally well with or without SWR disruptions. These data suggest that awake SWRs - and potentially replay - does not affect the immediate behavior nor the temporary memorization of relevant events at a short timescale that are required to successfully perform the spatial tasks. Based on these results, we hypothesize that the impact of awake replay on memory and behavior is long-term and cumulative over time.

A generative model of memory construction and consolidation

Episodic memories are (re)constructed, share neural substrates with imagination, combine unique features with schema-based predictions and show schema-based distortions that increase with consolidation. Here we present a computational model in which hippocampal replay (from an autoassociative network) trains generative models (variational autoencoders) to (re)create sensory experiences from latent variable representations in entorhinal, medial prefrontal and anterolateral temporal cortices via the hippocampal formation. Simulations show effects of memory age and hippocampal lesions in agreement with previous models, but also provide mechanisms for semantic memory, imagination, episodic future thinking, relational inference and schema-based distortions including boundary extension. The model explains how unique sensory and predictable conceptual elements of memories are stored and reconstructed by efficiently combining both hippocampal and neocortical systems, optimizing the use of limited hippocampal storage for new and unusual information. Overall, we believe hippocampal replay training generative models provides a comprehensive account of memory construction, imagination and consolidation.

Selection of experience for memory by hippocampal sharp wave ripples

A general wisdom is that experiences need to be tagged during learning for further consolidation. However, brain mechanisms that select experiences for lasting memory are not known. Combining large-scale neural recordings with a novel application of dimensionality reduction techniques, we observed that successive traversals in the maze were tracked by continuously drifting populations of neurons, providing neuronal signatures of both places visited and events encountered (trial number). When the brain state changed during reward consumption, sharp wave ripples (SPW-Rs) occurred on some trials and their unique spike content most often decoded the trial in which they occurred. In turn, during post-experience sleep, SPW-Rs continued to replay those trials that were reactivated most frequently during awake SPW-Rs. These findings suggest that replay content of awake SPW-Rs provides a tagging mechanism to select aspects of experience that are preserved and consolidated for future use.

Associative and predictive hippocampal codes support memory-guided behaviors

Episodic memory involves learning and recalling associations between items and their spatiotemporal context. Those memories can be further used to generate internal models of the world that enable predictions to be made. The mechanisms that support these associative and predictive aspects of memory are not yet understood. In this study, we used an optogenetic manipulation to perturb the sequential structure, but not global network dynamics, of place cells as rats traversed specific spatial trajectories. This perturbation abolished replay of those trajectories and the development of predictive representations, leading to impaired learning of new optimal trajectories during memory-guided navigation. However, place cell assembly reactivation and reward-context associative learning were unaffected. Our results show a mechanistic dissociation between two complementary hippocampal codes: an associative code (through coactivity) and a predictive code (through sequences).

Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.

Awake hippocampal synchronous events are incorporated into offline neuronal reactivation

Learning novel experiences reorganizes hippocampal neuronal circuits, represented as coordinated reactivation patterns in post-experience offline states for memory consolidation. This study examines how awake synchronous events during a novel run are related to post-run reactivation patterns. The disruption of awake sharp-wave ripples inhibited experience-induced increases in the contributions of neurons to post-experience synchronous events. Hippocampal place cells that participate more in awake synchronous events are more strongly reactivated during post-experience synchronous events. Awake synchronous neuronal patterns, in cooperation with place-selective firing patterns, determine cell ensembles that undergo pronounced increases and decreases in their correlated spikes. Taken together, awake synchronous events are fundamental for identifying hippocampal neuronal ensembles to be incorporated into synchronous reactivation during subsequent offline states, thereby facilitating memory consolidation.

Offline memory replay in recurrent neuronal networks emerges from constraints on online dynamics

During spatial exploration, neural circuits in the hippocampus store memories of sequences of sensory events encountered in the environment. When sensory information is absent during 'offline' resting periods, brief neuronal population bursts can 'replay' sequences of activity that resemble bouts of sensory experience. These sequences can occur in either forward or reverse order, and can even include spatial trajectories that have not been experienced, but are consistent with the topology of the environment. The neural circuit mechanisms underlying this variable and flexible sequence generation are unknown. Here we demonstrate in a recurrent spiking network model of hippocampal area CA3 that experimental constraints on network dynamics such as population sparsity, stimulus selectivity, rhythmicity and spike rate adaptation, as well as associative synaptic connectivity, enable additional emergent properties, including variable offline memory replay. In an online stimulus-driven state, we observed the emergence of neuronal sequences that swept from representations of past to future stimuli on the timescale of the theta rhythm. In an offline state driven only by noise, the network generated both forward and reverse neuronal sequences, and recapitulated the experimental observation that offline memory replay events tend to include salient locations like the site of a reward. These results demonstrate that biological constraints on the dynamics of recurrent neural circuits are sufficient to enable memories of sensory events stored in the strengths of synaptic connections to be flexibly read out during rest and sleep, which is thought to be important for memory consolidation and planning of future behaviour. KEY POINTS: A recurrent spiking network model of hippocampal area CA3 was optimized to recapitulate experimentally observed network dynamics during simulated spatial exploration. During simulated offline rest, the network exhibited the emergent property of generating flexible forward, reverse and mixed direction memory replay events. Network perturbations and analysis of model diversity and degeneracy identified associative synaptic connectivity and key features of network dynamics as important for offline sequence generation. Network simulations demonstrate that population over-representation of salient positions like the site of reward results in biased memory replay.

Effects of theta phase precessing optogenetic intervention on hippocampal neuronal reactivation and spatial maps

As animals explore environments, hippocampal place cells sequentially fire at progressively earlier phases of theta oscillations in hippocampal local field potentials. In this study, we evaluated the network-level significance of theta phase-entrained neuronal activity in organizing place cell spike patterns. A closed-loop system was developed in which optogenetic stimulation with a temporal pattern replicating theta phase precession is delivered to hippocampal CA1 neurons when rats traversed a particular region on a linear track. Place cells that had place fields during phase precessing stimulation, but not random phase stimulation, showed stronger reactivation during hippocampal sharp-wave ripples in a subsequent rest period. After the rest period, place cells with place fields that emerged during phase precessing stimulation showed more stable place fields. These results imply that neuronal reactivation and stability of spatial maps are mediated by theta phase precession in the hippocampus.

Learning to predict future locations with internally generated theta sequences

Representing past, present and future locations is key for spatial navigation. Indeed, within each cycle of the theta oscillation, the population of hippocampal place cells appears to represent trajectories starting behind the current position of the animal and sweeping ahead of it. In particular, we reported recently that the position represented by CA1 place cells at a given theta phase corresponds to the location where animals were or will be located at a fixed time interval into the past or future assuming the animal ran at its typical, not the current, speed through that part of the environment. This coding scheme leads to longer theta trajectories, larger place fields and shallower phase precession in areas where animals typically run faster. Here we present a mechanistic computational model that accounts for these experimental observations. The model consists of a continuous attractor network with short-term synaptic facilitation and depression that internally generates theta sequences that advance at a fixed pace. Spatial locations are then mapped onto the active units via modified Hebbian plasticity. As a result, neighboring units become associated with spatial locations further apart where animals run faster, reproducing our earlier experimental results. The model also accounts for the higher density of place fields generally observed where animals slow down, such as around rewards. Furthermore, our modeling results reveal that an artifact of the decoding analysis might be partly responsible for the observation that theta trajectories start behind the animal's current position. Overall, our results shed light on how the hippocampal code might arise from the interplay between behavior, sensory input and predefined network dynamics.

Hippocampal sharp wave ripples underlie stress susceptibility in male mice

The ventral hippocampus (vHC) is a core brain region for emotional memory. Here, we examined how the vHC regulates stress susceptibility from the level of gene expression to neuronal population dynamics in male mice. Transcriptome analysis of samples from stress-naïve mice revealed that intrinsic calbindin (Calb1) expression in the vHC is associated with susceptibility to social defeat stress. Mice with Calb1 gene knockdown in the vHC exhibited increased stress resilience and failed to show the increase in the poststress ventral hippocampal sharp wave ripple (SWR) rate. Poststress vHC SWRs triggered synchronous reactivation of stress memory-encoding neuronal ensembles and facilitated information transfer to the amygdala. Suppression of poststress vHC SWRs by real-time feedback stimulation or walking prevented social behavior deficits. Taken together, our results demonstrate that internal reactivation of memories of negative stressful episodes supported by ventral hippocampal SWRs serves as a crucial neurophysiological substrate for determining stress susceptibility.

Differential replay of reward and punishment paths predicts approach and avoidance

Neural replay is implicated in planning, where states relevant to a task goal are rapidly reactivated in sequence. It remains unclear whether, during planning, replay relates to an actual prospective choice. Here, using magnetoencephalography (MEG), we studied replay in human participants while they planned to either approach or avoid an uncertain environment containing paths leading to reward or punishment. We find evidence for forward sequential replay during planning, with rapid state-to-state transitions from 20 to 90 ms. Replay of rewarding paths was boosted, relative to aversive paths, before a decision to avoid and attenuated before a decision to approach. A trial-by-trial bias toward replaying prospective punishing paths predicted irrational decisions to approach riskier environments, an effect more pronounced in participants with higher trait anxiety. The findings indicate a coupling of replay with planned behavior, where replay prioritizes an online representation of a worst-case scenario for approaching or avoiding.

Learning predictive cognitive maps with spiking neurons during behavior and replays

The hippocampus has been proposed to encode environments using a representation that contains predictive information about likely future states, called the successor representation. However, it is not clear how such a representation could be learned in the hippocampal circuit. Here, we propose a plasticity rule that can learn this predictive map of the environment using a spiking neural network. We connect this biologically plausible plasticity rule to reinforcement learning, mathematically and numerically showing that it implements the TD-lambda algorithm. By spanning these different levels, we show how our framework naturally encompasses behavioral activity and replays, smoothly moving from rate to temporal coding, and allows learning over behavioral timescales with a plasticity rule acting on a timescale of milliseconds. We discuss how biological parameters such as dwelling times at states, neuronal firing rates and neuromodulation relate to the delay discounting parameter of the TD algorithm, and how they influence the learned representation. We also find that, in agreement with psychological studies and contrary to reinforcement learning theory, the discount factor decreases hyperbolically with time. Finally, our framework suggests a role for replays, in both aiding learning in novel environments and finding shortcut trajectories that were not experienced during behavior, in agreement with experimental data.

How our understanding of memory replay evolves

Memory reactivations and replay, widely reported in the hippocampus and cortex across species, have been implicated in memory consolidation, planning, and spatial and skill learning. Technological advances in electrophysiology, calcium imaging, and human neuroimaging techniques have enabled neuroscientists to measure large-scale neural activity with increasing spatiotemporal resolution and have provided opportunities for developing robust analytic methods to identify memory replay. In this article, we first review a large body of historically important and representative memory replay studies from the animal and human literature. We then discuss our current understanding of memory replay functions in learning, planning, and memory consolidation and further discuss the progress in computational modeling that has contributed to these improvements. Next, we review past and present analytic methods for replay analyses and discuss their limitations and challenges. Finally, looking ahead, we discuss some promising analytic methods for detecting nonstereotypical, behaviorally nondecodable structures from large-scale neural recordings. We argue that seamless integration of multisite recordings, real-time replay decoding, and closed-loop manipulation experiments will be essential for delineating the role of memory replay in a wide range of cognitive and motor functions.

Imagination as a fundamental function of the hippocampus

Imagination is a biological function that is vital to human experience and advanced cognition. Despite this importance, it remains unknown how imagination is realized in the brain. Substantial research focusing on the hippocampus, a brain structure traditionally linked to memory, indicates that firing patterns in spatially tuned neurons can represent previous and upcoming paths in space. This work has generally been interpreted under standard views that the hippocampus implements cognitive abilities primarily related to actual experience, whether in the past (e.g. recollection, consolidation), present (e.g. spatial mapping) or future (e.g. planning). However, relatively recent findings in rodents identify robust patterns of hippocampal firing corresponding to a variety of alternatives to actual experience, in many cases without overt reference to the past, present or future. Given these findings, and others on hippocampal contributions to human imagination, we suggest that a fundamental function of the hippocampus is to generate a wealth of hypothetical experiences and thoughts. Under this view, traditional accounts of hippocampal function in episodic memory and spatial navigation can be understood as particular applications of a more general system for imagination. This view also suggests that the hippocampus contributes to a wider range of cognitive abilities than previously thought. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

Disrupting ripples: Methods, results, and caveats in closed-loop approaches in rodents

Hippocampal ripple oscillations have been associated with memory reactivations during wake and sleep. These reactivations should contribute to working memory and memory consolidation respectively. In the past decade studies have moved from being observational to actively disrupting ripple-related activity in closed-loop approaches to enable causal investigations into their function. All together these studies have been able to provide evidence that wake, task-related ripple activity is important for working memory and planning but less important for stabilisation of spatial representations. Rest and sleep-related ripple activity, in contrast, is important for long-term memory performance and thus memory consolidation. In this review, we summarise results from different closed-loop approaches in rodents. Further, we highlight differences in detection and stimulation methods as well as controls and discuss how these differences could influence outcomes.

Postsynaptic burst reactivation of hippocampal neurons enables associative plasticity of temporally discontiguous inputs

A fundamental unresolved problem in neuroscience is how the brain associates in memory events that are separated in time. Here, we propose that reactivation-induced synaptic plasticity can solve this problem. Previously, we reported that the reinforcement signal dopamine converts hippocampal spike timing-dependent depression into potentiation during continued synaptic activity (Brzosko et al., 2015). Here, we report that postsynaptic bursts in the presence of dopamine produce input-specific LTP in mouse hippocampal synapses 10 min after they were primed with coincident pre- and post-synaptic activity (post-before-pre pairing; Δt = -20 ms). This priming activity induces synaptic depression and sets an NMDA receptor-dependent silent eligibility trace which, through the cAMP-PKA cascade, is rapidly converted into protein synthesis-dependent synaptic potentiation, mediated by a signaling pathway distinct from that of conventional LTP. This synaptic learning rule was incorporated into a computational model, and we found that it adds specificity to reinforcement learning by controlling memory allocation and enabling both 'instructive' and 'supervised' reinforcement learning. We predicted that this mechanism would make reactivated neurons activate more strongly and carry more spatial information than non-reactivated cells, which was confirmed in freely moving mice performing a reward-based navigation task.

An open-source, ready-to-use and validated ripple detector plugin for the Open Ephys GUI

OBJECTIVE

Sharp wave-ripples (SWRs, 100-250 Hz) are oscillatory events extracellularly recorded in the CA1 subfield of the hippocampus during sleep and quiet wakefulness. Many studies employed closed-loop strategies to either detect and abolish SWRs within the hippocampus or manipulate other relevant areas upon ripple detection. However, the code and schematics necessary to replicate the detection system are not always available, which hinders the reproducibility of experiments among different research groups. Furthermore, information about performance is not usually reported. Here, we sought to provide an open-source, validated ripple detector for the scientific community.

APPROACH

We developed and validated a ripple detection plugin integrated into the Open Ephys GUI. It contains a built-in movement detector based on accelerometer or electromyogram data that prevents false ripple events (due to chewing, grooming, or moving, for instance) from triggering the stimulation/manipulation device.

MAIN RESULTS

To determine the accuracy of the detection algorithm, we first carried out simulations in Matlab with real ripple recordings. Using a specific combination of detection parameters (amplitude threshold of 5 standard deviations above the mean, time threshold of 10 ms, and RMS block size of 7 samples), we obtained a 97% true positive rate and 2.48 false positives per minute. Next, an Open Ephys plugin based on the same detection algorithm was developed, and a closed-loop system was set up to evaluate the round trip (ripple onset-to-stimulation) latency over synthetic data. The lowest latency obtained was 34.5 ± 0.5 ms. The embedded movement monitoring was effective in reducing false positives and the plugin's flexibility to detect pathological events was also verified.

SIGNIFICANCE

Besides contributing to increased reproducibility, we anticipate that the developed ripple detector plugin will be helpful for many closed-loop applications in the field of systems neuroscience.

The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning

We advance a novel computational theory of the hippocampal formation as a hierarchical generative model that organizes sequential experiences, such as rodent trajectories during spatial navigation, into coherent spatiotemporal contexts. We propose that the hippocampal generative model is endowed with inductive biases to identify individual items of experience (first hierarchical layer), organize them into sequences (second layer) and cluster them into maps (third layer). This theory entails a novel characterization of hippocampal reactivations as generative replay: the offline resampling of fictive sequences from the generative model, which supports the continual learning of multiple sequential experiences. We show that the model learns and efficiently retains multiple spatial navigation trajectories, by organizing them into spatial maps. Furthermore, the model reproduces flexible and prospective aspects of hippocampal dynamics that are challenging to explain within existing frameworks. This theory reconciles multiple roles of the hippocampal formation in map-based navigation, episodic memory and imagination.

Multiple-Timescale Representations of Space: Linking Memory to Navigation

When navigating through space, we must maintain a representation of our position in real time; when recalling a past episode, a memory can come back in a flash. Interestingly, the brain's spatial representation system, including the hippocampus, supports these two distinct timescale functions. How are neural representations of space used in the service of both real-world navigation and internal mnemonic processes? Recent progress has identified sequences of hippocampal place cells, evolving at multiple timescales in accordance with either navigational behaviors or internal oscillations, that underlie these functions. We review experimental findings on experience-dependent modulation of these sequential representations and consider how they link real-world navigation to time-compressed memories. We further discuss recent work suggesting the prevalence of these sequences beyond hippocampus and propose that these multiple-timescale mechanisms may represent a general algorithm for organizing cell assemblies, potentially unifying the dual roles of the spatial representation system in memory and navigation.

Impact of optogenetic pulse design on CA3 learning and replay: A neural model

Optogenetic manipulation of hippocampal circuitry is an important tool for investigating learning in vivo. Numerous approaches to pulse design have been employed to elicit desirable circuit and behavioral outcomes. Here, we systematically test the outcome of different single-pulse waveforms in a rate-based model of hippocampal memory function at the level of mnemonic replay extension and de novo synaptic weight formation in CA3 and CA1. Lower-power waveforms with long forward or forward and backward ramps yield more natural sequence replay dynamics and induce synaptic plasticity that allows for more natural memory replay timing, in contrast to square or backward ramps. These differences between waveform shape and amplitude are preserved with the addition of noise in membrane potential, light scattering, and protein expression, improving the potential validity of predictions for in vivo work. These results inform future optogenetic experimental design choices in the field of learning and memory.

Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning

Recent models of spiking neuronal networks have been trained to perform behaviors in static environments using a variety of learning rules, with varying degrees of biological realism. Most of these models have not been tested in dynamic visual environments where models must make predictions on future states and adjust their behavior accordingly. The models using these learning rules are often treated as black boxes, with little analysis on circuit architectures and learning mechanisms supporting optimal performance. Here we developed visual/motor spiking neuronal network models and trained them to play a virtual racket-ball game using several reinforcement learning algorithms inspired by the dopaminergic reward system. We systematically investigated how different architectures and circuit-motifs (feed-forward, recurrent, feedback) contributed to learning and performance. We also developed a new biologically-inspired learning rule that significantly enhanced performance, while reducing training time. Our models included visual areas encoding game inputs and relaying the information to motor areas, which used this information to learn to move the racket to hit the ball. Neurons in the early visual area relayed information encoding object location and motion direction across the network. Neuronal association areas encoded spatial relationships between objects in the visual scene. Motor populations received inputs from visual and association areas representing the dorsal pathway. Two populations of motor neurons generated commands to move the racket up or down. Model-generated actions updated the environment and triggered reward or punishment signals that adjusted synaptic weights so that the models could learn which actions led to reward. Here we demonstrate that our biologically-plausible learning rules were effective in training spiking neuronal network models to solve problems in dynamic environments. We used our models to dissect the circuit architectures and learning rules most effective for learning. Our model shows that learning mechanisms involving different neural circuits produce similar performance in sensory-motor tasks. In biological networks, all learning mechanisms may complement one another, accelerating the learning capabilities of animals. Furthermore, this also highlights the resilience and redundancy in biological systems.

Spatial Learning Drives Rapid Goal Representation in Hippocampal Ripples without Place Field Accumulation or Goal-Oriented Theta Sequences

The hippocampus is critical for rapid acquisition of many forms of memory, although the circuit-level mechanisms through which the hippocampus rapidly consolidates novel information are unknown. Here, the activity of large ensembles of hippocampal neurons in adult male Long-Evans rats was monitored across a period of rapid spatial learning to assess how the network changes during the initial phases of memory formation and retrieval. In contrast to several reports, the hippocampal network did not display enhanced representation of the goal location via accumulation of place fields or elevated firing rates at the goal. Rather, population activity rates increased globally as a function of experience. These alterations in activity were mirrored in the power of the theta oscillation and in the quality of theta sequences, without preferential encoding of paths to the learned goal location. In contrast, during brief "offline" pauses in movement, representation of a novel goal location emerged rapidly in ripples, preceding other changes in network activity. These data demonstrate that the hippocampal network can facilitate active navigation without enhanced goal representation during periods of active movement, and further indicate that goal representation in hippocampal ripples before movement onset supports subsequent navigation, possibly through activation of downstream cortical networks.SIGNIFICANCE STATEMENT Understanding the mechanisms through which the networks of the brain rapidly assimilate information and use previously learned knowledge are fundamental areas of focus in neuroscience. In particular, the hippocampal circuit is a critical region for rapid formation and use of spatial memory. In this study, several circuit-level features of hippocampal function were quantified while rats performed a spatial navigation task requiring rapid memory formation and use. During periods of active navigation, a general increase in overall network activity is observed during memory acquisition, which plateaus during memory retrieval periods, without specific enhanced representation of the goal location. During pauses in navigation, rapid representation of the distant goal well emerges before either behavioral improvement or changes in online activity.

Planning in the brain

Recent breakthroughs in artificial intelligence (AI) have enabled machines to plan in tasks previously thought to be uniquely human. Meanwhile, the planning algorithms implemented by the brain itself remain largely unknown. Here, we review neural and behavioral data in sequential decision-making tasks that elucidate the ways in which the brain does-and does not-plan. To systematically review available biological data, we create a taxonomy of planning algorithms by summarizing the relevant design choices for such algorithms in AI. Across species, recording techniques, and task paradigms, we find converging evidence that the brain represents future states consistent with a class of planning algorithms within our taxonomy-focused, depth-limited, and serial. However, we argue that current data are insufficient for addressing more detailed algorithmic questions. We propose a new approach leveraging AI advances to drive experiments that can adjudicate between competing candidate algorithms.

Spatial goal coding in the hippocampal formation

The mammalian hippocampal formation contains several distinct populations of neurons involved in representing self-position and orientation. These neurons, which include place, grid, head direction, and boundary-vector cells, are thought to collectively instantiate cognitive maps supporting flexible navigation. However, to flexibly navigate, it is necessary to also maintain internal representations of goal locations, such that goal-directed routes can be planned and executed. Although it has remained unclear how the mammalian brain represents goal locations, multiple neural candidates have recently been uncovered during different phases of navigation. For example, during planning, sequential activation of spatial cells may enable simulation of future routes toward the goal. During travel, modulation of spatial cells by the prospective route, or by distance and direction to the goal, may allow maintenance of route and goal-location information, supporting navigation on an ongoing basis. As the goal is approached, an increased activation of spatial cells may enable the goal location to become distinctly represented within cognitive maps, aiding goal localization. Lastly, after arrival at the goal, sequential activation of spatial cells may represent the just-taken route, enabling route learning and evaluation. Here, we review and synthesize these and other evidence for goal coding in mammalian brains, relate the experimental findings to predictions from computational models, and discuss outstanding questions and future challenges.

Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice

Executing memory-guided behavior requires storage of information about experience and later recall of that information to inform choices. Awake hippocampal replay, when hippocampal neural ensembles briefly reactivate a representation related to prior experience, has been proposed to critically contribute to these memory-related processes. However, it remains unclear whether awake replay contributes to memory function by promoting the storage of past experiences, facilitating planning based on evaluation of those experiences, or both. We designed a dynamic spatial task that promotes replay before a memory-based choice and assessed how the content of replay related to past and future behavior. We found that replay content was decoupled from subsequent choice and instead was enriched for representations of previously rewarded locations and places that had not been visited recently, indicating a role in memory storage rather than in directly guiding subsequent behavior.

Hippocampal sharp-wave ripples in cognitive map maintenance versus episodic simulation

Hippocampal sharp-wave ripples (SWRs) have been proposed to support memory-based decision-making. A new study by Gillespie et al. (in this issue of Neuron) provides important new insights on how past experiences and future choices are reflected in neuronal activity during SWRs.

Hippocampal CA1 Neurons Represent Positive Feedback During the Learning Process of an Associative Memory Task

The hippocampus is crucial for forming associations between environmental stimuli. However, it is unclear how neural activities of hippocampal neurons dynamically change during the learning process. To address this question, we developed an associative memory task for rats with auditory stimuli. In this task, the rats were required to associate tone pitches (high and low) and ports (right and left) to obtain a reward. We recorded the firing activity of neurons in rats hippocampal CA1 during the learning process of the task. As a result, many hippocampal CA1 neurons increased their firing rates when the rats received a reward after choosing either the left or right port. We referred to these cells as "reward-direction cells." Furthermore, the proportion of the reward-direction cells increased in the middle-stage of learning but decreased after the completion of learning. This result suggests that the activity of reward-direction cells might serve as "positive feedback" signal that facilitates the formation of associations between tone pitches and port choice.

Learning offline: memory replay in biological and artificial reinforcement learning

Learning to act in an environment to maximise rewards is among the brain's key functions. This process has often been conceptualised within the framework of reinforcement learning, which has also gained prominence in machine learning and artificial intelligence (AI) as a way to optimise decision making. A common aspect of both biological and machine reinforcement learning is the reactivation of previously experienced episodes, referred to as replay. Replay is important for memory consolidation in biological neural networks and is key to stabilising learning in deep neural networks. Here, we review recent developments concerning the functional roles of replay in the fields of neuroscience and AI. Complementary progress suggests how replay might support learning processes, including generalisation and continual learning, affording opportunities to transfer knowledge across the two fields to advance the understanding of biological and artificial learning and memory.

Theta-Range Oscillations in Stress-Induced Mental Disorders as an Oscillotherapeutic Target

Emotional behavior and psychological disorders are expressed through coordinated interactions across multiple brain regions. Brain electrophysiological signals are composed of diverse neuronal oscillations, representing cell-level to region-level neuronal activity patterns, and serve as a biomarker of mental disorders. Here, we review recent observations from rodents demonstrating how neuronal oscillations in the hippocampus, amygdala, and prefrontal cortex are engaged in emotional behavior and altered by psychiatric changes such as anxiety and depression. In particular, we focus mainly on theta-range (4–12 Hz) oscillations, including several distinct oscillations in this frequency range. We then discuss therapeutic possibilities related to controlling such mental disease-related neuronal oscillations to ameliorate psychiatric symptoms and disorders in rodents and humans.

Experience replay is associated with efficient nonlocal learning

To make effective decisions, people need to consider the relationship between actions and outcomes. These are often separated by time and space. The neural mechanisms by which disjoint actions and outcomes are linked remain unknown. One promising hypothesis involves neural replay of nonlocal experience. Using a task that segregates direct from indirect value learning, combined with magnetoencephalography, we examined the role of neural replay in human nonlocal learning. After receipt of a reward, we found significant backward replay of nonlocal experience, with a 160-millisecond state-to-state time lag, which was linked to efficient learning of action values. Backward replay and behavioral evidence of nonlocal learning were more pronounced for experiences of greater benefit for future behavior. These findings support nonlocal replay as a neural mechanism for solving complex credit assignment problems during learning.

Robot navigation as hierarchical active inference

Localization and mapping has been a long standing area of research, both in neuroscience, to understand how mammals navigate their environment, as well as in robotics, to enable autonomous mobile robots. In this paper, we treat navigation as inferring actions that minimize (expected) variational free energy under a hierarchical generative model. We find that familiar concepts like perception, path integration, localization and mapping naturally emerge from this active inference formulation. Moreover, we show that this model is consistent with models of hippocampal functions, and can be implemented in silico on a real-world robot. Our experiments illustrate that a robot equipped with our hierarchical model is able to generate topologically consistent maps, and correct navigation behaviour is inferred when a goal location is provided to the system.

Prefrontal synaptic activation during hippocampal memory reactivation

Cooperative reactivation of hippocampal and prefrontal neurons is considered crucial for mnemonic processes. To directly record synaptic substances supporting the interregional interactions, we develop concurrent spike recordings of hippocampal neuronal ensembles and whole-cell patch-clamp recordings of medial prefrontal neurons in awake rats. We find that medial prefrontal neurons depolarize when hippocampal neurons synchronize. The depolarization in medial prefrontal neurons is larger when hippocampal place cells that encoded overlapping place fields and place cells that encoded a novel environment are synchronously reactivated. Our results suggest a functional circuit-synapse association that enables prefrontal neurons to read out specific memory traces from the hippocampus.