Hippocampal remapping as hidden state inference

A memory model of rodent spatial navigation in which place cells are memories arranged in a grid and grid cells are non-spatial

A theory and neurocomputational model are presented that explain grid cell responses as the byproduct of equally dissimilar hippocampal memories. On this account, place and grid cells are best understood as the natural consequence of memory encoding and retrieval; a precise hexagonal grid is the exception rather than the rule, emerging when the animal explores a large surface that is devoid of landmarks and objects. In the proposed memory model, place cells represent memories that are conjunctions of both spatial and non-spatial attributes, and grid cells primarily represent the non-spatial attributes (e.g. sounds, surface texture, etc.) found throughout the two-dimensional recording enclosure. Place cells support memories of the locations where non-spatial attributes can be found (e.g. positions with a particular sound), which are arranged in a hexagonal lattice owing to memory encoding and consolidation processes (pattern separation) as applied to situations in which the non-spatial attributes are found at all locations of a two-dimensional surface. Grid cells exhibit their spatial firing pattern owing to feedback from hippocampal place cells (i.e. a hexagonal pattern of remembered locations for the non-spatial attribute represented by a grid cell). Model simulations explain a wide variety of results in the rodent spatial navigation literature.

Mechanisms of experience-dependent place-cell referencing in hippocampal area CA1

Hippocampal CA1 place cells (PCs) encode both space- and goal-referenced information to support a cognitive map. The mechanism of this referencing and the role of experience remain poorly understood. Here we longitudinally recorded PC activity while head-fixed mice performed a spatial learning task on a treadmill. In a familiar environment, the CA1 representation consisted of PCs that were referenced to either specific spatial locations or a reward goal in approximately equal proportions; however, the CA1 representation became predominately goal-referenced upon exposure to a novel environment, as space-referenced PCs adaptively switched reference frames. Intracellular membrane potential recordings revealed that individual CA1 neurons simultaneously received both space- and goal-referenced synaptic inputs, and the ratio of these inputs was correlated with individual PC referencing. Furthermore, behavioral timescale synaptic plasticity shaped PC referencing. Together, these results suggest that experience-dependent adjustment of synaptic input shapes PC referencing to support a flexible cognitive map.

Learning produces an orthogonalized state machine in the hippocampus

Cognitive maps confer animals with flexible intelligence by representing spatial, temporal and abstract relationships that can be used to shape thought, planning and behaviour. Cognitive maps have been observed in the hippocampus1, but their algorithmic form and learning mechanisms remain obscure. Here we used large-scale, longitudinal two-photon calcium imaging to record activity from thousands of neurons in the CA1 region of the hippocampus while mice learned to efficiently collect rewards from two subtly different linear tracks in virtual reality. Throughout learning, both animal behaviour and hippocampal neural activity progressed through multiple stages, gradually revealing improved task representation that mirrored improved behavioural efficiency. The learning process involved progressive decorrelations in initially similar hippocampal neural activity within and across tracks, ultimately resulting in orthogonalized representations resembling a state machine capturing the inherent structure of the task. This decorrelation process was driven by individual neurons acquiring task-state-specific responses (that is, 'state cells'). Although various standard artificial neural networks did not naturally capture these dynamics, the clone-structured causal graph, a hidden Markov model variant, uniquely reproduced both the final orthogonalized states and the learning trajectory seen in animals. The observed cellular and population dynamics constrain the mechanisms underlying cognitive map formation in the hippocampus, pointing to hidden state inference as a fundamental computational principle, with implications for both biological and artificial intelligence.

Striatal arbitration between choice strategies guides few-shot adaptation

Animals often exhibit rapid action changes in context-switching environments. This study hypothesized that, compared to the expected outcome, an unexpected outcome leads to distinctly different action-selection strategies to guide rapid adaptation. We designed behavioral measures differentiating between trial-by-trial dynamics after expected and unexpected events. In various reversal learning data with different rodent species and task complexities, conventional learning models failed to replicate the choice behavior following an unexpected outcome. This discrepancy was resolved by the proposed model with two different decision variables contingent on outcome expectation: the support-stay and conflict-shift bias. Electrophysiological data analyses revealed that striatal neurons encode our model's key variables. Furthermore, the inactivation of striatal direct and indirect pathways neutralizes the effect of past expected and unexpected outcomes, respectively, on the action-selection strategy following an unexpected outcome. Our study suggests unique roles of the striatum in arbitrating between different action selection strategies for few-shot adaptation.

Context-dependent decision-making in the primate hippocampal-prefrontal circuit

What is good in one scenario may be bad in another. Despite the ubiquity of such contextual reasoning in everyday choice, how the brain flexibly uses different valuation schemes across contexts remains unknown. We addressed this question by monitoring neural activity from the hippocampus (HPC) and orbitofrontal cortex (OFC) of two monkeys performing a state-dependent choice task. We found that HPC neurons encoded state information as it became available and then, at the time of choice, relayed this information to the OFC via theta synchronization. During choice, the OFC represented value in a state-dependent manner; many OFC neurons uniquely coded for value in only one state but not the other. This suggests a functional dissociation whereby the HPC encodes contextual information that is broadcast to the OFC via theta synchronization to select a state-appropriate value subcircuit, thereby allowing for contextual reasoning in value-based choice.

Global remapping emerges as the mechanism for renewal of context-dependent behavior in a reinforcement learning model

Introduction

The hippocampal formation exhibits complex and context-dependent activity patterns and dynamics, e.g., place cell activity during spatial navigation in rodents or remapping of place fields when the animal switches between contexts. Furthermore, rodents show context-dependent renewal of extinguished behavior. However, the link between context-dependent neural codes and context-dependent renewal is not fully understood.

Methods

We use a deep neural network-based reinforcement learning agent to study the learning dynamics that occur during spatial learning and context switching in a simulated ABA extinction and renewal paradigm in a 3D virtual environment.

Results

Despite its simplicity, the network exhibits a number of features typically found in the CA1 and CA3 regions of the hippocampus. A significant proportion of neurons in deeper layers of the network are tuned to a specific spatial position of the agent in the environment-similar to place cells in the hippocampus. These complex spatial representations and dynamics occur spontaneously in the hidden layer of a deep network during learning. These spatial representations exhibit global remapping when the agent is exposed to a new context. The spatial maps are restored when the agent returns to the previous context, accompanied by renewal of the conditioned behavior. Remapping is facilitated by memory replay of experiences during training.

Discussion

Our results show that integrated codes that jointly represent spatial and task-relevant contextual variables are the mechanism underlying renewal in a simulated DQN agent.

Navigating uncertainty: reward location variability induces reorganization of hippocampal spatial representations

Navigating uncertainty is crucial for survival, with the location and availability of reward varying in different and unsignalled ways. Hippocampal place cell populations over-represent salient locations in an animal's environment, including those associated with rewards; however, how the spatial uncertainties impact the cognitive map is unclear. We report a virtual spatial navigation task designed to test the impact of different levels and types of uncertainty about reward on place cell populations. When the reward location changed on a trial-by-trial basis, inducing expected uncertainty, a greater proportion of place cells followed along, and the reward and the track end became anchors of a warped spatial metric. When the reward location then unexpectedly moved, the fraction of reward place cells that followed was greater when starting from a state of expected, compared to low, uncertainty. Overall, we show that different forms of potentially interacting uncertainty generate remapping in parallel, task-relevant, reference frames.

Contextualizing predictive minds

The structure of human memory seems to be optimized for efficient prediction, planning, and behavior. We propose that these capacities rely on a tripartite structure of memory that includes concepts, events, and contexts-three layers that constitute the mental world model. We suggest that the mechanism that critically increases adaptivity and flexibility is the tendency to contextualize. This tendency promotes local, context-encoding abstractions, which focus event- and concept-based planning and inference processes on the task and situation at hand. As a result, cognitive contextualization offers a solution to the frame problem-the need to select relevant features of the environment from the rich stream of sensorimotor signals. We draw evidence for our proposal from developmental psychology and neuroscience. Adopting a computational stance, we present evidence from cognitive modeling research which suggests that context sensitivity is a feature that is critical for maximizing the efficiency of cognitive processes. Finally, we turn to recent deep-learning architectures which independently demonstrate how context-sensitive memory can emerge in a self-organized learning system constrained by cognitively-inspired inductive biases.

Hidden state inference requires abstract contextual representations in the ventral hippocampus

The ability to use subjective, latent contextual representations to influence decision-making is crucial for everyday life. The hippocampus is hypothesized to bind together otherwise abstract combinations of stimuli to represent such latent contexts, to support the process of hidden state inference. Yet evidence for a role of the hippocampus in hidden state inference remains limited. We found that the ventral hippocampus is required for mice to perform hidden state inference during a two-armed bandit task. Hippocampal neurons differentiate the two abstract contexts required for this strategy in a manner similar to the differentiation of spatial locations, and their activity is essential for appropriate dopamine dynamics. These findings offer insight into how latent contextual information is used to optimize decisions, and they emphasize a key role for the hippocampus in hidden state inference.

A universal hippocampal memory code across animals and environments

How learning is affected by context is a fundamental question of neuroscience, as the ability to generalize learning to different contexts is necessary for navigating the world. An example of swift contextual generalization is observed in conditioning tasks, where performance is quickly generalized from one context to another. A key question in identifying the neural substrate underlying this ability is how the hippocampus (HPC) represents task-related stimuli across different environments, given that HPC cells exhibit place-specific activity that changes across contexts (remapping). In this study, we used calcium imaging to monitor hippocampal neuron activity as rats performed a conditioning task across multiple spatial contexts. We investigated whether hippocampal cells, which encode both spatial locations (place cells) and task-related information, could maintain their task representation even when their spatial encoding remapped in a new spatial context. To assess the consistency of task representations, we used advanced dimensionality reduction techniques combined with machine learning to develop manifold representations of population level HPC activity. The results showed that task-related neural representations remained stable even as place cell representations of spatial context changed, thus demonstrating similar embedding geometries of neural representations of the task across different spatial contexts. Notably, these patterns were not only consistent within the same animal across different contexts but also significantly similar across different animals, suggesting a standardized neural encoding or 'neural syntax' in the hippocampus. These findings bridge a critical gap between memory and navigation research, revealing how the hippocampus maintains cognitive consistency across different spatial environments. These findings also suggest that hippocampal function is governed by a neural framework shared between animals, an observation that may have broad implications for understanding memory, learning, and related cognitive processes. Looking ahead, this work opens new avenues for exploring the fundamental principles underlying hippocampal encoding strategies.

Control when confidence is costly

We develop a version of stochastic control that accounts for computational costs of inference. Past studies identified efficient coding without control, or efficient control that neglects the cost of synthesizing information. Here we combine these concepts into a framework where agents rationally approximate inference for efficient control. Specifically, we study Linear Quadratic Gaussian (LQG) control with an added internal cost on the relative precision of the posterior probability over the world state. This creates a trade-off: an agent can obtain more utility overall by sacrificing some task performance, if doing so saves enough bits during inference. We discover that the rational strategy that solves the joint inference and control problem goes through phase transitions depending on the task demands, switching from a costly but optimal inference to a family of suboptimal inferences related by rotation transformations, each misestimate the stability of the world. In all cases, the agent moves more to think less. This work provides a foundation for a new type of rational computations that could be used by both brains and machines for efficient but computationally constrained control.

Repulsion of CA3 / dentate gyrus representations is driven by distinct internal beliefs in the face of ambiguous sensory input

Recent human neuroimaging studies of episodic memory have revealed a counterintuitive phenomenon in the hippocampus: when events are highly similar, corresponding hippocampal activity patterns are sometimes less correlated than activity patterns associated with unrelated events. This phenomenon-repulsion-is not accounted for by most theories of the hippocampus, and the conditions that trigger repulsion remain poorly understood. Here, we used a spatial route-learning task and high-resolution fMRI in humans to test whether hippocampal repulsion is fundamentally driven by internal beliefs about the environment. By precisely measuring participants' internal beliefs and actively manipulating them, we show that repulsion selectively occurred in hippocampal subfields CA3 and dentate gyrus when visual input was ambiguous-or even identical-but internal beliefs were distinct. These findings firmly establish conditions that elicit repulsion and have broad relevance to theories of hippocampal function and to the fields of human episodic memory and rodent spatial navigation.

Predictive Representations: Building Blocks of Intelligence

Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This review integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation and its generalizations, which have been widely applied as both engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.

A population code for spatial representation in the zebrafish telencephalon

Spatial learning in teleost fish requires an intact telencephalon1, a brain region that contains putative analogues to components of the mammalian limbic system (for example, hippocampus)2-4. However, cells fundamental to spatial cognition in mammals-for example, place cells (PCs)5,6-have yet to be established in any fish species. In this study, using tracking microscopy to record brain-wide calcium activity in freely swimming larval zebrafish7, we compute the spatial information content8 of each neuron across the brain. Strikingly, in every recorded animal, cells with the highest spatial specificity were enriched in the zebrafish telencephalon. These PCs form a population code of space from which we can decode the animal's spatial location across time. By continuous recording of population-level activity, we found that the activity manifold of PCs refines and untangles over time. Through systematic manipulation of allothetic and idiothetic cues, we demonstrate that zebrafish PCs integrate multiple sources of information and can flexibly remap to form distinct spatial maps. Using analysis of neighbourhood distance between PCs across environments, we found evidence for a weakly preconfigured network in the telencephalon. The discovery of zebrafish PCs represents a step forward in our understanding of spatial cognition across species and the functional role of the early vertebrate telencephalon.

The estrous cycle modulates hippocampal spine dynamics, dendritic processing, and spatial coding

Histological evidence suggests that the estrous cycle exerts a powerful effect on CA1 neurons in mammalian hippocampus. Decades have passed since this landmark observation, yet how the estrous cycle shapes dendritic spine dynamics and hippocampal spatial coding in vivo remains a mystery. Here, we used a custom hippocampal microperiscope and two-photon calcium imaging to track CA1 pyramidal neurons in female mice over multiple cycles. Estrous cycle stage had a potent effect on spine dynamics, with heightened density during periods of greater estradiol (proestrus). These morphological changes were accompanied by greater somatodendritic coupling and increased infiltration of back-propagating action potentials into the apical dendrite. Finally, tracking CA1 response properties during navigation revealed enhanced place field stability during proestrus, evident at the single-cell and population level. These results establish the estrous cycle as a driver of large-scale structural and functional plasticity in hippocampal circuits essential for learning and memory.

Space is a latent sequence: A theory of the hippocampus

Fascinating phenomena such as landmark vector cells and splitter cells are frequently discovered in the hippocampus. Without a unifying principle, each experiment seemingly uncovers new anomalies or coding types. Here, we provide a unifying principle that the mental representation of space is an emergent property of latent higher-order sequence learning. Treating space as a sequence resolves numerous phenomena and suggests that the place field mapping methodology that interprets sequential neuronal responses in Euclidean terms might itself be a source of anomalies. Our model, clone-structured causal graph (CSCG), employs higher-order graph scaffolding to learn latent representations by mapping aliased egocentric sensory inputs to unique contexts. Learning to compress sequential and episodic experiences using CSCGs yields allocentric cognitive maps that are suitable for planning, introspection, consolidation, and abstraction. By explicating the role of Euclidean place field mapping and demonstrating how latent sequential representations unify myriad observed phenomena, our work positions the hippocampus in a sequence-centric paradigm, challenging the prevailing space-centric view.

Fundamental processes in sensorimotor learning: Reasoning, refinement, and retrieval

Motor learning is often viewed as a unitary process that operates outside of conscious awareness. This perspective has led to the development of sophisticated models designed to elucidate the mechanisms of implicit sensorimotor learning. In this review, we argue for a broader perspective, emphasizing the contribution of explicit strategies to sensorimotor learning tasks. Furthermore, we propose a theoretical framework for motor learning that consists of three fundamental processes: reasoning, the process of understanding action-outcome relationships; refinement, the process of optimizing sensorimotor and cognitive parameters to achieve motor goals; and retrieval, the process of inferring the context and recalling a control policy. We anticipate that this '3R' framework for understanding how complex movements are learned will open exciting avenues for future research at the intersection between cognition and action.

Grid Cells in Cognition: Mechanisms and Function

The activity patterns of grid cells form distinctively regular triangular lattices over the explored spatial environment and are largely invariant to visual stimuli, animal movement, and environment geometry. These neurons present numerous fascinating challenges to the curious (neuro)scientist: What are the circuit mechanisms responsible for creating spatially periodic activity patterns from the monotonic input-output responses of single neurons? How and why does the brain encode a local, nonperiodic variable-the allocentric position of the animal-with a periodic, nonlocal code? And, are grid cells truly specialized for spatial computations? Otherwise, what is their role in general cognition more broadly? We review efforts in uncovering the mechanisms and functional properties of grid cells, highlighting recent progress in the experimental validation of mechanistic grid cell models, and discuss the coding properties and functional advantages of the grid code as suggested by continuous attractor network models of grid cells.

Prediction error determines how memories are organized in the brain

How is new information organized in memory? According to latent state theories, this is determined by the level of surprise, or prediction error, generated by the new information: a small prediction error leads to the updating of existing memory, large prediction error leads to encoding of a new memory. We tested this idea using a protocol in which rats were first conditioned to fear a stimulus paired with shock. The stimulus was then gradually extinguished by progressively reducing the shock intensity until the stimulus was presented alone. Consistent with latent state theories, this gradual extinction protocol (small prediction errors) was better than standard extinction (large prediction errors) in producing long-term suppression of fear responses, and the benefit of gradual extinction was due to updating of the conditioning memory with information about extinction. Thus, prediction error determines how new information is organized in memory, and latent state theories adequately describe the ways in which this occurs.

The homogenous hippocampus: How hippocampal cells process available and potential goals

We present here a view of the firing patterns of hippocampal cells that is contrary, both functionally and anatomically, to conventional wisdom. We argue that the hippocampus responds to efference copies of goals encoded elsewhere; and that it uses these to detect and resolve conflict or interference between goals in general. While goals can involve space, hippocampal cells do not encode spatial (or other special types of) memory, as such. We also argue that the transverse circuits of the hippocampus operate in an essentially homogeneous way along its length. The apparently different functions of different parts (e.g. memory retrieval versus anxiety) result from the different (situational/motivational) inputs on which those parts perform the same fundamental computational operations. On this view, the key role of the hippocampus is the iterative adjustment, via Papez-like circuits, of synaptic weights in cell assemblies elsewhere.

Retrieval of contextual memory can be predicted by CA3 remapping and is differentially influenced by NMDAR activity in rat hippocampus subregions

Episodic memory is essential to navigate in a changing environment by recalling past events, creating new memories, and updating stored information from experience. Although the mechanisms for acquisition and consolidation have been profoundly studied, much less is known about memory retrieval. Hippocampal spatial representations are key for retrieval of contextually guided episodic memories. Indeed, hippocampal place cells exhibit stable location-specific activity which is thought to support contextual memory, but can also undergo remapping in response to environmental changes. It is unclear if remapping is directly related to the expression of different episodic memories. Here, using an incidental memory recognition task in rats, we showed that retrieval of a contextually guided memory is reflected by the levels of CA3 remapping, demonstrating a clear link between external cues, hippocampal remapping, and episodic memory retrieval that guides behavior. Furthermore, we describe NMDARs as key players in regulating the balance between retrieval and memory differentiation processes by controlling the reactivation of specific memory traces. While an increase in CA3 NMDAR activity boosts memory retrieval, dentate gyrus NMDAR activity enhances memory differentiation. Our results contribute to understanding how the hippocampal circuit sustains a flexible balance between memory formation and retrieval depending on the environmental cues and the internal representations of the individual. They also provide new insights into the molecular mechanisms underlying the contributions of hippocampal subregions to generate this balance.

Hippocampal and orbitofrontal neurons contribute to complementary aspects of associative structure

The ability to establish associations between environmental stimuli is fundamental for higher-order brain functions like state inference and generalization. Both the hippocampus and orbitofrontal cortex (OFC) play pivotal roles in this, demonstrating complex neural activity changes after associative learning. However, how precisely they contribute to representing learned associations remains unclear. Here, we train head-restrained mice to learn four 'odor-outcome' sequence pairs composed of several task variables-the past and current odor cues, sequence structure of 'cue-outcome' arrangement, and the expected outcome; and perform calcium imaging from these mice throughout learning. Sequence-splitting signals that distinguish between paired sequences are detected in both brain regions, reflecting associative memory formation. Critically, we uncover differential contents in represented associations by examining, in each area, how these task variables affect splitting signal generalization between sequence pairs. Specifically, the hippocampal splitting signals are influenced by the combination of past and current cues that define a particular sensory experience. In contrast, the OFC splitting signals are similar between sequence pairs that share the same sequence structure and expected outcome. These findings suggest that the hippocampus and OFC uniquely and complementarily organize the acquired associative structure.

Representational drift as a result of implicit regularization

Recent studies show that, even in constant environments, the tuning of single neurons changes over time in a variety of brain regions. This representational drift has been suggested to be a consequence of continuous learning under noise, but its properties are still not fully understood. To investigate the underlying mechanism, we trained an artificial network on a simplified navigational task. The network quickly reached a state of high performance, and many units exhibited spatial tuning. We then continued training the network and noticed that the activity became sparser with time. Initial learning was orders of magnitude faster than ensuing sparsification. This sparsification is consistent with recent results in machine learning, in which networks slowly move within their solution space until they reach a flat area of the loss function. We analyzed four datasets from different labs, all demonstrating that CA1 neurons become sparser and more spatially informative with exposure to the same environment. We conclude that learning is divided into three overlapping phases: (i) Fast familiarity with the environment; (ii) slow implicit regularization; and (iii) a steady state of null drift. The variability in drift dynamics opens the possibility of inferring learning algorithms from observations of drift statistics.

Conjunctive encoding of exploratory intentions and spatial information in the hippocampus

The hippocampus creates a cognitive map of the external environment by encoding spatial and self-motion-related information. However, it is unclear whether hippocampal neurons could also incorporate internal cognitive states reflecting an animal's exploratory intention, which is not driven by rewards or unexpected sensory stimuli. In this study, a subgroup of CA1 neurons was found to encode both spatial information and animals' investigatory intentions in male mice. These neurons became active before the initiation of exploration behaviors at specific locations and were nearly silent when the same fields were traversed without exploration. Interestingly, this neuronal activity could not be explained by object features, rewards, or mismatches in environmental cues. Inhibition of the lateral entorhinal cortex decreased the activity of these cells during exploration. Our findings demonstrate that hippocampal neurons may bridge external and internal signals, indicating a potential connection between spatial representation and intentional states in the construction of internal navigation systems.

Representational drift as a result of implicit regularization

Recent studies show that, even in constant environments, the tuning of single neurons changes over time in a variety of brain regions. This representational drift has been suggested to be a consequence of continuous learning under noise, but its properties are still not fully understood. To investigate the underlying mechanism, we trained an artificial network on a simplified navigational task. The network quickly reached a state of high performance, and many units exhibited spatial tuning. We then continued training the network and noticed that the activity became sparser with time. Initial learning was orders of magnitude faster than ensuing sparsification. This sparsification is consistent with recent results in machine learning, in which networks slowly move within their solution space until they reach a flat area of the loss function. We analyzed four datasets from different labs, all demonstrating that CA1 neurons become sparser and more spatially informative with exposure to the same environment. We conclude that learning is divided into three overlapping phases: (i) Fast familiarity with the environment; (ii) slow implicit regularization; (iii) a steady state of null drift. The variability in drift dynamics opens the possibility of inferring learning algorithms from observations of drift statistics.

Hippocampal sequences span experience relative to rewards

Hippocampal place cells fire in sequences that span spatial environments and non-spatial modalities, suggesting that hippocampal activity can anchor to the most behaviorally salient aspects of experience. As reward is a highly salient event, we hypothesized that sequences of hippocampal activity can anchor to rewards. To test this, we performed two-photon imaging of hippocampal CA1 neurons as mice navigated virtual environments with changing hidden reward locations. When the reward moved, the firing fields of a subpopulation of cells moved to the same relative position with respect to reward, constructing a sequence of reward-relative cells that spanned the entire task structure. The density of these reward-relative sequences increased with task experience as additional neurons were recruited to the reward-relative population. Conversely, a largely separate subpopulation maintained a spatially-based place code. These findings thus reveal separate hippocampal ensembles can flexibly encode multiple behaviorally salient reference frames, reflecting the structure of the experience.

Efficient, continual, and generalized learning in the brain - neural mechanism of Mental Schema 2.0

There has been tremendous progress in artificial neural networks (ANNs) over the past decade; however, the gap between ANNs and the biological brain as a learning device remains large. With the goal of closing this gap, this paper reviews learning mechanisms in the brain by focusing on three important issues in ANN research: efficiency, continuity, and generalization. We first discuss the method by which the brain utilizes a variety of self-organizing mechanisms to maximize learning efficiency, with a focus on the role of spontaneous activity of the brain in shaping synaptic connections to facilitate spatiotemporal learning and numerical processing. Then, we examined the neuronal mechanisms that enable lifelong continual learning, with a focus on memory replay during sleep and its implementation in brain-inspired ANNs. Finally, we explored the method by which the brain generalizes learned knowledge in new situations, particularly from the mathematical generalization perspective of topology. Besides a systematic comparison in learning mechanisms between the brain and ANNs, we propose "Mental Schema 2.0," a new computational property underlying the brain's unique learning ability that can be implemented in ANNs.

Goal-seeking compresses neural codes for space in the human hippocampus and orbitofrontal cortex

Humans can navigate flexibly to meet their goals. Here, we asked how the neural representation of allocentric space is distorted by goal-directed behavior. Participants navigated an agent to two successive goal locations in a grid world environment comprising four interlinked rooms, with a contextual cue indicating the conditional dependence of one goal location on another. Examining the neural geometry by which room and context were encoded in fMRI signals, we found that map-like representations of the environment emerged in both hippocampus and neocortex. Cognitive maps in hippocampus and orbitofrontal cortices were compressed so that locations cued as goals were coded together in neural state space, and these distortions predicted successful learning. This effect was captured by a computational model in which current and prospective locations are jointly encoded in a place code, providing a theory of how goals warp the neural representation of space in macroscopic neural signals.

The Computational and Neural Bases of Context-Dependent Learning

Flexible behavior requires the creation, updating, and expression of memories to depend on context. While the neural underpinnings of each of these processes have been intensively studied, recent advances in computational modeling revealed a key challenge in context-dependent learning that had been largely ignored previously: Under naturalistic conditions, context is typically uncertain, necessitating contextual inference. We review a theoretical approach to formalizing context-dependent learning in the face of contextual uncertainty and the core computations it requires. We show how this approach begins to organize a large body of disparate experimental observations, from multiple levels of brain organization (including circuits, systems, and behavior) and multiple brain regions (most prominently the prefrontal cortex, the hippocampus, and motor cortices), into a coherent framework. We argue that contextual inference may also be key to understanding continual learning in the brain. This theory-driven perspective places contextual inference as a core component of learning.

Remapping in a recurrent neural network model of navigation and context inference

Neurons in navigational brain regions provide information about position, orientation, and speed relative to environmental landmarks. These cells also change their firing patterns ('remap') in response to changing contextual factors such as environmental cues, task conditions, and behavioral states, which influence neural activity throughout the brain. How can navigational circuits preserve their local computations while responding to global context changes? To investigate this question, we trained recurrent neural network models to track position in simple environments while at the same time reporting transiently-cued context changes. We show that these combined task constraints (navigation and context inference) produce activity patterns that are qualitatively similar to population-wide remapping in the entorhinal cortex, a navigational brain region. Furthermore, the models identify a solution that generalizes to more complex navigation and inference tasks. We thus provide a simple, general, and experimentally-grounded model of remapping as one neural circuit performing both navigation and context inference.

Rethinking the hippocampal cognitive map as a meta-learning computational module

A hallmark of biological intelligence is the ability to adaptively draw on past experience to guide behaviour under novel situations. Yet, the neurobiological principles that underlie this form of meta-learning remain relatively unexplored. In this Opinion, we review the existing literature on hippocampal spatial representations and reinforcement learning theory and describe a novel theoretical framework that aims to account for biological meta-learning. We conjecture that so-called hippocampal cognitive maps of familiar environments are part of a larger meta-representation (meta-map) that encodes information states and sources, which support exploration and provides a foundation for learning. We also introduce concrete hypotheses on how these generic states can be encoded using a principle of superposition.

Emergence of a predictive model in the hippocampus

The brain organizes experiences into memories that guide future behavior. Hippocampal CA1 population activity is hypothesized to reflect predictive models that contain information about future events, but little is known about how they develop. We trained mice on a series of problems with or without a common statistical structure to observe how memories are formed and updated. Mice that learned structured problems integrated their experiences into a predictive model that contained the solutions to upcoming novel problems. Retrieving the model during learning improved discrimination accuracy and facilitated learning. Using calcium imaging to track CA1 activity during learning, we found that hippocampal ensemble activity became more stable as mice formed a predictive model. The hippocampal ensemble was reactivated during training and incorporated new activity patterns from each training problem. These results show how hippocampal activity supports building predictive models by organizing new information with respect to existing memories.

Goal-oriented representations in the human hippocampus during planning and navigation

Recent work in cognitive and systems neuroscience has suggested that the hippocampus might support planning, imagination, and navigation by forming cognitive maps that capture the abstract structure of physical spaces, tasks, and situations. Navigation involves disambiguating similar contexts, and the planning and execution of a sequence of decisions to reach a goal. Here, we examine hippocampal activity patterns in humans during a goal-directed navigation task to investigate how contextual and goal information are incorporated in the construction and execution of navigational plans. During planning, hippocampal pattern similarity is enhanced across routes that share a context and a goal. During navigation, we observe prospective activation in the hippocampus that reflects the retrieval of pattern information related to a key-decision point. These results suggest that, rather than simply representing overlapping associations or state transitions, hippocampal activity patterns are shaped by context and goals.

Remapping in a recurrent neural network model of navigation and context inference

Neurons in navigational brain regions provide information about position, orientation, and speed relative to environmental landmarks. These cells also change their firing patterns ("remap") in response to changing contextual factors such as environmental cues, task conditions, and behavioral state, which influence neural activity throughout the brain. How can navigational circuits preserve their local computations while responding to global context changes? To investigate this question, we trained recurrent neural network models to track position in simple environments while at the same time reporting transiently-cued context changes. We show that these combined task constraints (navigation and context inference) produce activity patterns that are qualitatively similar to population-wide remapping in the entorhinal cortex, a navigational brain region. Furthermore, the models identify a solution that generalizes to more complex navigation and inference tasks. We thus provide a simple, general, and experimentally-grounded model of remapping as one neural circuit performing both navigation and context inference.

Spatial uncertainty and environmental geometry in navigation

Variations in the geometry of the environment, such as the shape and size of an enclosure, have profound effects on navigational behavior and its neural underpinning. Here, we show that these effects arise as a consequence of a single, unifying principle: to navigate efficiently, the brain must maintain and update the uncertainty about one's location. We developed an image-computable Bayesian ideal observer model of navigation, continually combining noisy visual and self-motion inputs, and a neural encoding model optimized to represent the location uncertainty computed by the ideal observer. Through mathematical analysis and numerical simulations, we show that the ideal observer accounts for a diverse range of sometimes paradoxical distortions of human homing behavior in anisotropic and deformed environments, including 'boundary tethering', and its neural encoding accounts for distortions of rodent grid cell responses under identical environmental manipulations. Our results demonstrate that spatial uncertainty plays a key role in navigation.

From cognitive maps to spatial schemas

A schema refers to a structured body of prior knowledge that captures common patterns across related experiences. Schemas have been studied separately in the realms of episodic memory and spatial navigation across different species and have been grounded in theories of memory consolidation, but there has been little attempt to integrate our understanding across domains, particularly in humans. We propose that experiences during navigation with many similarly structured environments give rise to the formation of spatial schemas (for example, the expected layout of modern cities) that share properties with but are distinct from cognitive maps (for example, the memory of a modern city) and event schemas (such as expected events in a modern city) at both cognitive and neural levels. We describe earlier theoretical frameworks and empirical findings relevant to spatial schemas, along with more targeted investigations of spatial schemas in human and non-human animals. Consideration of architecture and urban analytics, including the influence of scale and regionalization, on different properties of spatial schemas may provide a powerful approach to advance our understanding of spatial schemas.

Temporal context and latent state inference in the hippocampal splitter signal

The hippocampus is thought to enable the encoding and retrieval of ongoing experience, the organization of that experience into structured representations like contexts, maps, and schemas, and the use of these structures to plan for the future. A central goal is to understand what the core computations supporting these functions are, and how these computations are realized in the collective action of single neurons. A potential access point into this issue is provided by 'splitter cells', hippocampal neurons that fire differentially on the overlapping segment of trajectories that differ in their past and/or future. However, the literature on splitter cells has been fragmented and confusing, owing to differences in terminology, behavioral tasks, and analysis methods across studies. In this review, we synthesize consistent findings from this literature, establish a common set of terms, and translate between single-cell and ensemble perspectives. Most importantly, we examine the combined findings through the lens of two major theoretical ideas about hippocampal function: representation of temporal context and latent state inference. We find that unique signature properties of each of these models are necessary to account for the data, but neither theory, by itself, explains all of its features. Specifically, the temporal gradedness of the splitter signal is strong support for temporal context, but is hard to explain using state models, while its flexibility and task-dependence is naturally accounted for using state inference, but poses a challenge otherwise. These theories suggest a number of avenues for future work, and we believe their application to splitter cells is a timely and informative domain for testing and refining theoretical ideas about hippocampal function.

Contextual inference in learning and memory

Context is widely regarded as a major determinant of learning and memory across numerous domains, including classical and instrumental conditioning, episodic memory, economic decision-making, and motor learning. However, studies across these domains remain disconnected due to the lack of a unifying framework formalizing the concept of context and its role in learning. Here, we develop a unified vernacular allowing direct comparisons between different domains of contextual learning. This leads to a Bayesian model positing that context is unobserved and needs to be inferred. Contextual inference then controls the creation, expression, and updating of memories. This theoretical approach reveals two distinct components that underlie adaptation, proper and apparent learning, respectively referring to the creation and updating of memories versus time-varying adjustments in their expression. We review a number of extensions of the basic Bayesian model that allow it to account for increasingly complex forms of contextual learning.

Neural circuit dynamics of drug-context associative learning in the mouse hippocampus

The environmental context associated with previous drug consumption is a potent trigger for drug relapse. However, the mechanism by which neural representations of context are modified to incorporate information associated with drugs of abuse remains unknown. Using longitudinal calcium imaging in freely behaving mice, we find that unlike the associative learning of natural reward, drug-context associations for psychostimulants and opioids are encoded in a specific subset of hippocampal neurons. After drug conditioning, these neurons weakened their spatial coding for the non-drug paired context, resulting in an orthogonal representation for the drug versus non-drug context that was predictive of drug-seeking behavior. Furthermore, these neurons were selected based on drug-spatial experience and were exclusively tuned to animals' allocentric position. Together, this work reveals how drugs of abuse alter the hippocampal circuit to encode drug-context associations and points to the possibility of targeting drug-associated memory in the hippocampus.

Artificial intelligence insights into hippocampal processing

Advances in artificial intelligence, machine learning, and deep neural networks have led to new discoveries in human and animal learning and intelligence. A recent artificial intelligence agent in the DeepMind family, muZero, can complete a variety of tasks with limited information about the world in which it is operating and with high uncertainty about features of current and future space. To perform, muZero uses only three functions that are general yet specific enough to allow learning across a variety of tasks without overgeneralization across different contexts. Similarly, humans and animals are able to learn and improve in complex environments while transferring learning from other contexts and without overgeneralizing. In particular, the mammalian extrahippocampal system (eHPCS) can guide spatial decision making while simultaneously encoding and processing spatial and contextual information. Like muZero, the eHPCS is also able to adjust contextual representations depending on the degree and significance of environmental changes and environmental cues. In this opinion, we will argue that the muZero functions parallel those of the hippocampal system. We will show that the different components of the muZero model provide a framework for thinking about generalizable learning in the eHPCS, and that the evaluation of how transitions in cell representations occur between similar and distinct contexts can be informed by advances in artificial intelligence agents such as muZero. We additionally explain how advances in AI agents will provide frameworks and predictions by which to investigate the expected link between state changes and neuronal firing. Specifically, we will discuss testable predictions about the eHPCS, including the functions of replay and remapping, informed by the mechanisms behind muZero learning. We conclude with additional ways in which agents such as muZero can aid in illuminating prospective questions about neural functioning, as well as how these agents may shed light on potential expected answers.

Rats use strategies to make object choices in spontaneous object recognition tasks

Rodent spontaneous object recognition (SOR) paradigms are widely used to study the mechanisms of complex memory in many laboratories. Due to the absence of explicit reinforcement in these tasks, there is an underlying assumption that object exploratory behaviour is 'spontaneous'. However, rodents can strategise, readily adapting their behaviour depending on the current information available and prior predications formed from learning and memory. Here, using the object-place-context (episodic-like) recognition task and novel analytic methods relying on multiple trials within a single session, we demonstrate that rats use a context-based or recency-based object recognition strategy for the same types of trials, depending on task conditions. Exposure to occasional ambiguous conditions changed animals' responses towards a recency-based preference. However, more salient and predictable conditions led to animals exploring objects on the basis of episodic novelty reliant on contextual information. The results have important implications for future research using SOR tasks, especially in the way experimenters design, analyse and interpret object recognition experiments in non-human animals.

How to build a cognitive map

Learning and interpreting the structure of the environment is an innate feature of biological systems, and is integral to guiding flexible behaviors for evolutionary viability. The concept of a cognitive map has emerged as one of the leading metaphors for these capacities, and unraveling the learning and neural representation of such a map has become a central focus of neuroscience. In recent years, many models have been developed to explain cellular responses in the hippocampus and other brain areas. Because it can be difficult to see how these models differ, how they relate and what each model can contribute, this Review aims to organize these models into a clear ontology. This ontology reveals parallels between existing empirical results, and implies new approaches to understand hippocampal-cortical interactions and beyond.

The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning

We advance a novel computational theory of the hippocampal formation as a hierarchical generative model that organizes sequential experiences, such as rodent trajectories during spatial navigation, into coherent spatiotemporal contexts. We propose that the hippocampal generative model is endowed with inductive biases to identify individual items of experience (first hierarchical layer), organize them into sequences (second layer) and cluster them into maps (third layer). This theory entails a novel characterization of hippocampal reactivations as generative replay: the offline resampling of fictive sequences from the generative model, which supports the continual learning of multiple sequential experiences. We show that the model learns and efficiently retains multiple spatial navigation trajectories, by organizing them into spatial maps. Furthermore, the model reproduces flexible and prospective aspects of hippocampal dynamics that are challenging to explain within existing frameworks. This theory reconciles multiple roles of the hippocampal formation in map-based navigation, episodic memory and imagination.

Model Sharing in the Human Medial Temporal Lobe

Effective planning involves knowing where different actions take us. However, natural environments are rich and complex, leading to an exponential increase in memory demand as a plan grows in depth. One potential solution is to filter out features of the environment irrelevant to the task at hand. This enables a shared model of transition dynamics to be used for planning over a range of different input features. Here, we asked human participants (13 male, 16 female) to perform a sequential decision-making task, designed so that knowledge should be integrated independently of the input features (visual cues) present in one case but not in another. Participants efficiently switched between using a low-dimensional (cue independent) and a high-dimensional (cue specific) representation of state transitions. fMRI data identified the medial temporal lobe as a locus for learning state transitions. Within this region, multivariate patterns of BOLD responses were less correlated between trials with differing input features but similar state associations in the high dimensional than in the low dimensional case, suggesting that these patterns switched between separable (specific to input features) and shared (invariant to input features) transition models. Finally, we show that transition models are updated more strongly following the receipt of positive compared with negative outcomes, a finding that challenges conventional theories of planning. Together, these findings propose a computational and neural account of how information relevant for planning can be shared and segmented in response to the vast array of contextual features we encounter in our world.SIGNIFICANCE STATEMENT Effective planning involves maintaining an accurate model of which actions take us to which locations. But in a world awash with information, mapping actions to states with the right level of complexity is critical. Using a new decision-making "heist task" in conjunction with computational modeling and fMRI, we show that patterns of BOLD responses in the medial temporal lobe-a brain region key for prospective planning-become less sensitive to the presence of visual features when these are irrelevant to the task at hand. By flexibly adapting the complexity of task-state representations in this way, state-action mappings learned under one set of features can be used to plan in the presence of others.

Taking stock of value in the orbitofrontal cortex

People with damage to the orbitofrontal cortex (OFC) have specific problems making decisions, whereas their other cognitive functions are spared. Neurophysiological studies have shown that OFC neurons fire in proportion to the value of anticipated outcomes. Thus, a central role of the OFC is to guide optimal decision-making by signalling values associated with different choices. Until recently, this view of OFC function dominated the field. New data, however, suggest that the OFC may have a much broader role in cognition by representing cognitive maps that can be used to guide behaviour and that value is just one of many variables that are important for behavioural control. In this Review, we critically evaluate these two alternative accounts of OFC function and examine how they might be reconciled.

Animal-to-Animal Variability in Partial Hippocampal Remapping in Repeated Environments

Hippocampal place cells form a map of the environment of an animal. Changes in the hippocampal map can be brought about in a number of ways, including changes to the environment, task, internal state of the subject, and the passage of time. These changes in the hippocampal map have been called remapping. In this study, we examine remapping during repeated exposure to the same environment. Different animals can have different remapping responses to the same changes. This variability across animals in remapping behavior is not well understood. In this work, we analyzed electrophysiological recordings from the CA3 region of the hippocampus performed by Alme et al. (2014), in which five male rats were exposed to 11 different environments, including a variety of repetitions of those environments. To compare the hippocampal maps between two experiences, we computed average rate map correlation coefficients. We found changes in the hippocampal maps between different sessions in the same environment. These changes consisted of partial remapping, a form of remapping in which some place cells maintain their place fields, whereas other place cells remap their place fields. Each animal exhibited partial remapping differently. We discovered that the heterogeneity in hippocampal representational changes across animals is structured; individual animals had consistently different levels of partial remapping across a range of independent comparisons. Our findings highlight that partial hippocampal remapping between repeated environments depends on animal-specific factors.SIGNIFICANCE STATEMENT Context identification is a difficult problem. Animals are not provided with objective context identity labels, so they must infer which experiences come from which contexts. Different animals may have different strategies for performing this inference. We find that different animals have stereotypically different extents of partial hippocampal remapping, a neural correlate of subjective assessment of context identity.

Spatial goal coding in the hippocampal formation

The mammalian hippocampal formation contains several distinct populations of neurons involved in representing self-position and orientation. These neurons, which include place, grid, head direction, and boundary-vector cells, are thought to collectively instantiate cognitive maps supporting flexible navigation. However, to flexibly navigate, it is necessary to also maintain internal representations of goal locations, such that goal-directed routes can be planned and executed. Although it has remained unclear how the mammalian brain represents goal locations, multiple neural candidates have recently been uncovered during different phases of navigation. For example, during planning, sequential activation of spatial cells may enable simulation of future routes toward the goal. During travel, modulation of spatial cells by the prospective route, or by distance and direction to the goal, may allow maintenance of route and goal-location information, supporting navigation on an ongoing basis. As the goal is approached, an increased activation of spatial cells may enable the goal location to become distinctly represented within cognitive maps, aiding goal localization. Lastly, after arrival at the goal, sequential activation of spatial cells may represent the just-taken route, enabling route learning and evaluation. Here, we review and synthesize these and other evidence for goal coding in mammalian brains, relate the experimental findings to predictions from computational models, and discuss outstanding questions and future challenges.

Advances in modeling learning and decision-making in neuroscience

An organism's survival depends on its ability to learn about its environment and to make adaptive decisions in the service of achieving the best possible outcomes in that environment. To study the neural circuits that support these functions, researchers have increasingly relied on models that formalize the computations required to carry them out. Here, we review the recent history of computational modeling of learning and decision-making, and how these models have been used to advance understanding of prefrontal cortex function. We discuss how such models have advanced from their origins in basic algorithms of updating and action selection to increasingly account for complexities in the cognitive processes required for learning and decision-making, and the representations over which they operate. We further discuss how a deeper understanding of the real-world complexities in these computations has shed light on the fundamental constraints on optimal behavior, and on the complex interactions between corticostriatal pathways to determine such behavior. The continuing and rapid development of these models holds great promise for understanding the mechanisms by which animals adapt to their environments, and what leads to maladaptive forms of learning and decision-making within clinical populations.