Learning predictive cognitive maps with spiking neurons during behavior and replays

Impact of symmetry in local learning rules on predictive neural representations and generalization in spatial navigation

In spatial cognition, the Successor Representation (SR) from reinforcement learning provides a compelling candidate of how predictive representations are used to encode space. In particular, hippocampal place cells are hypothesized to encode the SR. Here, we investigate how varying the temporal symmetry in learning rules influences those representations. To this end, we use a simple local learning rule which can be made insensitive to the temporal order. We analytically find that a symmetric learning rule results in a successor representation under a symmetrized version of the experienced transition structure. We then apply this rule to a two-layer neural network model loosely resembling hippocampal subfields CA3 - with a symmetric learning rule and recurrent weights - and CA1 - with an asymmetric learning rule and no recurrent weights. Here, when exposed repeatedly to a linear track, neurons in our model in CA3 show less shift of the centre of mass than those in CA1, in line with existing empirical findings. Investigating the functional benefits of such symmetry, we employ a simple reinforcement learning agent which may learn symmetric or classical successor representations. Here, we find that using a symmetric learning rule yields representations which afford better generalization, when the agent is probed to navigate to a new target without relearning the SR. This effect is reversed when the state space is not symmetric anymore. Thus, our results hint at a potential benefit of the inductive bias afforded by symmetric learning rules in areas employed in spatial navigation, where there naturally is a symmetry in the state space.

Learning produces an orthogonalized state machine in the hippocampus

Cognitive maps confer animals with flexible intelligence by representing spatial, temporal and abstract relationships that can be used to shape thought, planning and behaviour. Cognitive maps have been observed in the hippocampus1, but their algorithmic form and learning mechanisms remain obscure. Here we used large-scale, longitudinal two-photon calcium imaging to record activity from thousands of neurons in the CA1 region of the hippocampus while mice learned to efficiently collect rewards from two subtly different linear tracks in virtual reality. Throughout learning, both animal behaviour and hippocampal neural activity progressed through multiple stages, gradually revealing improved task representation that mirrored improved behavioural efficiency. The learning process involved progressive decorrelations in initially similar hippocampal neural activity within and across tracks, ultimately resulting in orthogonalized representations resembling a state machine capturing the inherent structure of the task. This decorrelation process was driven by individual neurons acquiring task-state-specific responses (that is, 'state cells'). Although various standard artificial neural networks did not naturally capture these dynamics, the clone-structured causal graph, a hidden Markov model variant, uniquely reproduced both the final orthogonalized states and the learning trajectory seen in animals. The observed cellular and population dynamics constrain the mechanisms underlying cognitive map formation in the hippocampus, pointing to hidden state inference as a fundamental computational principle, with implications for both biological and artificial intelligence.

A non-Hebbian code for episodic memory

Hebbian plasticity has long dominated neurobiological models of memory formation. Yet, plasticity rules operating on one-shot episodic memory timescales rarely depend on both pre- and postsynaptic spiking, challenging Hebbian theory in this crucial regime. Here, we present an episodic memory model governed by a simpler rule depending only on presynaptic activity. We show that this rule, capitalizing on high-dimensional neural activity with restricted transitions, naturally stores episodes as paths through complex state spaces like those underlying a world model. The resulting memory traces, which we term path vectors, are highly expressive and decodable with an odor-tracking algorithm. We show that path vectors are robust alternatives to Hebbian traces, support one-shot sequential and associative recall, along with policy learning, and shed light on specific hippocampal plasticity rules. Thus, non-Hebbian plasticity is sufficient for flexible memory and learning and well-suited to encode episodes and policies as paths through a world model.

Noise Resilience of Successor and Predecessor Feature Algorithms in One- and Two-Dimensional Environments

Noisy inputs pose significant challenges for reinforcement learning (RL) agents navigating real-world environments. While animals demonstrate robust spatial learning under dynamic conditions, the mechanisms underlying this resilience remain understudied in RL frameworks. This paper introduces a novel comparative analysis of predecessor feature (PF) and successor feature (SF) algorithms under controlled noise conditions, revealing several insights. Our key innovation lies in demonstrating that SF algorithms achieve superior noise resilience compared to traditional approaches, with cumulative rewards of 2216.88±3.83 (mean ± SEM), even under high noise conditions (σ=0.5) in one-dimensional environments, while Q learning achieves only 19.22±0.57. In two-dimensional environments, we discover an unprecedented nonlinear relationship between noise level and algorithm performance, with SF showing optimal performance at moderate noise levels (σ=0.25), achieving cumulative rewards of 2886.03±1.63 compared to 2798.16±3.54 for Q learning. The λ parameter in PF learning is a significant factor, with λ=0.7 consistently achieving higher λ values under most noise conditions. These findings bridge computational neuroscience and RL, offering practical insights for developing noise-resistant learning systems. Our results have direct applications in robotics, autonomous navigation, and sensor-based AI systems, particularly in environments with inherent observational uncertainty.

Predictive Representations: Building Blocks of Intelligence

Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This review integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation and its generalizations, which have been widely applied as both engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.

Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze

In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (αr) and the eligibility trace decay rate (λ), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman's correlation tests and linear regression. Our findings reveal that an αr of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for λ varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience.

Linking cognitive strategy, neural mechanism, and movement statistics in group foraging behaviors

Foraging for food is a rich and ubiquitous animal behavior that involves complex cognitive decisions, and interactions between different individuals and species. There has been exciting recent progress in understanding multi-agent foraging behavior from cognitive, neuroscience, and statistical perspectives, but integrating these perspectives can be elusive. This paper seeks to unify these perspectives, allowing statistical analysis of observational animal movement data to shed light on the viability of cognitive models of foraging strategies. We start with cognitive agents with internal preferences expressed as value functions, and implement this in a biologically plausible neural network, and an equivalent statistical model, where statistical predictors of agents' movements correspond to the components of the value functions. We test this framework by simulating foraging agents and using Bayesian statistical modeling to correctly identify the factors that best predict the agents' behavior. As further validation, we use this framework to analyze an open-source locust foraging dataset. Finally, we collect new multi-agent real-world bird foraging data, and apply this method to analyze the preferences of different species. Together, this work provides an initial roadmap to integrate cognitive, neuroscience, and statistical approaches for reasoning about animal foraging in complex multi-agent environments.

Neuromorphic one-shot learning utilizing a phase-transition material

Design of hardware based on biological principles of neuronal computation and plasticity in the brain is a leading approach to realizing energy- and sample-efficient AI and learning machines. An important factor in selection of the hardware building blocks is the identification of candidate materials with physical properties suitable to emulate the large dynamic ranges and varied timescales of neuronal signaling. Previous work has shown that the all-or-none spiking behavior of neurons can be mimicked by threshold switches utilizing material phase transitions. Here, we demonstrate that devices based on a prototypical metal-insulator-transition material, vanadium dioxide (VO2), can be dynamically controlled to access a continuum of intermediate resistance states. Furthermore, the timescale of their intrinsic relaxation can be configured to match a range of biologically relevant timescales from milliseconds to seconds. We exploit these device properties to emulate three aspects of neuronal analog computation: fast (~1 ms) spiking in a neuronal soma compartment, slow (~100 ms) spiking in a dendritic compartment, and ultraslow (~1 s) biochemical signaling involved in temporal credit assignment for a recently discovered biological mechanism of one-shot learning. Simulations show that an artificial neural network using properties of VO2 devices to control an agent navigating a spatial environment can learn an efficient path to a reward in up to fourfold fewer trials than standard methods. The phase relaxations described in our study may be engineered in a variety of materials and can be controlled by thermal, electrical, or optical stimuli, suggesting further opportunities to emulate biological learning in neuromorphic hardware.

RatInABox, a toolkit for modelling locomotion and neuronal activity in continuous environments

Generating synthetic locomotory and neural data is a useful yet cumbersome step commonly required to study theoretical models of the brain's role in spatial navigation. This process can be time consuming and, without a common framework, makes it difficult to reproduce or compare studies which each generate test data in different ways. In response, we present RatInABox, an open-source Python toolkit designed to model realistic rodent locomotion and generate synthetic neural data from spatially modulated cell types. This software provides users with (i) the ability to construct one- or two-dimensional environments with configurable barriers and visual cues, (ii) a physically realistic random motion model fitted to experimental data, (iii) rapid online calculation of neural data for many of the known self-location or velocity selective cell types in the hippocampal formation (including place cells, grid cells, boundary vector cells, head direction cells) and (iv) a framework for constructing custom cell types, multi-layer network models and data- or policy-controlled motion trajectories. The motion and neural models are spatially and temporally continuous as well as topographically sensitive to boundary conditions and walls. We demonstrate that out-of-the-box parameter settings replicate many aspects of rodent foraging behaviour such as velocity statistics and the tendency of rodents to over-explore walls. Numerous tutorial scripts are provided, including examples where RatInABox is used for decoding position from neural data or to solve a navigational reinforcement learning task. We hope this tool will significantly streamline computational research into the brain's role in navigation.

Abstract cognitive maps of social network structure aid adaptive inference

Social navigation-such as anticipating where gossip may spread, or identifying which acquaintances can help land a job-relies on knowing how people are connected within their larger social communities. Problematically, for most social networks, the space of possible relationships is too vast to observe and memorize. Indeed, people's knowledge of these social relations is well known to be biased and error-prone. Here, we reveal that these biased representations reflect a fundamental computation that abstracts over individual relationships to enable principled inferences about unseen relationships. We propose a theory of network representation that explains how people learn inferential cognitive maps of social relations from direct observation, what kinds of knowledge structures emerge as a consequence, and why it can be beneficial to encode systematic biases into social cognitive maps. Leveraging simulations, laboratory experiments, and "field data" from a real-world network, we find that people abstract observations of direct relations (e.g., friends) into inferences of multistep relations (e.g., friends-of-friends). This multistep abstraction mechanism enables people to discover and represent complex social network structure, affording adaptive inferences across a variety of contexts, including friendship, trust, and advice-giving. Moreover, this multistep abstraction mechanism unifies a variety of otherwise puzzling empirical observations about social behavior. Our proposal generalizes the theory of cognitive maps to the fundamental computational problem of social inference, presenting a powerful framework for understanding the workings of a predictive mind operating within a complex social world.

Hippocampal place code plasticity in CA1 requires postsynaptic membrane fusion

Rapid delivery of glutamate receptors to the postsynaptic membrane via vesicle fusion is a central component of synaptic plasticity. However, it is unknown how this process supports specific neural computations during behavior. To bridge this gap, we combined conditional genetic deletion of a component of the postsynaptic membrane fusion machinery, Syntaxin3 (Stx3), in hippocampal CA1 neurons of mice with population in vivo calcium imaging. This approach revealed that Stx3 is necessary for forming the neural dynamics that support novelty processing, spatial reward memory and offline memory consolidation. In contrast, CA1 Stx3 was dispensable for maintaining aspects of the neural code that exist presynaptic to CA1 such as representations of context and space. Thus, manipulating postsynaptic membrane fusion identified computations that specifically require synaptic restructuring via membrane trafficking in CA1 and distinguished them from neural representation that could be inherited from upstream brain regions or learned through other mechanisms.

Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.

Rapid learning of predictive maps with STDP and theta phase precession

The predictive map hypothesis is a promising candidate principle for hippocampal function. A favoured formalisation of this hypothesis, called the successor representation, proposes that each place cell encodes the expected state occupancy of its target location in the near future. This predictive framework is supported by behavioural as well as electrophysiological evidence and has desirable consequences for both the generalisability and efficiency of reinforcement learning algorithms. However, it is unclear how the successor representation might be learnt in the brain. Error-driven temporal difference learning, commonly used to learn successor representations in artificial agents, is not known to be implemented in hippocampal networks. Instead, we demonstrate that spike-timing dependent plasticity (STDP), a form of Hebbian learning, acting on temporally compressed trajectories known as 'theta sweeps', is sufficient to rapidly learn a close approximation to the successor representation. The model is biologically plausible - it uses spiking neurons modulated by theta-band oscillations, diffuse and overlapping place cell-like state representations, and experimentally matched parameters. We show how this model maps onto known aspects of hippocampal circuitry and explains substantial variance in the temporal difference successor matrix, consequently giving rise to place cells that demonstrate experimentally observed successor representation-related phenomena including backwards expansion on a 1D track and elongation near walls in 2D. Finally, our model provides insight into the observed topographical ordering of place field sizes along the dorsal-ventral axis by showing this is necessary to prevent the detrimental mixing of larger place fields, which encode longer timescale successor representations, with more fine-grained predictions of spatial location.

Neural learning rules for generating flexible predictions and computing the successor representation

The predictive nature of the hippocampus is thought to be useful for memory-guided cognitive behaviors. Inspired by the reinforcement learning literature, this notion has been formalized as a predictive map called the successor representation (SR). The SR captures a number of observations about hippocampal activity. However, the algorithm does not provide a neural mechanism for how such representations arise. Here, we show the dynamics of a recurrent neural network naturally calculate the SR when the synaptic weights match the transition probability matrix. Interestingly, the predictive horizon can be flexibly modulated simply by changing the network gain. We derive simple, biologically plausible learning rules to learn the SR in a recurrent network. We test our model with realistic inputs and match hippocampal data recorded during random foraging. Taken together, our results suggest that the SR is more accessible in neural circuits than previously thought and can support a broad range of cognitive functions.