Mice learn multi-step routes by memorizing subgoal locations

Astrocytes in the External Globus Pallidus Selectively Represent Routine Formation During Repeated Reward-Seeking in Mice

The external globus pallidus (GPe) is a central part of the basal ganglia indirect pathway implicated in movement and decision-making. As a hub connecting the dorsal striatum and subthalamic nucleus (STN), the GPe guides repetitive and routine behaviors. However, it remains unknown how diverse GPe cells engage in routine formation while learning action sequences in repetitive reward-seeking conditioning. Here, in male mice, we investigated the Ca2+ dynamics of two GPe cell types, astrocytes and parvalbumin-expressing neurons, during routine formation. Our findings show that the dynamics of GPe astrocytes may be involved in action sequence refinement, a characteristic potentially contributing to more efficient reward-seeking behavior.

Shortcutting from self-motion signals reveals a cognitive map in mice

Animals navigate by learning the spatial layout of their environment. We investigated spatial learning of mice in an open maze where food was hidden in one of a hundred holes. Mice leaving from a stable entrance learned to efficiently navigate to the food without the need for landmarks. We developed a quantitative framework to reveal how the mice estimate the food location based on analyses of trajectories and active hole checks. After learning, the computed 'target estimation vector' (TEV) closely approximated the mice's route and its hole check distribution. The TEV required learning both the direction and distance of the start to food vector, and our data suggests that different learning dynamics underlie these estimates. We propose that the TEV can be precisely connected to the properties of hippocampal place cells. Finally, we provide the first demonstration that, after learning the location of two food sites, the mice took a shortcut between the sites, demonstrating that they had generated a cognitive map.

Vector-Based Navigation Inspired by Directional Place Cells

We introduce a navigation algorithm inspired by directional sensitivity observed in CA1 place cells of the rat hippocampus. These cells exhibit directional polarization characterized by vector fields converging to specific locations in the environment, known as ConSinks [8]. By sampling from a population of such cells at varying orientations, an optimal vector of travel towards a goal can be determined. Our proposed algorithm aims to emulate this mechanism for learning goal-directed navigation tasks. We employ a novel learning rule that integrates environmental reward signals with an eligibility trace to determine the update eligibility of a cell's directional sensitivity. Compared to state-of-the-art Reinforcement Learning algorithms, our approach demonstrates superior performance and speed in learning to navigate towards goals in obstacle-filled environments. Additionally, we observe analogous behavior in our algorithm to experimental evidence, where the mean ConSink location dynamically shifts toward a new goal shortly after it is introduced.

Tonically active GABAergic neurons in the dorsal periaqueductal gray control instinctive escape in mice

Escape behavior is a set of locomotor actions that move an animal away from threat. While these actions can be stereotyped, it is advantageous for survival that they are flexible.1,2,3 For example, escape probability depends on predation risk and competing motivations,4,5,6,7,8,9,10,11 and flight to safety requires continuous adjustments of trajectory and must terminate at the appropriate place and time.12,13,14,15,16 This degree of flexibility suggests that modulatory components, like inhibitory networks, act on the neural circuits controlling instinctive escape.17,18,19,20,21,22 In mice, the decision to escape from imminent threats is implemented by a feedforward circuit in the midbrain, where excitatory vesicular glutamate transporter 2-positive (VGluT2+) neurons in the dorsal periaqueductal gray (dPAG) compute escape initiation and escape vigor.23,24,25 Here we tested the hypothesis that local GABAergic neurons within the dPAG control escape behavior by setting the excitability of the dPAG escape network. Using in vitro patch-clamp and in vivo neural activity recordings, we found that vesicular GABA transporter-positive (VGAT+) dPAG neurons fire action potentials tonically in the absence of synaptic inputs and are a major source of inhibition to VGluT2+ dPAG neurons. Activity in VGAT+ dPAG cells transiently decreases at escape onset and increases during escape, peaking at escape termination. Optogenetically increasing or decreasing VGAT+ dPAG activity changes the probability of escape when the stimulation is delivered at threat onset and the duration of escape when delivered after escape initiation. We conclude that the activity of tonically firing VGAT+ dPAG neurons sets a threshold for escape initiation and controls the execution of the flight action.

A vast space of compact strategies for effective decisions

Inference-based decision-making, which underlies a broad range of behavioral tasks, is typically studied using a small number of handcrafted models. We instead enumerate a complete ensemble of strategies that could be used to effectively, but not necessarily optimally, solve a dynamic foraging task. Each strategy is expressed as a behavioral "program" that uses a limited number of internal states to specify actions conditioned on past observations. We show that the ensemble of strategies is enormous-comprising a quarter million programs with up to five internal states-but can nevertheless be understood in terms of algorithmic "mutations" that alter the structure of individual programs. We devise embedding algorithms that reveal how mutations away from a Bayesian-like strategy can diversify behavior while preserving performance, and we construct a compositional description to link low-dimensional changes in algorithmic structure with high-dimensional changes in behavior. Together, this work provides an alternative approach for understanding individual variability in behavior across animals and tasks.

Stochastic characterization of navigation strategies in an automated variant of the Barnes maze

Animals can use a repertoire of strategies to navigate in an environment, and it remains an intriguing question how these strategies are selected based on the nature and familiarity of environments. To investigate this question, we developed a fully automated variant of the Barnes maze, characterized by 24 vestibules distributed along the periphery of a circular arena, and monitored the trajectories of mice over 15 days as they learned to navigate towards a goal vestibule from a random start vestibule. We show that the patterns of vestibule visits can be reproduced by the combination of three stochastic processes reminiscent of random, serial, and spatial strategies. The processes randomly selected vestibules based on either uniform (random) or biased (serial and spatial) probability distributions. They closely matched experimental data across a range of statistical distributions characterizing the length, distribution, step size, direction, and stereotypy of vestibule sequences, revealing a shift from random to spatial and serial strategies over time, with a strategy switch occurring approximately every six vestibule visits. Our study provides a novel apparatus and analysis toolset for tracking the repertoire of navigation strategies and demonstrates that a set of stochastic processes can largely account for exploration patterns in the Barnes maze.

Age dependent path integration deficit in 5xFAD mice

Alzheimer's disease (AD) is a severe neurodegenerative disorder and the most common form of dementia in elderly individuals, characterized by memory deficits, cognitive decline, and neuropathology. The identification of preclinical markers for AD remains elusive. We employed an ultrasound-evoked spatial memory assay to investigate path integration (PI) in wild type C57BL/6 J and 5xFAD mice. We observed significant recruitment of the mammillary bodies (MB) and subiculum (Sub) - core regions of the Papez circuit during PI, as indicated by increased expression of the immediate early gene c-Fos in C57BL/6 J mice. In 5xFAD mice, amyloid-beta (Aβ) vulnerability in the MB and Sub was evident at 3-months of age, preceding widespread pathology at 5-months of age. In parallel, we detected significant behavioral deficits in PI in the 5XFAD mice at 5- but not 3-months of age. Sex based analysis revealed a more profound deficit in males compared to females at 5-months of age. Our data suggest PI may be as an early indicator of AD, potentially associated with dysfunction within the Papez circuit.

Delayed escape behavior requires claustral activity

Animals are known to exhibit innate and learned forms of defensive behaviors, but it is unclear whether animals can escape through methods other than these forms. In this study, we develop the delayed escape task, in which male rats temporarily hold the information required for future escape, and we demonstrate that this task, in which the subject extrapolates from past experience without direct experience of its behavioral outcome, does not fall into either of the two forms of behavior. During the holding period, a subset of neurons in the rostral-to-striatum claustrum (rsCla), only when pooled together, sustain enhanced population activity without ongoing sensory stimuli. Transient inhibition of rsCla neurons during the initial part of the holding period produces prolonged inhibition of the enhanced activity. The transient inhibition also attenuates the delayed escape behavior. Our data suggest that the rsCla activity bridges escape-inducing stimuli to the delayed onset of escape.

The successor representation subserves hierarchical abstraction for goal-directed behavior

Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.

Interactions between rodent visual and spatial systems during navigation

Many behaviours that are critical for animals to survive and thrive rely on spatial navigation. Spatial navigation, in turn, relies on internal representations about one's spatial location, one's orientation or heading direction and the distance to objects in the environment. Although the importance of vision in guiding such internal representations has long been recognized, emerging evidence suggests that spatial signals can also modulate neural responses in the central visual pathway. Here, we review the bidirectional influences between visual and navigational signals in the rodent brain. Specifically, we discuss reciprocal interactions between vision and the internal representations of spatial position, explore the effects of vision on representations of an animal's heading direction and vice versa, and examine how the visual and navigational systems work together to assess the relative distances of objects and other features. Throughout, we consider how technological advances and novel ethological paradigms that probe rodent visuo-spatial behaviours allow us to advance our understanding of how brain areas of the central visual pathway and the spatial systems interact and enable complex behaviours.

Mice identify subgoal locations through an action-driven mapping process

Mammals form mental maps of the environments by exploring their surroundings. Here, we investigate which elements of exploration are important for this process. We studied mouse escape behavior, in which mice are known to memorize subgoal locations-obstacle edges-to execute efficient escape routes to shelter. To test the role of exploratory actions, we developed closed-loop neural-stimulation protocols for interrupting various actions while mice explored. We found that blocking running movements directed at obstacle edges prevented subgoal learning; however, blocking several control movements had no effect. Reinforcement learning simulations and analysis of spatial data show that artificial agents can match these results if they have a region-level spatial representation and explore with object-directed movements. We conclude that mice employ an action-driven process for integrating subgoals into a hierarchical cognitive map. These findings broaden our understanding of the cognitive toolkit that mammals use to acquire spatial knowledge.

Computational cross-species views of the hippocampal formation

The discovery of place cells and head direction cells in the hippocampal formation of freely foraging rodents has led to an emphasis of its role in encoding allocentric spatial relationships. In contrast, studies in head-fixed primates have additionally found representations of spatial views. We review recent experiments in freely moving monkeys that expand upon these findings and show that postural variables such as eye/head movements strongly influence neural activity in the hippocampal formation, suggesting that the function of the hippocampus depends on where the animal looks. We interpret these results in the light of recent studies in humans performing challenging navigation tasks which suggest that depending on the context, eye/head movements serve one of two roles-gathering information about the structure of the environment (active sensing) or externalizing the contents of internal beliefs/deliberation (embodied cognition). These findings prompt future experimental investigations into the information carried by signals flowing between the hippocampal formation and the brain regions controlling postural variables, and constitute a basis for updating computational theories of the hippocampal system to accommodate the influence of eye/head movements.

Understanding the encoding of object locations in small-scale spaces during free exploration using eye tracking

Navigation is instrumental to daily life and is often used to encode and locate objects, such as keys in one's house. Yet, little is known about how navigation works in more ecologically valid situations such as finding objects within a room. Specifically, it is not clear how vision vs. body movements contribute differentially to spatial memory in such small-scale spaces. In the current study, participants encoded object locations by viewing them while standing (stationary condition) or by additionally being guided by the experimenter while blindfolded (walking condition) after viewing the objects. They then retrieved the objects from the same or different viewpoint, creating a 2 × 2 within subject design. We simultaneously recorded participant eye movements throughout the experiment using mobile eye tracking. The results showed no statistically significant differences among our four conditions (stationary, same viewpoint as encoding; stationary, different viewpoint; walking, same viewpoint; walking, different viewpoint), suggesting that in a small real-world space, vision may be sufficient to remember object locations. Eye tracking analyses revealed that object locations were better remembered next to landmarks and that participants encoded items on one wall together, suggesting the use of local wall coordinates rather than global room coordinates. A multivariate regression analysis revealed that the only significant predictor of object placement accuracy was average looking time. These results suggest that vision may be sufficient for encoding object locations in a small-scale environment and that such memories may be formed largely locally rather than globally.

Adapting hippocampus multi-scale place field distributions in cluttered environments optimizes spatial navigation and learning

Extensive studies in rodents show that place cells in the hippocampus have firing patterns that are highly correlated with the animal's location in the environment and are organized in layers of increasing field sizes or scales along its dorsoventral axis. In this study, we use a spatial cognition model to show that different field sizes could be exploited to adapt the place cell representation to different environments according to their size and complexity. Specifically, we provide an in-depth analysis of how to distribute place cell fields according to the obstacles in cluttered environments to optimize learning time and path optimality during goal-oriented spatial navigation tasks. The analysis uses a reinforcement learning (RL) model that assumes that place cells allow encoding the state. While previous studies have suggested exploiting different field sizes to represent areas requiring different spatial resolutions, our work analyzes specific distributions that adapt the representation to the environment, activating larger fields in open areas and smaller fields near goals and subgoals (e.g., obstacle corners). In addition to assessing how the multi-scale representation may be exploited in spatial navigation tasks, our analysis and results suggest place cell representations that can impact the robotics field by reducing the total number of cells for path planning without compromising the quality of the paths learned.

Predictive maps in rats and humans for spatial navigation

Much of our understanding of navigation comes from the study of individual species, often with specific tasks tailored to those species. Here, we provide a novel experimental and analytic framework integrating across humans, rats, and simulated reinforcement learning (RL) agents to interrogate the dynamics of behavior during spatial navigation. We developed a novel open-field navigation task ("Tartarus maze") requiring dynamic adaptation (shortcuts and detours) to frequently changing obstructions on the path to a hidden goal. Humans and rats were remarkably similar in their trajectories. Both species showed the greatest similarity to RL agents utilizing a "successor representation," which creates a predictive map. Humans also displayed trajectory features similar to model-based RL agents, which implemented an optimal tree-search planning procedure. Our results help refine models seeking to explain mammalian navigation in dynamic environments and highlight the utility of modeling the behavior of different species to uncover the shared mechanisms that support behavior.

Protocol to Study Spatial Subgoal Learning Using Escape Behavior in Mice

Rodent spatial navigation is a key model system for studying mammalian cognition and its neural mechanisms. Of particular interest is how animals memorize the structure of their environments and compute multi-step routes to a goal. Previous work on multi-step spatial reasoning has generally involved placing rodents at the start of a maze until they learn to navigate to a reward without making wrong turns. It thus remains poorly understood how animals rapidly learn about the structure of naturalistic open environments with goals and obstacles. Here we present an assay in which mice spontaneously memorize two-step routes in an environment with a shelter and an obstacle. We allow the mice to explore this environment for 20 min, and then we remove the obstacle. We then present auditory threat stimuli, causing the mouse to escape to the shelter. Finally, we record each escape route and measure whether it targets the shelter directly (a 'homing-vector' escape) or instead targets the location where the obstacle edge was formerly located (an 'edge-vector' escape). Since the obstacle is no longer there, these obstacle-edge-directed escape routes provide evidence that the mouse has memorized a subgoal location, i.e., a waypoint targeted in order to efficiently get to the shelter in the presence of an obstacle. By taking advantage of instinctive escape responses, this assay probes a multi-step spatial memory that is learned in a single session without pretraining. The subgoal learning phenomenon it generates can be useful not only for researchers working on navigation and instinctive behavior, but also for neuroscientists studying the neural basis of multi-step spatial reasoning.

Innate heuristics and fast learning support escape route selection in mice

When faced with imminent danger, animals must rapidly take defensive actions to reach safety. Mice can react to threatening stimuli in ∼250 milliseconds¹ and, in simple environments, use spatial memory to quickly escape to shelter.^2,3 Natural habitats, however, often offer multiple routes to safety that animals must identify and choose from.⁴ This is challenging because although rodents can learn to navigate complex mazes,^5,6 learning the value of different routes through trial and error during escape could be deadly. Here, we investigated how mice learn to choose between different escape routes. Using environments with paths to shelter of varying length and geometry, we find that mice prefer options that minimize path distance and angle relative to the shelter. This strategy is already present during the first threat encounter and after only ∼10 minutes of exploration in a novel environment, indicating that route selection does not require experience of escaping. Instead, an innate heuristic assigns survival value to each path after rapidly learning the spatial environment. This route selection process is flexible and allows quick adaptation to arenas with dynamic geometries. Computational modeling shows that model-based reinforcement learning agents replicate the observed behavior in environments where the shelter location is rewarding during exploration. These results show that mice combine fast spatial learning with innate heuristics to choose escape routes with the highest survival value. The results further suggest that integrating prior knowledge acquired through evolution with knowledge learned from experience supports adaptation to changing environments and minimizes the need for trial and error when the errors are costly.

•

Mice learn to escape via the fastest route after ∼10 minutes in a new environment

•

Escape routes are learned during exploration and do not require threat exposure

•

Mice prefer escape routes that minimize path distance and angle to shelter

•

Fast route learning can be replicated by model-based reinforcement learning agents

Extending neural systems for navigation to hunting behavior

For decades, a central question in neuroscience has been: How does the brain support navigation? Recent research on navigation has explored how brain regions support the capacity to adapt to changes in the environment and track the distance and direction to goal locations. Here, we provide a brief review of this literature and speculate how these neural systems may be involved in another, parallel behavior-hunting. Hunting shares many of the same challenges as navigation. Like navigation, hunting requires the hunter to orient towards a goal while minimizing their distance from it while traveling. Likewise, hunting may require the accommodation of detours to locate prey or the exploitation of shortcuts for a quicker capture. Recent research suggests that neurons in the periaqueductal gray, hypothalamus, and dorsal anterior cingulate play key roles in such hunting behavior. In this review, we speculate on how these regions may operate functionally with other key brain regions involved in navigation, such as the hippocampus, to support hunting. Additionally, we posit that hunting in a group presents an additional set of challenges, where success relies on multicentric tracking and prediction of prey position as well as the position of co-hunters.