Dorsal hippocampus contributes to model-based planning

Female rats retain goal-directed planning of action sequences after acute stress despite changes in planning structure and action sequence execution

When making decisions under stress, organisms tend to deliberate less and rely on automatic habits. Prior investigation into the influence of stress on decision-making has primarily viewed goal-direction and habit as independent and competitive sources of control in static environments. The effects of acute stress on the integration of goal-direction and habit in hierarchical planning to solve dynamic tasks remain unclear. Here, our aim was to assess whether stress prompted the usage of habitual action sequences over the selection of discrete goal-directed actions in a serial decision task. We trained 16 female Long Evans rats in a two-stage binary choice task and performed two probe tests, one following acute restraint stress and one under control conditions, to identify how stress affected higher-level planning of behavior and intermediate action structures. We found that under both stressed and control conditions, rats exhibited goal-directed planning of habitual action sequences. However, following stress, rats showed a greater tendency to reiterate action sequences independent of reinforcement, indicating that stress may induce an aversion to exploration in action planning. Stress also increased the latency between responses - degrading action sequence integrity despite conserving their overall structure and performance. Taken together, these findings suggest that although acute stress does not disrupt the overall macrostructure of behavior in two-stage decision-making, it does alter the microstructure of goal-directed and habitual control individually. Further, these results imply that the extent to which stress impairs goal-direction in female rats may depend on the incentive structure and attentional demands of the decision environment.

Hippocampal output suppresses orbitofrontal cortex schema cell formation

Both the orbitofrontal cortex (OFC) and the hippocampus (HC) are implicated in the formation of cognitive maps and their generalization into schemas. However, how these areas interact in supporting this function remains unclear, with some proposals supporting a serial model in which the OFC draws on task representations created by the HC to extract key behavioral features and others suggesting a parallel model in which both regions construct representations that highlight different types of information. In the present study, we tested between these two models by asking how schema correlates in rat OFC would be affected by inactivating the output of the HC, after learning and during transfer across problems. We found that the prevalence and content of schema correlates were unaffected by inactivating one major HC output area, the ventral subiculum, after learning, whereas inactivation during transfer accelerated their formation. These results favor the proposal that the OFC and HC operate in parallel to extract different features defining cognitive maps and schemas.

Neural correlates of visual object recognition in rats

Invariant object recognition-the ability to recognize objects across size, rotation, or context-is fundamental for making sense of a dynamic visual world. Though traditionally studied in primates, emerging evidence suggests rodents recognize objects across a range of identity-preserving transformations. We demonstrate that rats robustly perform visual object recognition and explore a neural pathway that may underlie this capacity by developing a pipeline from high-throughput behavior training to cellular resolution imaging in awake, head-fixed animals. Leveraging our optical approach, we systematically profile neurons in primary and higher-order visual areas and their spatial organization. We find that rat visual cortex exhibits several features similar to those observed in the primate ventral stream but also marked deviations, suggesting species-specific differences in how brains solve visual object recognition. This work reinforces the sophisticated visual abilities of rats and offers the technical foundation to use them as a powerful model for mechanistic perception.

Adaptive planning depth in human problem-solving

We humans are capable of solving challenging planning problems, but the range of adaptive strategies that we use to address them is not yet fully characterized. Here, we designed a series of problem-solving tasks that require planning at different depths. After systematically comparing the performance of participants and planning models, we found that when facing problems that require planning to a certain number of subgoals (from 1 to 8), participants make an adaptive use of their cognitive resources-namely, they tend to select an initial plan having the minimum required depth, rather than selecting the same depth for all problems. These results support the view of problem-solving as a bounded rational process, which adapts costly cognitive resources to task demands.

Disentangling the Component Processes in Complex Planning Impairments Following Ventromedial Prefrontal Lesions

Damage to the ventromedial prefrontal cortex (vmPFC) in humans disrupts planning abilities in naturalistic settings. However, it is unknown which components of planning are affected in these patients, including selecting the relevant information, simulating future states, or evaluating between these states. To address this question, we leveraged computational paradigms to investigate the role of vmPFC in planning, using the board game task "Four-in-a-Row" (18 lesion patients, 9 female; 30 healthy control participants, 16 female) and the simpler "Two-Step" task measuring model-based reasoning (49 lesion patients, 27 female; 20 healthy control participants, 13 female). Damage to vmPFC disrupted performance in Four-in-a-Row compared with both control lesion patients and healthy age-matched controls. We leveraged a computational framework to assess different component processes of planning in Four-in-a-Row and found that impairments following vmPFC damage included shallower planning depth and a tendency to overlook game-relevant features. In the "Two-Step" task, which involves binary choices across a short future horizon, we found little evidence of planning in all groups and no behavioral differences between groups. Complex yet computationally tractable tasks such as "Four-in-a-Row" offer novel opportunities for characterizing neuropsychological planning impairments, which in vmPFC patients we find are associated with oversights and reduced planning depth.

A sequence bottleneck for animal intelligence and language?

We discuss recent findings suggesting that non-human animals lack memory for stimulus sequences, and therefore do not represent the order of stimuli faithfully. These observations have far-reaching consequences for animal cognition, neuroscience, and studies of the evolution of language and culture. This is because, if non-human animals do not remember or process information about order faithfully, then it is unlikely that non-human animals perform mental simulations, construct mental world models, have episodic memory, or transmit culture faithfully. If this suggested sequence bottleneck proves to be a prevalent characteristic of animal memory systems, as suggested by recent work, it would require a re-examination of some influential concepts and ideas.

Striatal arbitration between choice strategies guides few-shot adaptation

Animals often exhibit rapid action changes in context-switching environments. This study hypothesized that, compared to the expected outcome, an unexpected outcome leads to distinctly different action-selection strategies to guide rapid adaptation. We designed behavioral measures differentiating between trial-by-trial dynamics after expected and unexpected events. In various reversal learning data with different rodent species and task complexities, conventional learning models failed to replicate the choice behavior following an unexpected outcome. This discrepancy was resolved by the proposed model with two different decision variables contingent on outcome expectation: the support-stay and conflict-shift bias. Electrophysiological data analyses revealed that striatal neurons encode our model's key variables. Furthermore, the inactivation of striatal direct and indirect pathways neutralizes the effect of past expected and unexpected outcomes, respectively, on the action-selection strategy following an unexpected outcome. Our study suggests unique roles of the striatum in arbitrating between different action selection strategies for few-shot adaptation.

The value of hippocampal sub-region imaging features for the diagnosis and severity grading of ASD in children

BACKGROUND

Hippocampal structural changes in Autism Spectrum Disorder (ASD) are inconsistent. This study investigates hippocampal subregion changes in ASD patients to reveal intrinsic hippocampal anomalies.

METHODS

A retrospective study from Hainan Children's Hospital database (2020-2023) included ASD patients and matched controls. We classified ASD participants based on severity, dividing all subjects into four groups: normal, mild, moderate, and severe. High-resolution T1-weighted MRI images were analyzed for hippocampal subregion segmentation and volume calculations using Freesurfer. Texture features were extracted via the Gray-Level Co-occurrence Matrix. The Receiver Operating Characteristic curve was used to evaluate seven random forest predictive models constructed from volume, subregion, and texture features, as well as their combinations following feature selection.

RESULTS

The study included 114 ASD patients (98 boys, 2-8 years; 16 girls, 2-6 years; 17 mild, 57 moderate, 40 severe) and 111 healthy controls (HCs). No significant differences in volumes were found between ASD patients and HCs (adjusted P-value >0.05). The seven random forest models showed that single volume and texture features performed poorly for ASD classification; however, integrating various feature types improved AUC values. Further selection of texture, subregion, and volume features enhanced AUC performance across normal and varying severity categories, demonstrating the potential value of specific subregions and integrated features in ASD diagnosis.

CONCLUSION

Random forest models revealed that hippocampal volume, texture features, and subregion characteristics are crucial for diagnosing and assessing the severity of ASD. Integrating selected texture and subregion features optimized diagnostic efficacy, while combining texture, subregion, and volume features further improved severity grading effectiveness.

Maturation of striatal dopamine supports the development of habitual behavior through adolescence

Developmental trajectories during the transition from adolescence to adulthood contribute to the establishment of stable, adult forms of operation. Understanding the neural mechanisms underlying this transition is crucial for identifying variability in normal development and the onset of psychiatric disorders, which typically emerge during this time. Habitual behaviors can serve as a model for understanding brain mechanisms underlying the stabilization of adult behavior, while also conferring risk for psychopathologies. Dopaminergic (DA) processes in the basal ganglia are thought to facilitate the formation of habits; however, developmental trajectories of habits and the brain systems supporting them have not been characterized in vivo in developing humans. The current study examined trajectories of habitual behavior from adolescence to adulthood and sought to understand how the maturing striatal DA system may act as a potential mechanism in the process of habit formation. We used data from two longitudinal studies (combined n = 217, 10 - 32 years of age, 1-3 visits each, 320 total sessions) to characterize normative developmental trajectories of basal ganglia tissue iron concentration (a proxy for DA-related neurophysiology) and goal-direct and habitual control behaviors in a two-stage decision-making task. Tissue iron concentrations across the basal ganglia and habitual responding during the two-stage sequential decision-making task both increased with age (all p < 0.001). Importantly, habitual responding was associated with tissue iron concentrations in the putamen (F = 4.34, p = 0.014), such that increases in habitual responding were supported by increases in putamen tissue iron concentration during childhood through late adolescence. Exploratory analyses of further subdivisions of anatomical regions found that this association was specific to the posterior putamen. These results provide novel evidence in humans that habitual behavior continues to mature into adulthood and may be supported by increased specialization of reward systems.

Unifying Principles of Generalization: Past, Present, and Future

Generalization, defined as applying limited experiences to novel situations, represents a cornerstone of human intelligence. Our review traces the evolution and continuity of psychological theories of generalization, from its origins in concept learning (categorizing stimuli) and function learning (learning continuous input-output relationships) to domains such as reinforcement learning and latent structure learning. Historically, there have been fierce debates between approaches based on rule-based mechanisms, which rely on explicit hypotheses about environmental structure, and approaches based on similarity-based mechanisms, which leverage comparisons to prior instances. Each approach has unique advantages: Rules support rapid knowledge transfer, while similarity is computationally simple and flexible. Today, these debates have culminated in the development of hybrid models grounded in Bayesian principles, effectively marrying the precision of rules with the flexibility of similarity. The ongoing success of hybrid models not only bridges past dichotomies but also underscores the importance of integrating both rules and similarity for a comprehensive understanding of human generalization.

Environmental complexity modulates information processing and the balance between decision-making systems

Behavior in naturalistic scenarios occurs in diverse environments. Adaptive strategies rely on multiple neural circuits and competing decision systems. However, past studies of rodent decision making have largely measured behavior in simple environments. To fill this gap, we recorded neural ensembles from hippocampus (HC), dorsolateral striatum (DLS), and dorsomedial prefrontal cortex (dmPFC) while rats foraged for food under changing rules in environments with varying topological complexity. Environmental complexity increased behavioral variability, lengthened HC nonlocal sequences, and modulated action caching. We found contrasting representations between DLS and HC, supporting a competition between decision systems. dmPFC activity was indicative of setting this balance, in particular predicting the extent of HC non-local coding. Inactivating mPFC impaired short-term behavioral adaptation and produced long-term deficits in balancing decision systems. Our findings reveal the dynamic nature of decision-making systems and how environmental complexity modulates their engagement with implications for behavior in naturalistic environments.

Human hippocampus and dorsomedial prefrontal cortex infer and update latent causes during social interaction

Latent-cause inference is the process of identifying features of the environment that have caused an outcome. This problem is especially important in social settings where individuals may not make equal contributions to the outcomes they achieve together. Here, we designed a novel task in which participants inferred which of two characters was more likely to have been responsible for outcomes achieved by working together. Using computational modeling, univariate and multivariate analysis of human fMRI, and continuous theta-burst stimulation, we identified two brain regions that solved the task. Notably, as each outcome occurred, it was possible to decode the inference of its cause (the responsible character) from hippocampal activity. Activity in dorsomedial prefrontal cortex (dmPFC) updated estimates of association between cause-responsible character-and the outcome. Disruption of dmPFC activity impaired participants' ability to update their estimate as a function of inferred responsibility but spared their ability to infer responsibility.

Global neural encoding of behavioral strategies in mice during perceptual decision-making task with two different sensory patterns

When a simple model-free strategy does not provide sufficient outcomes, an inference-based strategy estimating a hidden task structure becomes essential for optimizing choices. However, the neural circuitry involved in inference-based strategies is still unclear. We developed a tone frequency discrimination task in head-fixed mice in which the tone category of the current trial depended on the category of the previous trial. When the tone category was repeated, the mice continued using the default model-free strategy, as well as when the tone was randomly presented, to bias choices. In contrast, when the tone was alternated, the default strategy gradually shifted to a hybrid of model-free and inference-based strategies, although we did not observe distinct strategy changes. Brain-wide electrophysiological recording suggested that the neural activity of the frontal and sensory cortices, hippocampus, and striatum was correlated with the reward expectation in different task conditions, suggesting the global encoding of multiple strategies in the brain.

Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning

Different brain systems have been hypothesized to subserve multiple "experts" that compete to generate behavior. In reinforcement learning, two general processes, one model-free (MF) and one model-based (MB), are often modeled as a mixture of agents (MoA) and hypothesized to capture differences between automaticity vs. deliberation. However, shifts in strategy cannot be captured by a static MoA. To investigate such dynamics, we present the mixture-of-agents hidden Markov model (MoA-HMM), which simultaneously learns inferred action values from a set of agents and the temporal dynamics of underlying "hidden" states that capture shifts in agent contributions over time. Applying this model to a multi-step, reward-guided task in rats reveals a progression of within-session strategies: a shift from initial MB exploration to MB exploitation, and finally to reduced engagement. The inferred states predict changes in both response time and OFC neural encoding during the task, suggesting that these states are capturing real shifts in dynamics.

Understanding dual process cognition via the minimum description length principle

Dual-process theories play a central role in both psychology and neuroscience, figuring prominently in domains ranging from executive control to reward-based learning to judgment and decision making. In each of these domains, two mechanisms appear to operate concurrently, one relatively high in computational complexity, the other relatively simple. Why is neural information processing organized in this way? We propose an answer to this question based on the notion of compression. The key insight is that dual-process structure can enhance adaptive behavior by allowing an agent to minimize the description length of its own behavior. We apply a single model based on this observation to findings from research on executive control, reward-based learning, and judgment and decision making, showing that seemingly diverse dual-process phenomena can be understood as domain-specific consequences of a single underlying set of computational principles.

Hippocampal representations of alternative possibilities are flexibly generated to meet cognitive demands

The cognitive ability to go beyond the present to consider alternative possibilities, including potential futures and counterfactual pasts, can support adaptive decision making. Complex and changing real-world environments, however, have many possible alternatives. Whether and how the brain can select among them to represent alternatives that meet current cognitive needs remains unknown. We therefore examined neural representations of alternative spatial locations in the rat hippocampus during navigation in a complex patch foraging environment with changing reward probabilities. We found representations of multiple alternatives along paths ahead and behind the animal, including in distant alternative patches. Critically, these representations were modulated in distinct patterns across successive trials: alternative paths were represented proportionate to their evolving relative value and predicted subsequent decisions, whereas distant alternatives were prevalent during value updating. These results demonstrate that the brain modulates the generation of alternative possibilities in patterns that meet changing cognitive needs for adaptive behavior.

Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning

Dopamine release in the nucleus accumbens has been hypothesized to signal reward prediction error, the difference between observed and predicted reward, suggesting a biological implementation for reinforcement learning. Rigorous tests of this hypothesis require assumptions about how the brain maps sensory signals to reward predictions, yet this mapping is still poorly understood. In particular, the mapping is non-trivial when sensory signals provide ambiguous information about the hidden state of the environment. Previous work using classical conditioning tasks has suggested that reward predictions are generated conditional on probabilistic beliefs about the hidden state, such that dopamine implicitly reflects these beliefs. Here we test this hypothesis in the context of an instrumental task (a two-armed bandit), where the hidden state switches repeatedly. We measured choice behavior and recorded dLight signals reflecting dopamine release in the nucleus accumbens core. Model comparison among a wide set of cognitive models based on the behavioral data favored models that used Bayesian updating of probabilistic beliefs. These same models also quantitatively matched the dopamine measurements better than non-Bayesian alternatives. We conclude that probabilistic belief computation contributes to instrumental task performance in mice and is reflected in mesolimbic dopamine signaling.

Model-free decision-making underlies motor errors in rapid sequential movements under threat

Our movements, especially sequential ones, are usually goal-directed, i.e., coupled with task-level goals. Consequently, cognitive strategies for decision-making and motor performance are likely to influence each other. However, evidence linking decision-making strategies and motor performance remains elusive. Here, we designed a modified version of the two-step task, named the two-step sequential movement task, where participants had to conduct rapid sequential finger movements to obtain rewards (n = 40). In the shock session, participants received an electrical shock if they made an erroneous or slow movement, while in the no-shock session, they only received zero reward. We found that participants who prioritised model-free decision-making committed more motor errors in the presence of the shock stimulus (shock sessions) than those who prioritised model-based decision-making. Using a mediation analysis, we also revealed a strong link between the balance of the model-based and the model-free learning strategies and sequential movement performances. These results suggested that model-free decision-making produces more motor errors than model-based decision-making in rapid sequential movements under the threat of stressful stimuli.

The Role of the Medial Prefrontal Cortex in Spatial Margin of Safety Calculations

Naturalistic observations show that animals pre-empt danger by moving to locations that increase their success in avoiding future threats. To test this in humans, we created a spatial margin of safety (MOS) decision task that quantifies pre-emptive avoidance by measuring the distance subjects place themselves to safety when facing different threats whose attack locations vary in predictability. Behavioral results show that human participants place themselves closer to safe locations when facing threats that attack in spatial locations with more outliers. Using both univariate and multivariate pattern analysis (MVPA) on fMRI data collected during a 2 h session on participants of both sexes, we demonstrate a dissociable role for the vmPFC in MOS-related decision-making. MVPA results revealed that the posterior vmPFC encoded for more unpredictable threats with univariate analyses showing a functional coupling with the amygdala and hippocampus. Conversely, the anterior vmPFC was more active for the more predictable attacks and showed coupling with the striatum. Our findings converge in showing that during pre-emptive danger, the anterior vmPFC may provide a safety signal, possibly via foreseeable outcomes, while the posterior vmPFC drives unpredictable danger signals.

Bad habits-good goals? Meta-analysis and translation of the habit construct to alcoholism

Excessive alcohol consumption remains a global public health crisis, with millions suffering from alcohol use disorder (AUD, or simply "alcoholism"), leading to significantly reduced life expectancy. This review examines the interplay between habitual and goal-directed behaviors and the associated neurobiological changes induced by chronic alcohol exposure. Contrary to a strict habit-goal dichotomy, our meta-analysis of the published animal experiments combined with a review of human studies reveals a nuanced transition between these behavioral control systems, emphasizing the need for refined terminology to capture the probabilistic nature of decision biases in individuals with a history of chronic alcohol exposure. Furthermore, we distinguish habitual responding from compulsivity, viewing them as separate entities with diverse roles throughout the stages of the addiction cycle. By addressing species-specific differences and translational challenges in habit research, we provide insights to enhance future investigations and inform strategies for combatting AUD.

The role of the human hippocampus in decision-making under uncertainty

The role of the hippocampus in decision-making is beginning to be more understood. Because of its prospective and inferential functions, we hypothesized that it might be required specifically when decisions involve the evaluation of uncertain values. A group of individuals with autoimmune limbic encephalitis-a condition known to focally affect the hippocampus-were tested on how they evaluate reward against uncertainty compared to reward against another key attribute: physical effort. Across four experiments requiring participants to make trade-offs between reward, uncertainty and effort, patients with acute limbic encephalitis demonstrated blunted sensitivity to reward and effort whenever uncertainty was considered, despite demonstrating intact uncertainty sensitivity. By contrast, the valuation of these two attributes (reward and effort) was intact on uncertainty-free tasks. Reduced sensitivity to changes in reward under uncertainty correlated with the severity of hippocampal damage. Together, these findings provide evidence for a context-sensitive role of the hippocampus in value-based decision-making, apparent specifically under conditions of uncertainty.

Encoding of Predictive Associations in Human Prefrontal and Medial Temporal Neurons During Pavlovian Appetitive Conditioning

Pavlovian conditioning is thought to involve the formation of learned associations between stimuli and values, and between stimuli and specific features of outcomes. Here, we leveraged human single neuron recordings in ventromedial prefrontal, dorsomedial frontal, hippocampus, and amygdala while patients of both sexes performed an appetitive Pavlovian conditioning task probing both stimulus-value and stimulus-stimulus associations. Ventromedial prefrontal cortex encoded predictive value along with the amygdala, and also encoded predictions about the identity of stimuli that would subsequently be presented, suggesting a role for neurons in this region in encoding predictive information beyond value. Unsigned error signals were found in dorsomedial frontal areas and hippocampus, potentially supporting learning of non-value related outcome features. Our findings implicate distinct human prefrontal and medial temporal neuronal populations in mediating predictive associations which could partially support model-based mechanisms during Pavlovian conditioning.

Emergent neural dynamics and geometry for generalization in a transitive inference task

Relational cognition-the ability to infer relationships that generalize to novel combinations of objects-is fundamental to human and animal intelligence. Despite this importance, it remains unclear how relational cognition is implemented in the brain due in part to a lack of hypotheses and predictions at the levels of collective neural activity and behavior. Here we discovered, analyzed, and experimentally tested neural networks (NNs) that perform transitive inference (TI), a classic relational task (if A > B and B > C, then A > C). We found NNs that (i) generalized perfectly, despite lacking overt transitive structure prior to training, (ii) generalized when the task required working memory (WM), a capacity thought to be essential to inference in the brain, (iii) emergently expressed behaviors long observed in living subjects, in addition to a novel order-dependent behavior, and (iv) expressed different task solutions yielding alternative behavioral and neural predictions. Further, in a large-scale experiment, we found that human subjects performing WM-based TI showed behavior inconsistent with a class of NNs that characteristically expressed an intuitive task solution. These findings provide neural insights into a classical relational ability, with wider implications for how the brain realizes relational cognition.

Working memory features are embedded in hippocampal place fields

Hippocampal principal neurons display both spatial tuning properties and memory features. Whether this distinction corresponds to separate neuron types or a context-dependent continuum has been debated. We report here that the task-context ("splitter") feature is highly variable along both trial and spatial position axes. Neurons acquire or lose splitter features across trials even when place field features remain unaltered. Multiple place fields of the same neuron can individually encode both past or future run trajectories, implying that splitter fields are under the control of assembly activity. Place fields can be differentiated into subfields by the behavioral choice of the animal, and splitting within subfields evolves across trials. Interneurons also differentiate choices by integrating inputs from pyramidal cells. Finally, bilateral optogenetic inactivation of the medial entorhinal cortex reversibly decreases the fraction of splitter fields. Our findings suggest that place or splitter features are different manifestations of the same hippocampal computation.

Tracking subjects' strategies in behavioural choice experiments at trial resolution

Investigating how, when, and what subjects learn during decision-making tasks requires tracking their choice strategies on a trial-by-trial basis. Here, we present a simple but effective probabilistic approach to tracking choice strategies at trial resolution using Bayesian evidence accumulation. We show this approach identifies both successful learning and the exploratory strategies used in decision tasks performed by humans, non-human primates, rats, and synthetic agents. Both when subjects learn and when rules change the exploratory strategies of win-stay and lose-shift, often considered complementary, are consistently used independently. Indeed, we find the use of lose-shift is strong evidence that subjects have latently learnt the salient features of a new rewarded rule. Our approach can be extended to any discrete choice strategy, and its low computational cost is ideally suited for real-time analysis and closed-loop control.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Dopamine-independent effect of rewards on choices through hidden-state inference

Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world's state, not the dopaminergic RPEs they produce.

The Hippocampus in Pigeons Contributes to the Model-Based Valuation and the Relationship between Temporal Context States

Model-based decision-making guides organism behavior by the representation of the relationships between different states. Previous studies have shown that the mammalian hippocampus (Hp) plays a key role in learning the structure of relationships among experiences. However, the hippocampal neural mechanisms of birds for model-based learning have rarely been reported. Here, we trained six pigeons to perform a two-step task and explore whether their Hp contributes to model-based learning. Behavioral performance and hippocampal multi-channel local field potentials (LFPs) were recorded during the task. We estimated the subjective values using a reinforcement learning model dynamically fitted to the pigeon's choice of behavior. The results show that the model-based learner can capture the behavioral choices of pigeons well throughout the learning process. Neural analysis indicated that high-frequency (12-100 Hz) power in Hp represented the temporal context states. Moreover, dynamic correlation and decoding results provided further support for the high-frequency dependence of model-based valuations. In addition, we observed a significant increase in hippocampal neural similarity at the low-frequency band (1-12 Hz) for common temporal context states after learning. Overall, our findings suggest that pigeons use model-based inferences to learn multi-step tasks, and multiple LFP frequency bands collaboratively contribute to model-based learning. Specifically, the high-frequency (12-100 Hz) oscillations represent model-based valuations, while the low-frequency (1-12 Hz) neural similarity is influenced by the relationship between temporal context states. These results contribute to our understanding of the neural mechanisms underlying model-based learning and broaden the scope of hippocampal contributions to avian behavior.

Distinct value computations support rapid sequential decisions

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Model-free decision making resists improved instructions and is enhanced by stimulus-response associations

Human behaviour may be thought of as supported by two different computational-learning mechanisms, model-free and model-based respectively. In model-free strategies, stimulus-response associations are strengthened when actions are followed by a reward and weakened otherwise. In model-based learning, previous to selecting an action, the current values of the different possible actions are computed based on a detailed model of the environment. Previous research with the two-stage task suggests that participants' behaviour usually shows a mixture of both strategies. But, interestingly, a recent study by da Silva and Hare (2020) found that participants primarily deploy model-based behaviour when they are given detailed instructions about the structure of the task. In the present study, we reproduce this essential experiment. Our results confirm that improved instructions give rise to a stronger model-based component. Crucially, we also found a significant effect of reward that became stronger under conditions that favoured the development of strong stimulus-response associations. This suggests that the effect of reward, often taken as indicator of a model-free component, is related to stimulus-response learning.

Generative replay underlies compositional inference in the hippocampal-prefrontal circuit

Human reasoning depends on reusing pieces of information by putting them together in new ways. However, very little is known about how compositional computation is implemented in the brain. Here, we ask participants to solve a series of problems that each require constructing a whole from a set of elements. With fMRI, we find that representations of novel constructed objects in the frontal cortex and hippocampus are relational and compositional. With MEG, we find that replay assembles elements into compounds, with each replay sequence constituting a hypothesis about a possible configuration of elements. The content of sequences evolves as participants solve each puzzle, progressing from predictable to uncertain elements and gradually converging on the correct configuration. Together, these results suggest a computational bridge between apparently distinct functions of hippocampal-prefrontal circuitry and a role for generative replay in compositional inference and hypothesis testing.

Advances in cellular resolution microscopy for brain imaging in rats

Rats are used in neuroscience research because of their physiological similarities with humans and accessibility as model organisms, trainability, and behavioral repertoire. In particular, rats perform a wide range of sophisticated social, cognitive, motor, and learning behaviors within the contexts of both naturalistic and laboratory environments. Further progress in neuroscience can be facilitated by using advanced imaging methods to measure the complex neural and physiological processes during behavior in rats. However, compared with the mouse, the rat nervous system offers a set of challenges, such as larger brain size, decreased neuron density, and difficulty with head restraint. Here, we review recent advances in in vivo imaging techniques in rats with a special focus on open-source solutions for calcium imaging. Finally, we provide suggestions for both users and developers of in vivo imaging systems for rats.

Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.

Mixtures of strategies underlie rodent behavior during reversal learning

In reversal learning tasks, the behavior of humans and animals is often assumed to be uniform within single experimental sessions to facilitate data analysis and model fitting. However, behavior of agents can display substantial variability in single experimental sessions, as they execute different blocks of trials with different transition dynamics. Here, we observed that in a deterministic reversal learning task, mice display noisy and sub-optimal choice transitions even at the expert stages of learning. We investigated two sources of the sub-optimality in the behavior. First, we found that mice exhibit a high lapse rate during task execution, as they reverted to unrewarded directions after choice transitions. Second, we unexpectedly found that a majority of mice did not execute a uniform strategy, but rather mixed between several behavioral modes with different transition dynamics. We quantified the use of such mixtures with a state-space model, block Hidden Markov Model (block HMM), to dissociate the mixtures of dynamic choice transitions in individual blocks of trials. Additionally, we found that blockHMM transition modes in rodent behavior can be accounted for by two different types of behavioral algorithms, model-free or inference-based learning, that might be used to solve the task. Combining these approaches, we found that mice used a mixture of both exploratory, model-free strategies and deterministic, inference-based behavior in the task, explaining their overall noisy choice sequences. Together, our combined computational approach highlights intrinsic sources of noise in rodent reversal learning behavior and provides a richer description of behavior than conventional techniques, while uncovering the hidden states that underlie the block-by-block transitions.

An automated, low-latency environment for studying the neural basis of behavior in freely moving rats

BACKGROUND

Behavior consists of the interaction between an organism and its environment, and is controlled by the brain. Brain activity varies at sub-second time scales, but behavioral measures are usually coarse (often consisting of only binary trial outcomes).

RESULTS

To overcome this mismatch, we developed the Rat Interactive Foraging Facility (RIFF): a programmable interactive arena for freely moving rats with multiple feeding areas, multiple sound sources, high-resolution behavioral tracking, and simultaneous electrophysiological recordings. The paper provides detailed information about the construction of the RIFF and the software used to control it. To illustrate the flexibility of the RIFF, we describe two complex tasks implemented in the RIFF, a foraging task and a sound localization task. Rats quickly learned to obtain rewards in both tasks. Neurons in the auditory cortex as well as neurons in the auditory field in the posterior insula had sound-driven activity during behavior. Remarkably, neurons in both structures also showed sensitivity to non-auditory parameters such as location in the arena and head-to-body angle.

CONCLUSIONS

The RIFF provides insights into the cognitive capabilities and learning mechanisms of rats and opens the way to a better understanding of how brains control behavior. The ability to do so depends crucially on the combination of wireless electrophysiology and detailed behavioral documentation available in the RIFF.

Case report: "Proust phenomenon" after right posterior cerebral artery occlusion

Odors evoking vivid and intensely felt autobiographical memories are known as the "Proust phenomenon," delineating the particularity of olfaction in being more effective with eliciting emotional memories than other sensory modalities. The phenomenon has been described extensively in healthy participants as well as in patients during pre-epilepsy surgery evaluation after focal stimulation of the amygdalae and post-traumatic stress disorder (PTSD). In this study, we provide the inaugural description of aversive odor-evoked autobiographical memories after stroke in the right hippocampal, parahippocampal, and thalamic nuclei. As potential underlying neural signatures of the phenomenon, we discuss the disinhibition of limbic circuits and impaired communication between the major networks, such as saliency, central executive, and default mode network.

Expertise increases planning depth in human gameplay

A hallmark of human intelligence is the ability to plan multiple steps into the future1,2. Despite decades of research3-5, it is still debated whether skilled decision-makers plan more steps ahead than novices6-8. Traditionally, the study of expertise in planning has used board games such as chess, but the complexity of these games poses a barrier to quantitative estimates of planning depth. Conversely, common planning tasks in cognitive science often have a lower complexity9,10 and impose a ceiling for the depth to which any player can plan. Here we investigate expertise in a complex board game that offers ample opportunity for skilled players to plan deeply. We use model fitting methods to show that human behaviour can be captured using a computational cognitive model based on heuristic search. To validate this model, we predict human choices, response times and eye movements. We also perform a Turing test and a reconstruction experiment. Using the model, we find robust evidence for increased planning depth with expertise in both laboratory and large-scale mobile data. Experts memorize and reconstruct board features more accurately. Using complex tasks combined with precise behavioural modelling might expand our understanding of human planning and help to bridge the gap with progress in artificial intelligence.

Rethinking model-based and model-free influences on mental effort and striatal prediction errors

A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.

Computational approaches to modeling gambling behaviour: Opportunities for understanding disordered gambling

Computational modeling has become an important tool in neuroscience and psychiatry research to provide insight into the cognitive processes underlying normal and pathological behavior. There are two modeling frameworks, reinforcement learning (RL) and drift diffusion modeling (DDM), that are well-developed in cognitive science, and have begun to be applied to Gambling Disorder. RL models focus on explaining how an agent uses reward to learn about the environment and make decisions based on outcomes. The DDM is a binary choice framework that breaks down decision making into psychologically meaningful components based on choice reaction time analyses. Both approaches have begun to yield insight into aspects of cognition that are important for, but not unique to, gambling, and thus relevant to the development of Gambling Disorder. However, these approaches also oversimplify or neglect various aspects of decision making seen in real-world gambling behavior. Gambling Disorder presents an opportunity for 'bespoke' modeling approaches to consider these neglected components. In this review, we discuss studies that have used RL and DDM frameworks to investigate some of the key cognitive components in gambling and Gambling Disorder. We also include an overview of Bayesian models, a methodology that could be useful for more tailored modeling approaches. We highlight areas in which computational modeling could enable progression in the investigation of the cognitive mechanisms relevant to gambling.

Negative symptoms, striatal dopamine and model-free reward decision-making in schizophrenia

Negative symptoms, such as lack of motivation or social withdrawal, are highly prevalent and debilitating in patients with schizophrenia. Underlying mechanisms of negative symptoms are incompletely understood, thereby preventing the development of targeted treatments. We hypothesized that in patients with schizophrenia during psychotic remission, impaired influences of both model-based and model-free reward predictions on decision-making ('reward prediction influence', RPI) underlie negative symptoms. We focused on psychotic remission, because psychotic symptoms might confound reward-based decision-making. Moreover, we hypothesized that impaired model-based/model-free RPIs depend on alterations of both associative striatum dopamine synthesis and storage (DSS) and executive functioning. Both factors influence RPI in healthy subjects and are typically impaired in schizophrenia. Twenty-five patients with schizophrenia with pronounced negative symptoms during psychotic remission and 24 healthy controls were included in the study. Negative symptom severity was measured by the Positive and Negative Syndrome Scale negative subscale, model-based/model-free RPI by the two-stage decision task, associative striatum DSS by 18F-DOPA positron emission tomography and executive functioning by the symbol coding task. Model-free RPI was selectively reduced in patients and associated with negative symptom severity as well as with reduced associative striatum DSS (in patients only) and executive functions (both in patients and controls). In contrast, model-based RPI was not altered in patients. Results provide evidence for impaired model-free reward prediction influence as a mechanism for negative symptoms in schizophrenia as well as for reduced associative striatum dopamine and executive dysfunction as relevant factors. Data suggest potential treatment targets for patients with schizophrenia and pronounced negative symptoms.

Distinct replay signatures for prospective decision-making and memory preservation

Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.

Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task

Humans form complex societies in which we routinely engage in social decision-making regarding the allocation of resources among ourselves and others. One dimension that characterizes social decision-making in particular is whether to prioritize self-interest or respect for others-proself or prosocial. What causes this individual difference in social value orientation? Recent developments in the social dual-process theory argue that social decision-making is characterized by its underlying domain-general learning systems: the model-free and model-based systems. In line with this "learning" approach, we propose and experimentally test the hypothesis that differences in social preferences stem from which learning system is dominant in an individual. Here, we used a non-social state transition task that allowed us to assess the balance between model-free/model-based learning and investigate its relation to the social value orientations. The results showed that proselfs depended more on model-based learning, whereas prosocials depended more on model-free learning. Reward amount and reaction time analyses showed that proselfs learned the task structure earlier in the session than prosocials, reflecting their difference in model-based/model-free learning dependence. These findings support the learning hypothesis on what makes differences in social preferences and have implications for understanding the mechanisms of prosocial behavior.

Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

Arithmetic value representation for hierarchical behavior composition

The ability to compose new skills from a preacquired behavior repertoire is a hallmark of biological intelligence. Although artificial agents extract reusable skills from past experience and recombine them in a hierarchical manner, whether the brain similarly composes a novel behavior is largely unknown. In the present study, I show that deep reinforcement learning agents learn to solve a novel composite task by additively combining representations of prelearned action values of constituent subtasks. Learning efficacy in the composite task was further augmented by the introduction of stochasticity in behavior during pretraining. These theoretical predictions were empirically tested in mice, where subtask pretraining enhanced learning of the composite task. Cortex-wide, two-photon calcium imaging revealed analogous neural representations of combined action values, with improved learning when the behavior variability was amplified. Together, these results suggest that the brain composes a novel behavior with a simple arithmetic operation of preacquired action-value representations with stochastic policies.

Imagination as a fundamental function of the hippocampus

Imagination is a biological function that is vital to human experience and advanced cognition. Despite this importance, it remains unknown how imagination is realized in the brain. Substantial research focusing on the hippocampus, a brain structure traditionally linked to memory, indicates that firing patterns in spatially tuned neurons can represent previous and upcoming paths in space. This work has generally been interpreted under standard views that the hippocampus implements cognitive abilities primarily related to actual experience, whether in the past (e.g. recollection, consolidation), present (e.g. spatial mapping) or future (e.g. planning). However, relatively recent findings in rodents identify robust patterns of hippocampal firing corresponding to a variety of alternatives to actual experience, in many cases without overt reference to the past, present or future. Given these findings, and others on hippocampal contributions to human imagination, we suggest that a fundamental function of the hippocampus is to generate a wealth of hypothetical experiences and thoughts. Under this view, traditional accounts of hippocampal function in episodic memory and spatial navigation can be understood as particular applications of a more general system for imagination. This view also suggests that the hippocampus contributes to a wider range of cognitive abilities than previously thought. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

Prospective and retrospective values integrated in frontal cortex drive predictive choice

To make a deliberate action in a volatile environment, the brain must frequently reassess the value of each action (action-value). Choice can be initially made from the experience of trial-and-errors, but once the dynamics of the environment is learned, the choice can be made from the knowledge of the environment. The action-values constructed from the experience (retrospective value) and the ones from the knowledge (prospective value) were identified in various regions of the brain. However, how and which neural circuit integrates these values and executes the chosen action remains unknown. Combining reinforcement learning and two-photon calcium imaging, we found that the preparatory activity of neurons in a part of the frontal cortex, the anterior-lateral motor (ALM) area, initially encodes retrospective value, but after extensive training, they jointly encode the retrospective and prospective value. Optogenetic inhibition of ALM preparatory activity specifically abolished the expert mice's predictive choice behavior and returned them to the novice-like state. Thus, the integrated action-value encoded in the preparatory activity of ALM plays an important role to bias the action toward the knowledge-dependent, predictive choice behavior.

The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors

The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.

Sampling motion trajectories during hippocampal theta sequences

Efficient planning in complex environments requires that uncertainty associated with current inferences and possible consequences of forthcoming actions is represented. Representation of uncertainty has been established in sensory systems during simple perceptual decision making tasks but it remains unclear if complex cognitive computations such as planning and navigation are also supported by probabilistic neural representations. Here, we capitalized on gradually changing uncertainty along planned motion trajectories during hippocampal theta sequences to capture signatures of uncertainty representation in population responses. In contrast with prominent theories, we found no evidence of encoding parameters of probability distributions in the momentary population activity recorded in an open-field navigation task in rats. Instead, uncertainty was encoded sequentially by sampling motion trajectories randomly and efficiently in subsequent theta cycles from the distribution of potential trajectories. Our analysis is the first to demonstrate that the hippocampus is well equipped to contribute to optimal planning by representing uncertainty.

Predictive maps in rats and humans for spatial navigation

Much of our understanding of navigation comes from the study of individual species, often with specific tasks tailored to those species. Here, we provide a novel experimental and analytic framework integrating across humans, rats, and simulated reinforcement learning (RL) agents to interrogate the dynamics of behavior during spatial navigation. We developed a novel open-field navigation task ("Tartarus maze") requiring dynamic adaptation (shortcuts and detours) to frequently changing obstructions on the path to a hidden goal. Humans and rats were remarkably similar in their trajectories. Both species showed the greatest similarity to RL agents utilizing a "successor representation," which creates a predictive map. Humans also displayed trajectory features similar to model-based RL agents, which implemented an optimal tree-search planning procedure. Our results help refine models seeking to explain mammalian navigation in dynamic environments and highlight the utility of modeling the behavior of different species to uncover the shared mechanisms that support behavior.

Amygdala-cortical collaboration in reward learning and decision making

Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.