Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans

Prefrontal encoding of an internal model for emotional inference

A key function of brain systems mediating emotion is to learn to anticipate unpleasant experiences. Although organisms readily associate sensory stimuli with aversive outcomes, higher-order forms of emotional learning and memory require inference to extrapolate the circumstances surrounding directly experienced aversive events to other indirectly related sensory patterns that were not part of the original experience. This type of learning requires internal models of emotion, which flexibly track directly experienced and inferred aversive associations. Although the brain mechanisms of simple forms of aversive learning have been well studied in areas such as the amygdala1-4, whether and how the brain forms and represents internal models of emotionally relevant associations are not known5. Here we report that neurons in the rodent dorsomedial prefrontal cortex (dmPFC) encode a flexible internal model of emotion by linking sensory stimuli in the environment with aversive events, whether they were directly or indirectly associated with that experience. These representations form through a multi-step encoding mechanism involving recruitment and stabilization of dmPFC cells that support inference. Although dmPFC population activity encodes all salient associations, dmPFC neurons projecting to the amygdala specifically represent and are required to express inferred associations. Together, these findings reveal how internal models of emotion are encoded in the dmPFC to regulate subcortical systems for recall of inferred emotional memories.

Diminished State Space Theory of Human Aging

Many new technologies, such as smartphones, computers, or public-access systems (like ticket-vending machines), are a challenge for older adults. One feature that these technologies have in common is that they involve underlying, partially observable, structures (state spaces) that determine the actions that are necessary to reach a certain goal (e.g., to move from one menu to another, to change a function, or to activate a new service). In this work we provide a theoretical, neurocomputational account to explain these behavioral difficulties in older adults. Based on recent findings from age-comparative computational- and cognitive-neuroscience studies, we propose that age-related impairments in complex goal-directed behavior result from an underlying deficit in the representation of state spaces of cognitive tasks. Furthermore, we suggest that these age-related deficits in adaptive decision-making are due to impoverished neural representations in the orbitofrontal cortex and hippocampus.

Memory consolidation from a reinforcement learning perspective

Memory consolidation refers to the process of converting temporary memories into long-lasting ones. It is widely accepted that new experiences are initially stored in the hippocampus as rapid associative memories, which then undergo a consolidation process to establish more permanent traces in other regions of the brain. Over the past two decades, studies in humans and animals have demonstrated that the hippocampus is crucial not only for memory but also for imagination and future planning, with the CA3 region playing a pivotal role in generating novel activity patterns. Additionally, a growing body of evidence indicates the involvement of the hippocampus, especially the CA1 region, in valuation processes. Based on these findings, we propose that the CA3 region of the hippocampus generates diverse activity patterns, while the CA1 region evaluates and reinforces those patterns most likely to maximize rewards. This framework closely parallels Dyna, a reinforcement learning algorithm introduced by Sutton in 1991. In Dyna, an agent performs offline simulations to supplement trial-and-error value learning, greatly accelerating the learning process. We suggest that memory consolidation might be viewed as a process of deriving optimal strategies based on simulations derived from limited experiences, rather than merely strengthening incidental memories. From this perspective, memory consolidation functions as a form of offline reinforcement learning, aimed at enhancing adaptive decision-making.

The causal structure and computational value of narratives

Many human behavioral and brain imaging studies have used narratively structured stimuli (e.g., written, audio, or audiovisual stories) to better emulate real-world experience in the laboratory. However, narratives are a special class of real-world experience, largely defined by their causal connections across time. Much contemporary neuroscience research does not consider this key property. We review behavioral and neuroscientific work that speaks to how causal structure shapes comprehension of and memory for narratives. We further draw connections between this work and reinforcement learning, highlighting how narratives help link causes to outcomes in complex environments. By incorporating the plausibility of causal connections between classes of actions and outcomes, reinforcement learning models may become more ecologically valid, while simultaneously elucidating the value of narratives.

The role of the human hippocampus in decision-making under uncertainty

The role of the hippocampus in decision-making is beginning to be more understood. Because of its prospective and inferential functions, we hypothesized that it might be required specifically when decisions involve the evaluation of uncertain values. A group of individuals with autoimmune limbic encephalitis-a condition known to focally affect the hippocampus-were tested on how they evaluate reward against uncertainty compared to reward against another key attribute: physical effort. Across four experiments requiring participants to make trade-offs between reward, uncertainty and effort, patients with acute limbic encephalitis demonstrated blunted sensitivity to reward and effort whenever uncertainty was considered, despite demonstrating intact uncertainty sensitivity. By contrast, the valuation of these two attributes (reward and effort) was intact on uncertainty-free tasks. Reduced sensitivity to changes in reward under uncertainty correlated with the severity of hippocampal damage. Together, these findings provide evidence for a context-sensitive role of the hippocampus in value-based decision-making, apparent specifically under conditions of uncertainty.

Heuristics in risky decision-making relate to preferential representation of information

When making choices, individuals differ from one another, as well as from normativity, in how they weigh different types of information. One explanation for this relates to idiosyncratic preferences in what information individuals represent when evaluating choice options. Here, we test this explanation with a simple risky-decision making task, combined with magnetoencephalography (MEG). We examine the relationship between individual differences in behavioral markers of information weighting and neural representation of stimuli pertinent to incorporating that information. We find that the extent to which individuals (N = 19) behaviorally weight probability versus reward information is related to how preferentially they neurally represent stimuli most informative for making probability and reward comparisons. These results are further validated in an additional behavioral experiment (N = 88) that measures stimulus representation as the latency of perceptual detection following priming. Overall, the results suggest that differences in the information individuals consider during choice relate to their risk-taking tendencies.

Grid-like entorhinal representation of an abstract value space during prospective decision making

How valuable a choice option is often changes over time, making the prediction of value changes an important challenge for decision making. Prior studies identified a cognitive map in the hippocampal-entorhinal system that encodes relationships between states and enables prediction of future states, but does not inherently convey value during prospective decision making. In this fMRI study, participants predicted changing values of choice options in a sequence, forming a trajectory through an abstract two-dimensional value space. During this task, the entorhinal cortex exhibited a grid-like representation with an orientation aligned to the axis through the value space most informative for choices. A network of brain regions, including ventromedial prefrontal cortex, tracked the prospective value difference between options. These findings suggest that the entorhinal grid system supports the prediction of future values by representing a cognitive map, which might be used to generate lower-dimensional value signals to guide prospective decision making.

The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate

Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.

The Hippocampus Contributes to Temporal Discounting When Delays and Rewards Are Experienced in the Moment

Temporal discounting (TD) represents the mental devaluation of rewards that are available after a delay. Whether the hippocampus is critical for TD remains unclear, with marked discrepancies between animal and human studies: although animals with discrete hippocampal lesions display impaired TD, human participants with similar lesions show intact performance on classic intertemporal choice tasks. A candidate explanation for this discrepancy is that delays and rewards are experienced in the moment in animal studies but tend to be hypothetical in human studies. We tested this hypothesis by examining the performance of amnesic participants with hippocampal lesions (one female, six males) on a novel experiential intertemporal choice task that used interesting photographs occluded by thick lines as rewards (Patt et al., 2021). Using a logistic function to model indifference points data, we compared performance to that on a classic intertemporal choice task with hypothetical outcomes. Participants with hippocampal lesions displayed impaired patterns of choices in the experiential task but not in the hypothetical task. Specifically, hippocampal lesions were associated with decreased amplitude of the delay-reward trade-off, with persistent choice of the delayed option despite delay increase. These results help explain previous discrepancies across animal and human studies, indicating that the hippocampus plays a critical role in temporal discounting when the outcomes of decisions are experienced in the moment, but not necessarily when they are hypothetical.SIGNIFICANCE STATEMENT Impaired temporal discounting (TD) has been related to maladaptive behaviors, including substance dependence and nonadherence to medical treatment. There is consensus that TD recruits the brain valuation network but whether the hippocampal memory system is additionally recruited remains unclear. This study examined TD in hippocampal amnesia, providing a unique opportunity to explore the role of the hippocampus in cognition. Whereas most human studies have used hypothetical outcomes, this study used a novel experiential task with real-time delays and rewards. Results demonstrated hippocampal involvement in the experiential task, but not in a classic hypothetical task administered for comparison. These findings elucidate previous discrepancies between animal and human TD studies. This reconciliation is critical as animals serve as models of human neurocognition.

Associative memory retrieval modulates upcoming perceptual decisions

Expectations can inform fast, accurate decisions. But what informs expectations? Here we test the hypothesis that expectations are set by dynamic inference from memory. Participants performed a cue-guided perceptual decision task with independently-varying memory and sensory evidence. Cues established expectations by reminding participants of past stimulus-stimulus pairings, which predicted the likely target in a subsequent noisy image stream. Participant's responses used both memory and sensory information, in accordance to their relative reliability. Formal model comparison showed that the sensory inference was best explained when its parameters were set dynamically at each trial by evidence sampled from memory. Supporting this model, neural pattern analysis revealed that responses to the probe were modulated by the specific content and fidelity of memory reinstatement that occurred before the probe appeared. Together, these results suggest that perceptual decisions arise from the continuous sampling of memory and sensory evidence.

Chunking as a rational solution to the speed-accuracy trade-off in a serial reaction time task

When exposed to perceptual and motor sequences, people are able to gradually identify patterns within and form a compact internal description of the sequence. One proposal of how sequences can be compressed is people's ability to form chunks. We study people's chunking behavior in a serial reaction time task. We relate chunk representation with sequence statistics and task demands, and propose a rational model of chunking that rearranges and concatenates its representation to jointly optimize for accuracy and speed. Our model predicts that participants should chunk more if chunks are indeed part of the generative model underlying a task and should, on average, learn longer chunks when optimizing for speed than optimizing for accuracy. We test these predictions in two experiments. In the first experiment, participants learn sequences with underlying chunks. In the second experiment, participants were instructed to act either as fast or as accurately as possible. The results of both experiments confirmed our model's predictions. Taken together, these results shed new light on the benefits of chunking and pave the way for future studies on step-wise representation learning in structured domains.

The blockade of orexin receptors within the dentate gyrus of the hippocampus attenuated methamphetamine-induced reward learning during conditioning place preference

Orexins and orexinergic receptors have been shown to play a critical role in reward processing and drug addiction. Previous studies showed that the orexinergic system in the dentate gyrus (DG) region of the hippocampus affects the conditioning (acquisition) and post-conditioning (expression) phases of morphine-induced conditioned place preference (CPP). The action of each orexin receptor within the DG during conditioning and expression phases for methamphetamine (METH)-induced CPP remains unclear. The present study aimed to determine the role of orexin-1 and -2 receptors in the hippocampal DG in METH CPP acquisition and expression. During the 5-day conditioning phase, rats received an intra-DG microinjection of SB334867, a selective orexin-1 receptor (OX1R) antagonist, or TCS OX2-29, a selective orexin-2 receptor (OX2R) antagonist, before injection of METH (1 mg/kg; sc). In different sets of animals on the expression day, rats received each antagonist before the CPP test. The results showed that SB334867 (3, 10, and 30 nmol) and TCS OX2-29 (3, 10, and 30 nmol) significantly decreased the acquisition of METH CPP during the conditioning phase. Furthermore, administration of SB 334867 (10 and 30 nmol) and TCS OX2-29 (3 and 10 nmol) on the post-conditioning day significantly reduced METH-induced CPP expression. The results also indicated that orexin receptors play a more critical role in the conditioning phase than in the expression phase. In summary, the orexin receptors in the DG play a crucial role in drug learning and memory and are essential for METH reward acquisition and expression.

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Distinct replay signatures for prospective decision-making and memory preservation

Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.

Choice Type Impacts Human Reinforcement Learning

In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus-response values that change incrementally. RL models consider any response type indiscriminately, ranging from more concretely defined motor choices (pressing a key with the index finger), to more general choices that can be executed in a number of ways (selecting dinner at the restaurant). However, does the learning process vary as a function of the choice type? In Experiment 1, we show that it does: Participants were slower and less accurate in learning correct choices of a general format compared with learning more concrete motor actions. Using computational modeling, we show that two mechanisms contribute to this. First, there was evidence of irrelevant credit assignment: The values of motor actions interfered with the values of other choice dimensions, resulting in more incorrect choices when the correct response was not defined by a single motor action; second, information integration for relevant general choices was slower. In Experiment 2, we replicated and further extended the findings from Experiment 1 by showing that slowed learning was attributable to weaker working memory use, rather than slowed RL. In both experiments, we ruled out the explanation that the difference in performance between two condition types was driven by difficulty/different levels of complexity. We conclude that defining a more abstract choice space used by multiple learning systems for credit assignment recruits executive resources, limiting how much such processes then contribute to fast learning.

Hippocampal Contribution to Probabilistic Feedback Learning: Modeling Observation- and Reinforcement-based Processes

Simple probabilistic reinforcement learning is recognized as a striatum-based learning system, but in recent years, has also been associated with hippocampal involvement. This study examined whether such involvement may be attributed to observation-based learning (OL) processes, running in parallel to striatum-based reinforcement learning. A computational model of OL, mirroring classic models of reinforcement-based learning (RL), was constructed and applied to the neuroimaging data set of Palombo, Hayes, Reid, and Verfaellie (2019). Hippocampal contributions to value-based learning: Converging evidence from fMRI and amnesia. Cognitive, Affective & Behavioral Neuroscience, 19(3), 523-536. Results suggested that OL processes may indeed take place concomitantly to reinforcement learning and involve activation of the hippocampus and central orbitofrontal cortex. However, rather than independent mechanisms running in parallel, the brain correlates of the OL and RL prediction errors indicated collaboration between systems, with direct implication of the hippocampus in computations of the discrepancy between the expected and actual reinforcing values of actions. These findings are consistent with previous accounts of a role for the hippocampus in encoding the strength of observed stimulus-outcome associations, with updating of such associations through striatal reinforcement-based computations. In addition, enhanced negative RL prediction error signaling was found in the anterior insula with greater use of OL over RL processes. This result may suggest an additional mode of collaboration between the OL and RL systems, implicating the error monitoring network.

Humans can navigate complex graph structures acquired during latent learning

Humans appear to represent many forms of knowledge in associative networks whose nodes are multiply connected, including sensory, spatial, and semantic. Recent work has shown that explicitly augmenting artificial agents with such graph-structured representations endows them with more human-like capabilities of compositionality and transfer learning. An open question is how humans acquire these representations. Previously, it has been shown that humans can learn to navigate graph-structured conceptual spaces on the basis of direct experience with trajectories that intentionally draw the network contours (Schapiro, Kustner, & Turk-Browne, 2012; Schapiro, Turk-Browne, Botvinick, & Norman, 2016), or through direct experience with rewards that covary with the underlying associative distance (Wu, Schulz, Speekenbrink, Nelson, & Meder, 2018). Here, we provide initial evidence that this capability is more general, extending to learning to reason about shortest-path distances across a graph structure acquired across disjoint experiences with randomized edges of the graph - a form of latent learning. In other words, we show that humans can infer graph structures, assembling them from disordered experiences. We further show that the degree to which individuals learn to reason correctly and with reference to the structure of the graph corresponds to their propensity, in a separate task, to use model-based reinforcement learning to achieve rewards. This connection suggests that the correct acquisition of graph-structured relationships is a central ability underlying forward planning and reasoning, and may be a core computation across the many domains in which graph-based reasoning is advantageous.

Cognitive and neural principles of a memory bias on preferential choices

Value-based decisions depend on different forms of memory. However, the respective roles of memory and valuation processes that give rise to these decisions are often vaguely described and have rarely been investigated jointly. In this review article, we address the problem of memory-based decision making from a neuroeconomic perspective. We first describe the neural and cognitive processes involved in decisions requiring memory processes, with a focus on episodic memory. Based on the results of a systematic research program, we then spotlight the phenomenon of the memory bias, a general preference for choice options that can be retrieved from episodic memory more successfully. Our findings indicate that failed memory recall biases neural valuation processes as indicated by altered effective connectivity between the hippocampus and ventromedial prefrontal cortex. This bias can be attributed to meta-cognitive beliefs about the relationship between subjective value and memory as well as to uncertainty aversion. After summarizing the findings, we outline potential future research endeavors to integrate the two research traditions of memory and decision making.

Gated recurrence enables simple and accurate sequence prediction in stochastic, changing, and structured environments

From decision making to perception to language, predicting what is coming next is crucial. It is also challenging in stochastic, changing, and structured environments; yet the brain makes accurate predictions in many situations. What computational architecture could enable this feat? Bayesian inference makes optimal predictions but is prohibitively difficult to compute. Here, we show that a specific recurrent neural network architecture enables simple and accurate solutions in several environments. This architecture relies on three mechanisms: gating, lateral connections, and recurrent weight training. Like the optimal solution and the human brain, such networks develop internal representations of their changing environment (including estimates of the environment’s latent variables and the precision of these estimates), leverage multiple levels of latent structure, and adapt their effective learning rate to changes without changing their connection weights. Being ubiquitous in the brain, gated recurrence could therefore serve as a generic building block to predict in real-life environments.

Mixing memory and desire: How memory reactivation supports deliberative decision-making

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

Neural Substrates of Incidental Associations and Mediated Learning: The Role of Cannabinoid Receptors

The ability to form associations between different stimuli in the environment to guide adaptive behavior is a central element of learning processes, from perceptual learning in humans to Pavlovian conditioning in animals. Like so, classical conditioning paradigms that test direct associations between low salience sensory stimuli and high salience motivational reinforcers are extremely informative. However, a large part of everyday learning cannot be solely explained by direct conditioning mechanisms - this includes to a great extent associations between individual sensory stimuli, carrying low or null immediate motivational value. This type of associative learning is often described as incidental learning and can be captured in animal models through sensory preconditioning procedures. Here we summarize the evolution of research on incidental and mediated learning, overview the brain systems involved and describe evidence for the role of cannabinoid receptors in such higher-order learning tasks. This evidence favors a number of contemporary hypotheses concerning the participation of the endocannabinoid system in psychosis and psychotic experiences and provides a conceptual framework for understanding how the use of cannabinoid drugs can lead to altered perceptive states.

Cannabidiol modulates the METH-induced conditioned place preference through D2-like dopamine receptors in the hippocampal CA1 region

The main problem with addiction is a relapse with a high rate in methamphetamine (METH) abusers. Using addictive drugs repetitively will cause the reward. METH reward is due to an increase in dopamine levels, and the endocannabinoid system (ECS) has a modulatory role in reward through CB1 receptors. On the other hand, the hippocampus plays an important role in learning and memory, so it is involved in the neuroplasticity caused by METH abuse. Cannabidiol (CBD) has been shown to reduce the effects of METH through different mechanisms such as increasing the ECS activity, regulating emotional memory in the ventral hippocampus through D2-like dopamine receptors, and decreasing the mesolimbic dopaminergic activity. The present study tried to find out the role of hippocampal CA1 D2-like dopamine receptors (D2R) in the effects of cannabidiol on the acquisition and expression of METH-induced conditioned place preference (METH-CPP) in rats by using microinjection of sulpiride as a D2R antagonist. For this purpose, different groups of animals received different doses of sulpiride (0.25, 1, and 4 μg/0.5 μL DMSO; CA1), once prior to the injection of CBD (10 μg/5 μL for acquisition and 50 μg/5 μL for expression; ICV) and once in the absence of CBD. Control groups were also considered. In brief, findings showed that cannabidiol decreases METH-induced CPP. Intra-CA1 administration of sulpiride reversed the decreasing effects of cannabidiol on METH-induced CPP in both acquisition and expression phases but more prominent in the expression phase. The results showed that sulpiride did not affect the METH-induced CPP in the absence of cannabidiol. In conclusion, this study demonstrated that cannabidiol decreased METH-induced CPP in part through interaction with hippocampal CA1 D2-dopamine receptors.

Sequence Structure Has a Differential Effect on Underlying Motor Learning Processes

Current methods to understand implicit motor sequence learning inadequately assess motor skill acquisition in daily life. Using fixed sequences in the serial reaction time task is not ideal as participants may become aware of the sequence, thereby changing the learning from implicit to explicit. Probabilistic sequences, in which stimuli are linked by statistical, rather than deterministic, associations can ensure that learning remains implicit. Additionally, the processes underlying the learning of motor sequences may differ based on sequence structure. Here, the authors compared the learning of fixed and probabilistic sequences to randomly ordered stimuli using a modified serial reaction time task. Both the fixed and probabilistic sequence groups exhibited learning as indicated by decreased response time and variability. In the initial stage of learning, fixed sequences exhibited both online and offline gains in response time; however, only the offline gain was observed during the learning of probabilistic sequences. These results indicated that probabilistic structures may be learned differently from fixed structures and have important implications for our current understanding of motor learning. Probabilistic sequences more accurately reflect motor skill acquisition in daily life, offer ecological validity to the serial reaction time framework, and advance our understanding of motor learning.

Beyond the mean reaction time: Trial-by-trial reaction time reveals the distraction effect on perceptual-motor sequence learning

Perceptual-motor sequences can be learned quickly under distraction, often demonstrated by the mean reaction time (RT) change in a serial reaction time (SRT) task. However, any arbitrary mean RT can arise from one of many distinct trial-by-trial RT patterns. It is surprising that previous sequence learning studies have hinged only on the mean RT metrics while little is known about the distraction effect on its trial-by-trial processes. In an SRT task with or without distraction, we found that initially learning a fixed repeating sequence without distraction was expressed by a micro-online learning process where reaction time (RT) progressively improved within learning blocks as adults continuously performed the SRT task. Such online RT improvements, however, vanished when the SRT task was performed under distraction. Despite the detrimental effect of distraction on micro-online RT improvements, we observed offline enhancements in RT following rest intervals of 3 min that emerged to secure sequence learning under distraction. We reasoned that distraction may exert influence on the micro-online and offline learning by mediating the engagement of explicit and implicit memory. Given the offline RT change under distraction, a short rest between learning blocks may be a key player in early perceptual-motor sequence learning under distraction. We thus suggest that future studies investigating the distraction effect on sequence learning need to control the length of rest between learning blocks, while previous research with equivocal interpretations of the distraction effect failed to do so.

"Chasing the first high": memory sampling in drug choice

Although vivid memories of drug experiences are prevalent within clinical contexts and addiction folklore ("chasing the first high"), little is known about the relevance of cognitive processes governing memory retrieval to substance use disorder. Drawing on recent work that identifies episodic memory's influence on decisions for reward, we propose a framework in which drug choices are biased by selective sampling of individual memories during two phases of addiction: (i) downward spiral into persistent use and (ii) relapse. Consideration of how memory retrieval influences the addiction process suggests novel treatment strategies. Rather than try to break learned associations between drug cues and drug rewards, treatment should aim to strengthen existing and/or create new associations between drug cues and drug-inconsistent rewards.

Mouse tracking reveals structure knowledge in the absence of model-based choice

Converging evidence has demonstrated that humans exhibit two distinct strategies when learning in complex environments. One is model-free learning, i.e., simple reinforcement of rewarded actions, and the other is model-based learning, which considers the structure of the environment. Recent work has argued that people exhibit little model-based behavior unless it leads to higher rewards. Here we use mouse tracking to study model-based learning in stochastic and deterministic (pattern-based) environments of varying difficulty. In both tasks participants’ mouse movements reveal that they learned the structures of their environments, despite the fact that standard behavior-based estimates suggested no such learning in the stochastic task. Thus, we argue that mouse tracking can reveal whether subjects have structure knowledge, which is necessary but not sufficient for model-based choice.

Mouse tracking can reveal people’s subjective beliefs and whether they understand the structure of a task. These data demonstrate that people often do not use this information to make good choices.

Goal-oriented and habitual decisions: Neural signatures of model-based and model-free learning

Human decision-making is mainly driven by two fundamental learning processes: a slow, deliberative, goal-directed model-based process that maps out the potential outcomes of all options and a rapid habitual model-free process that enables reflexive repetition of previously successful choices. Although many model-informed neuroimaging studies have examined the neural correlates of model-based and model-free learning, the concordant activity among these two processes remains unclear. We used quantitative meta-analyses of functional magnetic resonance imaging experiments to identify the concordant activity pertaining to model-based and model-free learning over a range of reward-related paradigms. We found that: 1) both processes yielded concordant ventral striatum activity, 2) model-based learning activated the medial prefrontal cortex and orbital frontal cortex, and 3) model-free learning specifically activated the left globus pallidus and right caudate head. Our findings suggest that model-free and model-based decision making engage overlapping yet distinct neural regions. These stereotaxic maps improve our understanding of how deliberative goal-directed and reflexive habitual learning are implemented in the brain.

Transmission Electron Microscopy Study of Mitochondria in Aging Brain Synapses

The brain is sensitive to aging-related morphological changes, where many neurodegenerative diseases manifest accompanied by a reduction in memory. The hippocampus is especially vulnerable to damage at an early stage of aging. The present transmission electron microscopy study examined the synapses and synaptic mitochondria of the CA1 region of the hippocampal layer in young-adult and old rats by means of a computer-assisted image analysis technique. Comparing young-adult (10 months of age) and old (22 months) male Fischer (CDF) rats, the total numerical density of synapses was significantly lower in aged rats than in the young adults. This age-related synaptic loss involved degenerative changes in the synaptic architectonic organization, including damage to mitochondria in both pre- and post-synaptic compartments. The number of asymmetric synapses with concave curvature decreased with age, while the number of asymmetric synapses with flat and convex curvatures increased. Old rats had a greater number of damaged mitochondria in their synapses, and most of this was type II and type III mitochondrial structural damage. These results demonstrate age-dependent changes in the morphology of synaptic mitochondria that may underlie declines in age-related synaptic function and may couple to age-dependent loss of synapses.

Computing Social Value Conversion in the Human Brain

Social signals play powerful roles in shaping self-oriented reward valuation and decision making. These signals activate social and valuation/decision areas, but the core computation for their integration into the self-oriented decision machinery remains unclear. Here, we study how a fundamental social signal, social value (others' reward value), is converted into self-oriented decision making in the human brain. Using behavioral analysis, modeling, and neuroimaging, we show three-stage processing of social value conversion from the offer to the effective value and then to the final decision value. First, a value of others' bonus on offer, called offered value, was encoded uniquely in the right temporoparietal junction (rTPJ) and also in the left dorsolateral prefrontal cortex (ldlPFC), which is commonly activated by offered self-bonus value. The effective value, an intermediate value representing the effective influence of the offer on the decision, was represented in the right anterior insula (rAI), and the final decision value was encoded in the medial prefrontal cortex (mPFC). Second, using psychophysiological interaction and dynamic causal modeling analyses, we demonstrated three-stage feedforward processing from the rTPJ and ldPFC to the rAI and then from rAI to the mPFC. Further, we showed that these characteristics of social conversion underlie distinct sociobehavioral phenotypes. We demonstrate that the variability in the conversion underlies the difference between prosocial and selfish subjects, as seen from the differential strength of the rAI and ldlPFC coupling to the mPFC responses, respectively. Together, these findings identified fundamental neural computation processes for social value conversion underlying complex social decision making behaviors.SIGNIFICANCE STATEMENT In daily life, we make decisions based on self-interest, but also in consideration for others' status. These social influences modulate valuation and decision signals in the brain, suggesting a fundamental process called value conversion that translates social information into self-referenced decisions. However, little is known about the conversion process and its underlying brain mechanisms. We investigated value conversion using human fMRI with computational modeling and found three essential stages in a progressive brain circuit from social to empathic and decision areas. Interestingly, the brain mechanism of conversion differed between prosocial and individualistic subjects. These findings reveal how the brain processes and merges social information into the elemental flow of self-interested decision making.

Understanding the Neural Bases of Implicit and Statistical Learning

Both implicit learning and statistical learning focus on the ability of learners to pick up on patterns in the environment. It has been suggested that these two lines of research may be combined into a single construct of "implicit statistical learning." However, by comparing the neural processes that give rise to implicit versus statistical learning, we may determine the extent to which these two learning paradigms do indeed describe the same core mechanisms. In this review, we describe current knowledge about neural mechanisms underlying both implicit learning and statistical learning, highlighting converging findings between these two literatures. A common thread across all paradigms is that learning is supported by interactions between the declarative and nondeclarative memory systems of the brain. We conclude by discussing several outstanding research questions and future directions for each of these two research fields. Moving forward, we suggest that the two literatures may interface by defining learning according to experimental paradigm, with "implicit learning" reserved as a specific term to denote learning without awareness, which may potentially occur across all paradigms. By continuing to align these two strands of research, we will be in a better position to characterize the neural bases of both implicit and statistical learning, ultimately improving our understanding of core mechanisms that underlie a wide variety of human cognitive abilities.

Habits without values

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Remembering rewarding futures: A simulation-selection model of the hippocampus

Despite tremendous progress, the neural circuit dynamics underlying hippocampal mnemonic processing remain poorly understood. We propose a new model for hippocampal function-the simulation-selection model-based on recent experimental findings and neuroecological considerations. Under this model, the mammalian hippocampus evolved to simulate and evaluate arbitrary navigation sequences. Specifically, we suggest that CA3 simulates unexperienced navigation sequences in addition to remembering experienced ones, and CA1 selects from among these CA3-generated sequences, reinforcing those that are likely to maximize reward during offline idling states. High-value sequences reinforced in CA1 may allow flexible navigation toward a potential rewarding location during subsequent navigation. We argue that the simulation-selection functions of the hippocampus have evolved in mammals mostly because of the unique navigational needs of land mammals. Our model may account for why the mammalian hippocampus has evolved not only to remember, but also to imagine episodes, and how this might be implemented in its neural circuits.

The influence of episodic memory decline on value-based choice

Recent studies suggest the involvement of episodic memory in value-based decisions as a source of information about subjective values of choice options. We therefore tested the link between age-related memory decline and inconsistencies in value-based decisions in 30 cognitively healthy older adults. Within the pre-registered experiment, the inconsistencies were measured in two ways: i) the consistency between stated preferences and revealed choices; ii) the amount of intransitivities in choice triplets, revealed in a forced paired choice task including all possible pairings of 20 food products. Although no significant association of memory functions to number of intransitive triplets was observed, participants with lower memory scores were more likely to choose the item for which they stated a lower preference. The results suggest a higher noise in the underlying preference signal in participants with lower memory. We discuss the results in the context of the unique needs of elderly consumers.

Learning Naturalistic Temporal Structure in the Posterior Medial Network

The posterior medial network is at the apex of a temporal integration hierarchy in the brain, integrating information over many seconds of viewing intact, but not scrambled, movies. This has been interpreted as an effect of temporal structure. Such structure in movies depends on preexisting event schemas, but temporal structure can also arise de novo from learning. Here, we examined the relative role of schema-consistent temporal structure and arbitrary but consistent temporal structure on the human posterior medial network. We tested whether, with repeated viewing, the network becomes engaged by scrambled movies with temporal structure. Replicating prior studies, activity in posterior medial regions was immediately locked to stimulus structure upon exposure to intact, but not scrambled, movies. However, for temporally structured scrambled movies, functional coupling within the network increased across stimulus repetitions, rising to the level of intact movies. Thus, temporal structure is a key determinant of network dynamics and function in the posterior medial network.

Manipulating memory efficacy affects the behavioral and neural profiles of deterministic learning and decision-making

When making a decision, we have to identify, collect, and evaluate relevant bits of information to ensure an optimal outcome. How we approach a given choice can be influenced by prior experience. Contextual factors and structural elements of these past decisions can cause a shift in how information is encoded and can in turn influence later decision-making. In this two-experiment study, we sought to manipulate declarative memory efficacy and decision-making in a concurrent discrimination learning task by altering the amount of information to be learned. Subjects learned correct responses to pairs of items across several repetitions of a 50- or 100-pair set and were tested for memory retention. In one experiment, this memory test interrupted learning after an initial encoding experience in order to test for early encoding differences and associate those differences with changes in decision-making. In a second experiment, we used fMRI to probe neural differences between the two list-length groups related to decision-making across learning and assessed subsequent memory retention. We found that a striatum-based system was associated with decision-making patterns when learning a longer list of items, while a medial cortical network was associated with patterns when learning a shorter list. Additionally, the hippocampus was exclusively active for the shorter list group. Altogether, these behavioral, computational, and imaging results provide evidence that multiple types of mnemonic representations contribute to experienced-based decision-making. Moreover, contextual and structural factors of the task and of prior decisions can influence what types of evidence are drawn upon during decision-making.

Differential patterns of contextual organization of memory in first-episode psychosis

Contextual information is used to support and organize episodic memory. Prior research has reliably shown memory deficits in psychosis; however, little research has characterized how this population uses contextual information during memory recall. We employed an approach founded in a computational framework of free recall to quantify how individuals with first episode of psychosis (FEP, N = 97) and controls (CON, N = 55) use temporal and semantic context to organize memory recall. Free recall was characterized using the Hopkins Verbal Learning Test-Revised (HVLT-R). We compared FEP and CON on three measures of free recall: proportion recalled, temporal clustering, and semantic clustering. Measures of temporal/semantic clustering quantified how individuals use contextual information to organize memory recall. We also assessed to what extent these measures relate to antipsychotic use and differentiated between different types of psychosis. We also explored the relationship between these measures and intelligence. In comparison to CON, FEP had reduced recall and less temporal clustering during free recall (p < 0.05, Bonferroni-corrected), and showed a trend towards greater semantic clustering (p = 0.10, Bonferroni-corrected). Within FEP, antipsychotic use and diagnoses did not differentiate between free recall accuracy or contextual organization of memory. IQ was related to free recall accuracy, but not the use of contextual information during recall in either group (p < 0.05, Bonferroni-corrected). These results show that in addition to deficits in memory recall, FEP differed in how they organize memories compared to CON.

First-episode psychosis patients exhibit impaired memory recall and deviation in how context is used to support recall ability. A US team of researchers led by the University of Pittsburgh’s Vishnu Murty examined how FEP affects an individual’s ability to organize memory based on context, by noting how well patients could recall words from a spoken list. Alongside recollection accuracy, Murty’s team assesed participant ability to recall words said proximally in sequence, and the ability to recall words from the same category—measuring ‘temporal clustering’ and ‘semantic clustering.’ The researchers found that patients with FEP had reduced recall ability and less temporal clustering. Recall accuracy and IQ were also found to be related. This study increases knowledge of FEP-related cognitive changes and could help to target specific therapies.

Parsing the Role of the Hippocampus in Approach-Avoidance Conflict

The hippocampus plays a central role in the approach-avoidance conflict that is central to the genesis of anxiety. However, its exact functional contribution has yet to be identified. We designed a novel gambling task that generated approach-avoidance conflict while controlling for spatial processing. We fit subjects' behavior using a model that quantified the subjective values of choice options, and recorded neural signals using functional magnetic resonance imaging (fMRI). Distinct functional signals were observed in anterior hippocampus, with inferior hippocampus selectively recruited when subjects rejected a gamble, to a degree that covaried with individual differences in anxiety. The superior anterior hippocampus, in contrast, uniquely demonstrated value signals that were potentiated in the context of approach-avoidance conflict. These results implicate the anterior hippocampus in behavioral avoidance and choice monitoring, in a manner relevant to understanding its role in anxiety. Our findings highlight interactions between subregions of the hippocampus as an important focus for future study.

Decision Making and Sequential Sampling from Memory

Decisions take time, and as a rule more difficult decisions take more time. But this only raises the question of what consumes the time. For decisions informed by a sequence of samples of evidence, the answer is straightforward: more samples are available with more time. Indeed, the speed and accuracy of such decisions are explained by the accumulation of evidence to a threshold or bound. However, the same framework seems to apply to decisions that are not obviously informed by sequences of evidence samples. Here, we proffer the hypothesis that the sequential character of such tasks involves retrieval of evidence from memory. We explore this hypothesis by focusing on value-based decisions and argue that mnemonic processes can account for regularities in choice and decision time. We speculate on the neural mechanisms that link sampling of evidence from memory to circuits that represent the accumulated evidence bearing on a choice. We propose that memory processes may contribute to a wider class of decisions that conform to the regularities of choice-reaction time predicted by the sequential sampling framework.

Reminders of past choices bias decisions for reward in humans

We provide evidence that decisions are made by consulting memories for individual past experiences, and that this process can be biased in favour of past choices using incidental reminders. First, in a standard rewarded choice task, we show that a model that estimates value at decision-time using individual samples of past outcomes fits choices and decision-related neural activity better than a canonical incremental learning model. In a second experiment, we bias this sampling process by incidentally reminding participants of individual past decisions. The next decision after a reminder shows a strong influence of the action taken and value received on the reminded trial. These results provide new empirical support for a decision architecture that relies on samples of individual past choice episodes rather than incrementally averaged rewards in evaluating options and has suggestive implications for the underlying cognitive and neural mechanisms.

Unraveling the Role of the Hippocampus in Reversal Learning

Research in reversal learning has mainly focused on the functional role of dopamine and striatal structures in driving behavior on the basis of classic reinforcement learning mechanisms. However, recent evidence indicates that, beyond classic reinforcement learning adaptations, individuals may also learn the inherent task structure and anticipate the occurrence of reversals. A candidate structure to support such task representation is the hippocampus, which might create a flexible representation of the environment that can be adaptively applied to goal-directed behavior. To investigate the functional role of the hippocampus in the implementation of anticipatory strategies in reversal learning, we first studied, in 20 healthy individuals (11 women), whether the gray matter anatomy and volume of the hippocampus were related to anticipatory strategies in a reversal learning task. Second, we tested 20 refractory temporal lobe epileptic patients (11 women) with unilateral hippocampal sclerosis, who served as a hippocampal lesion model. Our results indicate that healthy participants were able to learn the task structure and use it to guide their behavior and optimize their performance. Participants' ability to adopt anticipatory strategies correlated with the gray matter volume of the hippocampus. In contrast, hippocampal patients were unable to grasp the higher-order structure of the task with the same success than controls. Present results indicate that the hippocampus is necessary to respond in an appropriately flexible manner to high-order environments, and disruptions in this structure can render behavior habitual and inflexible.SIGNIFICANCE STATEMENT Understanding the neural substrates involved in reversal learning has provoked a great deal of interest in the last years. Studies with nonhuman primates have shown that, through repetition, individuals are able to anticipate the occurrence of reversals and, thus, adjust their behavior accordingly. The present investigation is devoted to know the role of the hippocampus in such strategies. Importantly, our findings evidence that the hippocampus is necessary to anticipate the occurrence of reversals, and disruptions in this structure can render behavior habitual and inflexible.

Neural Signals Related to Outcome Evaluation Are Stronger in CA1 than CA3

We have shown previously that CA1 conveys significant neural signals necessary to update value of the chosen target, namely chosen value and reward signals. To better understand hippocampal neural processes related to valuation, we compared chosen value- and reward-related neural activity between the CA3 and CA1 regions. Single units were recorded with tetrodes from the dorsal CA3 and CA1 regions of rats performing a dynamic foraging task, and chosen value- and reward-related neural activity was estimated using a reinforcement learning model and multiple regression analyses. Neural signals for chosen value and reward converged in both CA3 and CA1 when a trial outcome was revealed. However, these neural signals were stronger in CA1 than CA3. Consequently, neural signals for reward prediction error and updated chosen value were stronger in CA1 than CA3. Together with our previous finding that CA1 conveys stronger value signals than the subiculum, our results raise the possibility that CA1 might play a particularly important role among hippocampal subregions in evaluating experienced events.

Flexible weighting of diverse inputs makes hippocampal function malleable

Classic theories of hippocampal function have emphasized its role as a dedicated memory system, but recent research has shown that it contributes broadly to many aspects of cognition, including attention and perception. We propose that the reason the hippocampus plays such a broad role in cognition is that its function is particularly malleable. We argue that this malleability arises because the hippocampus receives diverse anatomical inputs and these inputs are flexibly weighted based on behavioral goals. We discuss examples of how hippocampal representations can be flexibly weighted, focusing on hippocampal modulation by attention. Finally, we suggest some general neural mechanisms and core hippocampal computations that may enable the hippocampus to support diverse cognitive functions, including attention, perception, and memory. Together, this work suggests that great progress can and has been made in understanding the hippocampus by considering how the domain-general computations it performs allow it to dynamically contribute to many different behaviors.

A map of abstract relational knowledge in the human hippocampal-entorhinal cortex

The hippocampal-entorhinal system encodes a map of space that guides spatial navigation. Goal-directed behaviour outside of spatial navigation similarly requires a representation of abstract forms of relational knowledge. This information relies on the same neural system, but it is not known whether the organisational principles governing continuous maps may extend to the implicit encoding of discrete, non-spatial graphs. Here, we show that the human hippocampal-entorhinal system can represent relationships between objects using a metric that depends on associative strength. We reconstruct a map-like knowledge structure directly from a hippocampal-entorhinal functional magnetic resonance imaging adaptation signal in a situation where relationships are non-spatial rather than spatial, discrete rather than continuous, and unavailable to conscious awareness. Notably, the measure that best predicted a behavioural signature of implicit knowledge and blood oxygen level-dependent adaptation was a weighted sum of future states, akin to the successor representation that has been proposed to account for place and grid-cell firing patterns.

Children and Adults Both Learn Motor Sequences Quickly, But Do So Differently

Both children and adults can learn motor sequences quickly in one learning session, yet little is known about potential age-related processes that underlie this fast sequence acquisition. Here, we examined the progressive performance changes in a one-session modified serial reaction time task in 6- and 10-year-old children and adults. We found that rapid sequence learning, as reflected by reaction time (RT), was comparable between groups. The learning was expressed through two behavioral processes: online progressive changes in RT while the task was performed in a continuous manner and offline changes in RT that emerged following a short rest. These offline and online RT changes were age-related; learning in 6-year-olds was primarily reflected through the offline process. In contrast, learning in adults was reflected through the online process; and both online and offline processes occurred concurrently in 10-year-olds. Our results suggest that early rapid sequence learning has a developmental profile. Although the unifying mechanism underlying these two age-related processes is unclear, we discuss possible explanations that need to be systematically elucidated in future studies.

The Neural Representation of Prospective Choice during Spatial Planning and Decisions

We are remarkably adept at inferring the consequences of our actions, yet the neuronal mechanisms that allow us to plan a sequence of novel choices remain unclear. We used functional magnetic resonance imaging (fMRI) to investigate how the human brain plans the shortest path to a goal in novel mazes with one (shallow maze) or two (deep maze) choice points. We observed two distinct anterior prefrontal responses to demanding choices at the second choice point: one in rostrodorsal medial prefrontal cortex (rd-mPFC)/superior frontal gyrus (SFG) that was also sensitive to (deactivated by) demanding initial choices and another in lateral frontopolar cortex (lFPC), which was only engaged by demanding choices at the second choice point. Furthermore, we identified hippocampal responses during planning that correlated with subsequent choice accuracy and response time, particularly in mazes affording sequential choices. Psychophysiological interaction (PPI) analyses showed that coupling between the hippocampus and rd-mPFC increases during sequential (deep versus shallow) planning and is higher before correct versus incorrect choices. In short, using a naturalistic spatial planning paradigm, we reveal how the human brain represents sequential choices during planning without extensive training. Our data highlight a network centred on the cortical midline and hippocampus that allows us to make prospective choices while maintaining initial choices during planning in novel environments.

Using neuroimaging and computational modelling, this study explains how the human brain represents initial versus subsequent choices during spatial planning in novel environments.

We are remarkably adept at inferring the consequences of our actions, even in novel situations. However, the neuronal mechanisms that allow us to plan a sequence of novel choices remain a mystery. One hypothesis is that anterior prefrontal brain regions can jump ahead from an initial decision to evaluate subsequent choices. Here, we examine how the brain represents initial versus subsequent choices of varying difficulty during spatial planning in novel environments. Specifically, participants visually searched for the shortest path to a goal in pictures of novel mazes that contained one or two path junctions. We monitored the participants’ brain activity during the task with functional magnetic resonance imaging (fMRI). We observed, in the anterior prefrontal brain, two distinct responses to demanding choices at the second junction: one in the rostrodorsal medial prefrontal cortex (rd-mPFC), which also signalled less demanding initial choices, and another one in the lateral frontopolar cortex (lFPC), which was only engaged by demanding choices at the second junction. Notably, interactions of the rd-mPFC with the hippocampus, a region associated with memory, increased when planning required extensive deliberation and particularly when planning led to accurate choices. Our findings show how humans can rapidly formulate a plan in novel environments. More broadly, these data uncover potential neural mechanisms underlying how we make inferences about states beyond a current subjective state.

Human Inferences about Sequences: A Minimal Transition Probability Model

The brain constantly infers the causes of the inputs it receives and uses these inferences to generate statistical expectations about future observations. Experimental evidence for these expectations and their violations include explicit reports, sequential effects on reaction times, and mismatch or surprise signals recorded in electrophysiology and functional MRI. Here, we explore the hypothesis that the brain acts as a near-optimal inference device that constantly attempts to infer the time-varying matrix of transition probabilities between the stimuli it receives, even when those stimuli are in fact fully unpredictable. This parsimonious Bayesian model, with a single free parameter, accounts for a broad range of findings on surprise signals, sequential effects and the perception of randomness. Notably, it explains the pervasive asymmetry between repetitions and alternations encountered in those studies. Our analysis suggests that a neural machinery for inferring transition probabilities lies at the core of human sequence knowledge.

Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex

The hippocampus and the orbitofrontal cortex (OFC) both have important roles in cognitive processes such as learning, memory and decision making. Nevertheless, research on the OFC and hippocampus has proceeded largely independently, and little consideration has been given to the importance of interactions between these structures. Here, evidence is reviewed that the hippocampus and OFC encode parallel, but interactive, cognitive 'maps' that capture complex relationships between cues, actions, outcomes and other features of the environment. A better understanding of the interactions between the OFC and hippocampus is important for understanding the neural bases of flexible, goal-directed decision making.