Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior

The parietal association cortex and its projections to the dorsal striatum are involved in histaminergic and nonhistaminergic itch processing

Itch is an unpleasant sensation accompanied by the urge to scratch. The act of scratching not only alleviates the itch but also activates the reward circuitry, inducing a pleasurable sensation that can perpetuate further scratching. Therefore, scratching can be characterized as both a goal-directed behavior and a reward-motivated behavior. As a key hub for sensorymotor integration and information processing in goal-directed tasks, the specific role of the posterior parietal cortex (PPC) in modulating itch remains to be elucidated. Using immunofluorescence and calcium-signal fiber photometric recordings, we found that neurons in the parietal association cortex (PtA), a subregion of the PPC, were activated during acute itch. Pharmacogenetic experiments demonstrated that both nonselective inhibition of neurons in the PtA and selective inhibition of pyramidal neuron activity in the PtA reduced the experimental itch-scratching behavior induced by subcutaneous injections of 5-HT and compound 48/80. The PtA projects to the dorsal striatum (DS), a critical component of the brain's reward circuitry, and inhibition of this pathway also diminished experimental itch-scratching behavior. Therefore, this study demonstrated for the first time that the PtA may be involved in the regulation of the goal-directed behavior of scratching of histaminergic and nonhistaminergic itch through projections to the DS.

Genetic changes linked to two different syndromic forms of autism enhance reinforcement learning in adolescent male but not female mice

Autism Spectrum Disorder (ASD) is characterized by restricted and repetitive behaviors and social differences, both of which may manifest, in part, from underlying differences in corticostriatal circuits and reinforcement learning. Here, we investigated reinforcement learning in mice with mutations in either Tsc2 or Shank3, both high-confidence ASD risk genes associated with major syndromic forms of ASD. Using an odor-based two-alternative forced choice (2AFC) task, we tested adolescent mice of both sexes and found male Tsc2 and Shank3B heterozygote (Het) mice showed enhanced learning performance compared to their wild type (WT) siblings. No gain of function was observed in females. Using a novel reinforcement learning (RL) based computational model to infer learning rate as well as policy-level task engagement and disengagement, we found that the gain of function in males was driven by an enhanced positive learning rate in both Tsc2 and Shank3B Het mice. The gain of function in Het males was absent when mice were trained with a probabilistic reward schedule. These findings in two ASD mouse models reveal a convergent learning phenotype that shows similar sensitivity to sex and environmental uncertainty. These data can inform our understanding of both strengths and challenges associated with autism, while providing further evidence that sex and experience of uncertainty modulate autism-related phenotypes.

Significance Statement

Reinforcement learning is a foundational form of learning that is widely used in behavioral interventions for autism. Here, we measured reinforcement learning in adolescent mice carrying genetic mutations linked to two different syndromic forms of autism. We found that males showed strengths in reinforcement learning compared to their wild type siblings, while females showed no differences. This gain of function in males was no longer observed when uncertainty was introduced into the reward schedule for correct choices. These findings support a model in which diverse genetic changes interact with sex to generate common phenotypes underlying autism. Our data further support the idea that autism risk genes may produce strengths as well as challenges in behavioral function.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions, with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, the OFC leads the processing of symbolic RL in the ventral frontostriatal circuitry.

Contribution of amygdala to dynamic model arbitration under uncertainty

Intrinsic uncertainty in the reward environment requires the brain to run multiple models simultaneously to predict outcomes based on preceding cues or actions, commonly referred to as stimulus- and action-based learning. Ultimately, the brain also must adopt appropriate choice behavior using reliability of these models. Here, we combined multiple experimental and computational approaches to quantify concurrent learning in monkeys performing tasks with different levels of uncertainty about the model of the environment. By comparing behavior in control monkeys and monkeys with bilateral lesions to the amygdala or ventral striatum, we found evidence for dynamic, competitive interaction between stimulus-based and action-based learning, and for a distinct role of the amygdala. Specifically, we demonstrate that the amygdala adjusts the initial balance between the two learning systems, thereby altering the interaction between arbitration and learning that shapes the time course of both learning and choice behaviors. This novel role of the amygdala can account for existing contradictory observations and provides testable predictions for future studies into circuit-level mechanisms of flexible learning and choice under uncertainty.

Preferences reveal dissociable encoding across prefrontal-limbic circuits

Individual preferences for the flavor of different foods and fluids exert a strong influence on behavior. Most current theories posit that preferences are integrated with other state variables in the orbitofrontal cortex (OFC), which is thought to derive the relative subjective value of available options to guide choice behavior. Here, we report that instead of a single integrated valuation system in the OFC, another complementary one is centered in the ventrolateral prefrontal cortex (vlPFC) in macaques. Specifically, we found that the OFC and vlPFC preferentially represent outcome flavor and outcome probability, respectively, and that preferences are separately integrated into value representations in these areas. In addition, the vlPFC, but not the OFC, represented the probability of receiving the available outcome flavors separately, with the difference between these representations reflecting the degree of preference for each flavor. Thus, both the vlPFC and OFC exhibit dissociable but complementary representations of subjective value, both of which are necessary for decision-making.

Neural mechanisms of information seeking

We ubiquitously seek information to make better decisions. Particularly in the modern age, when more information is available at our fingertips than ever, the information we choose to collect determines the quality of our decisions. Decision neuroscience has long adopted empirical approaches where the information available to decision-makers is fully controlled by the researchers, leaving neural mechanisms of information seeking less understood. Although information seeking has long been studied in the context of the exploration-exploitation trade-off, recent studies have widened the scope to investigate more overt information seeking in a way distinct from other decision processes. Insights gained from these studies, accumulated over the last few years, raise the possibility that information seeking is driven by the reward system signaling the subjective value of information. In this piece, we review findings from the recent studies, highlighting the conceptual and empirical relationships between distinct literatures, and discuss future research directions necessary to establish a more comprehensive understanding of how individuals seek information as a part of value-based decision-making.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and the mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, OFC leads the process of symbolic reinforcement learning in the ventral frontostriatal circuitry.

Brain mechanism of foraging: Reward-dependent synaptic plasticity versus neural integration of values

During foraging behavior, action values are persistently encoded in neural activity and updated depending on the history of choice outcomes. What is the neural mechanism for action value maintenance and updating? Here, we explore two contrasting network models: synaptic learning of action value versus neural integration. We show that both models can reproduce extant experimental data, but they yield distinct predictions about the underlying biological neural circuits. In particular, the neural integrator model but not the synaptic model requires that reward signals are mediated by neural pools selective for action alternatives and their projections are aligned with linear attractor axes in the valuation system. We demonstrate experimentally observable neural dynamical signatures and feasible perturbations to differentiate the two contrasting scenarios, suggesting that the synaptic model is a more robust candidate mechanism. Overall, this work provides a modeling framework to guide future experimental research on probabilistic foraging.

Neurons in the monkey frontopolar cortex encode learning stage and goal during a fast learning task

The frontopolar cortex (FPC) is, to date, one of the least understood regions of the prefrontal cortex. The current understanding of its function suggests that it plays a role in the control of exploratory behaviors by coordinating the activities of other prefrontal cortex areas involved in decision-making and exploiting actions based on their outcomes. Based on this hypothesis, FPC would drive fast-learning processes through a valuation of the different alternatives. In our study, we used a modified version of a well-known paradigm, the object-in-place (OIP) task, to test this hypothesis in electrophysiology. This paradigm is designed to maximize learning, enabling monkeys to learn in one trial, which is an ability specifically impaired after a lesion of the FPC. We showed that FPC neurons presented an extremely specific pattern of activity by representing the learning stage, exploration versus exploitation, and the goal of the action. However, our results do not support the hypothesis that neurons in the frontal pole compute an evaluation of different alternatives. Indeed, the position of the chosen target was strongly encoded at its acquisition, but the position of the unchosen target was not. Once learned, this representation was also found at the problem presentation, suggesting a monitoring activity of the synthetic goal preceding its acquisition. Our results highlight important features of FPC neurons in fast-learning processes without confirming their role in the disengagement of cognitive control from the current goals.

Latent-state and model-based learning in PTSD

Post-traumatic stress disorder (PTSD) is characterized by altered emotional and behavioral responding following a traumatic event. In this article, we review the concepts of latent-state and model-based learning (i.e., learning and inferring abstract task representations) and discuss their relevance for clinical and neuroscience models of PTSD. Recent data demonstrate evidence for brain and behavioral biases in these learning processes in PTSD. These new data potentially recast excessive fear towards trauma cues as a problem in learning and updating abstract task representations, as opposed to traditional conceptualizations focused on stimulus-specific learning. Biases in latent-state and model-based learning may also be a common mechanism targeted in common therapies for PTSD. We highlight key knowledge gaps that need to be addressed to further elaborate how latent-state learning and its associated neurocircuitry mechanisms function in PTSD and how to optimize treatments to target these processes.

Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

Electrophysiological Markers of Aberrant Cue-Specific Exploration in Hazardous Drinkers

Background

Hazardous drinking is associated with maladaptive alcohol-related decision-making. Existing studies have often focused on how participants learn to exploit familiar cues based on prior reinforcement, but little is known about the mechanisms that drive hazardous drinkers to explore novel alcohol cues when their value is not known.

Methods

We investigated exploration of novel alcohol and non-alcohol cues in hazardous drinkers (N = 27) and control participants (N = 26) during electroencephalography (EEG). A normative computational model with two free parameters was fit to estimate participants' weighting of the future value of exploration and immediate value of exploitation.

Results

Hazardous drinkers demonstrated increased exploration of novel alcohol cues, and conversely, increased probability of exploiting familiar alternatives instead of exploring novel non-alcohol cues. The motivation to explore novel alcohol stimuli in hazardous drinkers was driven by an elevated relative future valuation of uncertain alcohol cues. P3a predicted more exploratory decision policies driven by an enhanced relative future valuation of novel alcohol cues. P3b did not predict choice behavior, but computational parameter estimates suggested that hazardous drinkers with enhanced P3b to alcohol cues were likely to learn to exploit their immediate expected value.

Conclusions

Hazardous drinkers did not display atypical choice behavior, different P3a/P3b amplitudes, or computational estimates to novel non-alcohol cues-diverging from previous studies in addiction showing atypical generalized explore-exploit decisions with non-drug-related cues. These findings reveal that cue-specific neural computations may drive aberrant alcohol-related decision-making in hazardous drinkers-highlighting the importance of drug-relevant cues in studies of decision-making in addiction.

Pathways to the persistence of drug use despite its adverse consequences

The persistence of drug taking despite its adverse consequences plays a central role in the presentation, diagnosis, and impacts of addiction. Eventual recognition and appraisal of these adverse consequences is central to decisions to reduce or cease use. However, the most appropriate ways of conceptualizing persistence in the face of adverse consequences remain unclear. Here we review evidence that there are at least three pathways to persistent use despite the negative consequences of that use. A cognitive pathway for recognition of adverse consequences, a motivational pathway for valuation of these consequences, and a behavioral pathway for responding to these adverse consequences. These pathways are dynamic, not linear, with multiple possible trajectories between them, and each is sufficient to produce persistence. We describe these pathways, their characteristics, brain cellular and circuit substrates, and we highlight their relevance to different pathways to self- and treatment-guided behavior change.

What Does the Frontopolar Cortex Contribute to Goal-Directed Cognition and Action?

Understanding the unique functions of different subregions of primate prefrontal cortex has been a longstanding goal in cognitive neuroscience. Yet, the anatomy and function of one of its largest subregions (the frontopolar cortex) remain enigmatic and underspecified. Our Society for Neuroscience minisymposium Primate Frontopolar Cortex: From Circuits to Complex Behaviors will comprise a range of new anatomic and functional approaches that have helped to clarify the basic circuit anatomy of the frontal pole, its functional involvement during performance of cognitively demanding behavioral paradigms in monkeys and humans, and its clinical potential as a target for noninvasive brain stimulation in patients with brain disorders. This review consolidates knowledge about the anatomy and connectivity of frontopolar cortex and provides an integrative summary of its function in primates. We aim to answer the question: what, if anything, does frontopolar cortex contribute to goal-directed cognition and action?

The effect of approach bias modification during alcohol withdrawal treatment on craving, and its relationship to post-treatment alcohol use in a randomised controlled trial

BACKGROUND

Approach bias modification (ApBM) for alcohol use disorder helps prevent relapse, yet the psychological mechanisms underlying its efficacy remain unclear. Alcohol craving predicts relapse and appears to be related to the biased processing of alcohol stimuli which is reduced by ApBM. However, there is little research examining whether ApBM reduces alcohol craving.

METHODS

In a randomised controlled trial testing the effect of 4 ApBM sessions (vs. sham training) on post-treatment alcohol use in 300 alcohol withdrawal inpatients, we administered the Alcohol Craving Questionnaire - Short Form - Revised (ACQ-SF-R) pre and post-training and at 2-week, 3, 6 and 12-month follow ups; and a cue-induced craving measure pre and post training.

RESULTS

Groups did not significantly differ in terms of declines in ACQ-SF-R total scores (p = .712) or cue-induced craving (p = .841) between the first and last training session, nor in terms of ACQ-SF-R scores at follow-ups (p = .509). However, the ACQ-SF-R Expectancy subscale, which assesses craving based on anticipated positive reinforcement from alcohol, was significantly lower in the ApBM group than in controls following training (p = .030), although the group x time interaction for this subscale was non-significant (p = .062). Post-intervention Expectancy scores mediated only a small portion of ApBM's effect on post-discharge alcohol use (14% in intention-to-treat analysis, p = .046; 15% in per-protocol analysis, p = .020).

CONCLUSIONS

ApBM does not appear to have robust, sustained effects on alcohol craving. Reduced craving is unlikely to account for ApBM's relapse prevention effects. However, further research on whether ApBM's effects are related to devaluation of alcohol reward expectancy is warranted.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry Identifier: ACTRN12617001241325.

Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System

To effectively behave within ever-changing environments, biological agents must learn and act at varying hierarchical levels such that a complex task may be broken down into more tractable subtasks. Hierarchical reinforcement learning (HRL) is a computational framework that provides an understanding of this process by combining sequential actions into one temporally extended unit called an option. However, there are still open questions within the HRL framework, including how options are formed and how HRL mechanisms might be realized within the brain. In this review, we propose that the existing human motor sequence literature can aid in understanding both of these questions. We give specific emphasis to visuomotor sequence learning tasks such as the discrete sequence production task and the M × N (M steps × N sets) task to understand how hierarchical learning and behavior manifest across sequential action tasks as well as how the dorsal cortical-subcortical circuitry could support this kind of behavior. This review highlights how motor chunks within a motor sequence can function as HRL options. Furthermore, we aim to merge findings from motor sequence literature with reinforcement learning perspectives to inform experimental design in each respective subfield.

Shared mechanisms mediate the explore-exploit tradeoff in macaques and humans

Ancestors of macaques and humans separated into distinct lineages 25 million years ago. Despite this long separation, Hogeveen et al. (2022) show, in this issue of Neuron, that they mediate the explore-exploit tradeoff, which must be managed by any agent adapting to a dynamic environment, using similar computational and neural mechanisms.

The neurocomputational bases of explore-exploit decision-making

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.