Reinforcement-learning in fronto-striatal circuits

Adaptive circuits for action and value information in rodent operant learning

Operant learning is a behavioral paradigm where animals learn to associate their actions with consequences, adapting their behavior accordingly. This review delves into the neural circuits that underpin operant learning in rodents, emphasizing the dynamic interplay between neural pathways, synaptic plasticity, and gene expression changes. We explore the cortico-basal ganglia circuits, highlighting the pivotal role of dopamine in modulating these pathways to reinforce behaviors that yield positive outcomes. We include insights from recent studies, which reveals the intricate roles of midbrain dopamine neurons in integrating action initiation and reward feedback, thereby enhancing movement-related activities in the dorsal striatum. Additionally, we discuss the molecular diversity of striatal neurons and their specific roles in reinforcement learning. The review also covers advances in transcriptome analysis techniques, such as single-cell RNA sequencing, which have provided deeper insights into the gene expression profiles associated with different neuronal populations during operant learning.

Reward expectation and receipt differentially modulate the spiking of accumbens D1+ and D2+ neurons

The nucleus accumbens (NAc) helps govern motivation to pursue reward. Two distinct sets of NAc projection neurons-expressing dopamine D1 vs. D2 receptors-are thought to promote and suppress motivated behaviors, respectively. However, support for this conceptual framework is limited: in particular, the spiking patterns of these distinct cell types during motivated behavior have been largely unknown. Using optogenetic tagging, we recorded the spiking of identified D1+ and D2+ neurons in the NAc core as unrestrained rats performed an operant task in which motivation to initiate work tracks recent reward rate. D1+ neurons preferentially increased firing as rats initiated trials and fired more when reward expectation was higher. By contrast, D2+ cells preferentially increased firing later in the trial, especially in response to reward delivery-a finding not anticipated from current theoretical models. Our results provide new evidence for the specific contribution of NAc D1+ cells to self-initiated approach behavior and will spur updated models of how D2+ cells contribute to learning.

Contractions in human cerebellar-cortical manifold structure underlie motor reinforcement learning

How the brain learns new motor commands through reinforcement involves distributed neural circuits beyond known frontal-striatal pathways, yet a comprehensive understanding of this broader neural architecture remains elusive. Here, using human functional MRI (N = 46, 27 females) and manifold learning techniques, we identified a low-dimensional neural space that captured the dynamic changes in whole-brain functional organization during a reward-based trajectory learning task. By quantifying participants' learning rates through an Actor-Critic model, we discovered that periods of accelerated learning were characterized by significant manifold contractions across multiple brain regions, including areas of limbic and hippocampal cortex, as well as the cerebellum. This contraction reflected enhanced network integration, with notably stronger connectivity between several of these regions and the sensorimotor cerebellum correlating with higher learning rates. These findings challenge the traditional view of the cerebellum as solely involved in error-based learning, supporting the emerging view that it coordinates with other brain regions during reinforcement learning.Significance Statement This study reveals how distributed brain systems, including the cerebellum and hippocampus, alter their functional connectivity to support motor learning through reinforcement. Using advanced manifold learning techniques on functional MRI data, we examined changes in regional connectivity during reward-based learning and their relationship to learning rate. For several brain regions, we found that periods of heightened learning were associated with increased cerebellar connectivity, suggesting a key role for the cerebellum in reward-based motor learning. These findings challenge the traditional view of the cerebellum as solely involved in supervised (error-based) learning and add to a growing rodent literature supporting a role for cerebellar circuits in reward-driven learning.

Neural dynamics of reversal learning in the prefrontal cortex and recurrent neural networks

In probabilistic reversal learning, the choice option yielding reward with higher probability switches at a random trial. To perform optimally in this task, one has to accumulate evidence across trials to infer the probability that a reversal has occurred. We investigated how this reversal probability is represented in cortical neurons by analyzing the neural activity in the prefrontal cortex of monkeys and recurrent neural networks trained on the task. We found that in a neural subspace encoding reversal probability, its activity represented integration of reward outcomes as in a line attractor model. The reversal probability activity at the start of a trial was stationary, stable and consistent with the attractor dynamics. However, during the trial, the activity was associated with task-related behavior and became non-stationary, thus deviating from the line attractor. Fitting a predictive model to neural data showed that the stationary state at the trial start serves as an initial condition for launching the non-stationary activity. This suggested an extension of the line attractor model with behavior-induced non-stationary dynamics. The non-stationary trajectories were separable indicating that they can represent distinct probabilistic values. Perturbing the reversal probability activity in the recurrent neural networks biased choice outcomes demonstrating its functional significance. In sum, our results show that cortical networks encode reversal probability in stable stationary state at the start of a trial and utilize it to initiate non-stationary dynamics that accommodates task-related behavior while maintaining the reversal information.

Neural Correlates and Sex-Specific Effects of Affectively Driven Processes Underlying Decision-Making in Adult ADHD

AIMS

Adults with attention-deficit/hyperactivity disorder (ADHD) often exhibit heightened risk-taking behavior due to disadvantageous decision-making. This study investigates the influence of preceding unconscious and affectively driven processes on this behavior, with a focus on sex-specific effects.

METHODS

Functional magnetic resonance imaging was used to examine neural activity during the anticipation phase of decision-making in 18 individuals with ADHD (10 females and 8 males) and 20 healthy controls (10 females and 10 males) using a modified version of the Balloon Analogue Risk Task.

RESULTS

During the anticipation of decision-making, individuals with ADHD exhibited reduced activation in the right precuneus and the right superior frontal gyrus compared to healthy controls. Sex-specific effects were exclusively observed within the ADHD group, showing increased neural activity in females compared to males in areas including the dorsolateral prefrontal cortex, left insula, right caudate, right cuneus, and precuneus.

CONCLUSION

These findings indicate altered neural activity in adult patients with ADHD with sex-specific differences during the anticipation of a risky decision. The study underscores the importance of the right precuneus and superior frontal gyrus in relation to metacognitive functioning and interoceptive awareness. However, further research is needed to explore the interplay of unconscious processes during decision-making in ADHD.

Prospective and retrospective representations of saccadic movements in primate prefrontal cortex

The dorso-lateral prefrontal cortex (dlPFC) contributes to flexible, goal-directed behaviors. However, a coherent picture of dlPFC function is lacking, as its activity is often studied only in relation to a few events within a fully learned behavioral task. Here we obtain a comprehensive description of dlPFC activity across different task epochs, saccade types, tasks, and learning stages. We consistently observe the strongest modulation of neural activity in relation to a retrospective representation of the most recent saccade. Prospective, planning-like activity is limited to task-related, delayed saccades directly eligible for a reward. The link between prospective and retrospective representations is highly structured, potentially reflecting a hard-wired feature of saccade responses. Only prospective representations are modulated by the recent behavioral history, but neither representation is modulated by day-to-day behavioral improvements. The dlPFC thus combines tightly linked flexible and rigid representations with a dominant contribution from retrospective signals maintaining the memory of past actions.

The hemispheric differences in prefrontal function of Internet game disorder and non-Internet game disorder: an activation likelihood estimation meta-analysis

This study explored the differences in brain activation between individuals with and without Internet gaming disorder (IGD) through activation likelihood estimation analysis. In total, 39 studies were included based on the inclusion and exclusion criteria by searching the literature in the PubMed and Web of Science databases, as well as reading other reviews. The analysis revealed that the activated brain regions in IGD were the right inferior frontal gyrus, left cingulate gyrus, and left lentiform nucleus. In comparison, the activated brain regions in non-IGD were the left middle frontal, left inferior frontal, left anterior cingulate, left precentral, and right precentral gyri. The results of the present study on differences in activation further confirm existing theoretical hypotheses. Future studies should explore hemispheric differences in prefrontal brain function between IGD and non-IGD.

Electrical brain activations in preadolescents during a probabilistic reward-learning task reflect cognitive processes and behavior strategies

Both adults and children learn through feedback to associate environmental events and choices with reward, a process known as reinforcement learning (RL). However, tasks to assess RL-related neurocognitive processes in children have been limited. This study validated a child version of the Probabilistic Reward Learning task in preadolescents (8-12 years) while recording event-related-potential (ERPs), focusing on: (1) reward-feedback sensitivity (frontal Reward-related Positivity, RewP), (2) late attention-related responses to feedback (parietal P300), and (3) attentional shifting toward favored stimuli (N2pc). Behaviorally, as expected, preadolescents could learn stimulus-reward outcome associations, but with varying performance levels. Poor learners showed greater RewP amplitudes compared to good learners. Learning strategies (i.e., Win-Lose-Stay-Shift) were reflected by feedback-elicited P300 amplitudes. Lastly, attention shifted toward to-be-chosen stimuli, as evidenced by the N2pc, but not toward more highly rewarded stimuli as in adults. These findings provide novel insights into the neural processes underlying RL in preadolescents.

Negative valuation of ambiguous feedback may predict near-term risk for suicide attempt in Veterans at high risk for suicide

Background

Learning from feedback - adapting behavior based on reinforcing and punishing outcomes - has been implicated in numerous psychiatric disorders, including substance misuse, post-traumatic stress disorder, and depression; an emerging literature suggests it may also play a role in suicidality. This study examined whether a feedback-based learning task with rewarding, punishing and ambiguous outcomes, followed by computational modeling, could improve near-term prospective prediction of suicide attempt in a high-risk sample.

Method

Veterans (N=60) at high-risk for suicide were tested on a task of reward- and punishment-based learning, at multiple sessions across a one-year period. Each session was coded according to whether the participant had (1) an actual suicide attempt (ASA); (2) another suicide-related event (OtherSE) such as suicidal behavior or suicidal ideation-related hospital admission (but not an ASA); or (3) neither (noSE) in the next 90 days. Computational modeling was used to estimate latent cognitive variables including learning rates from positive and negative outcomes, and the subjective value of ambiguous feedback.

Results

Optimal responding on the reward-based trials was positively associated with upcoming ASA, and remained predictive even after controlling for other standard clinical variables such as current suicidal ideation severity and prior suicide attempts. Computational modeling revealed that patients with upcoming ASA tended to view ambiguous outcomes as similar to weak punishment, while OtherSE and noSE both tended to view the ambiguous outcome as similar to weak reward. Differences in the reinforcement value of the neutral outcome remained predictive for ASA even after controlling for current suicidal ideation and prior suicide attempts.

Conclusion

A reinforcement learning task with ambiguous neutral outcomes may provide a useful tool to help predict near-term risk of ASA in at-risk patients. While most individuals interpret ambiguous feedback as mildly reinforcing (a "glass half full" interpretation), those with upcoming ASA tend to view it as mildly punishing (a "glass half empty" interpretation). While the current results are based on a very small sample with relatively few ASA events, and require replication in a larger sample, they provide support for the role of negative biases in feedback-based learning in the cognitive profile of suicide risk.

Distinct neural computations scale the violation of expected reward and emotion in social transgressions

Traditional decision-making models conceptualize humans as adaptive learners utilizing the differences between expected and actual rewards (prediction errors, PEs) to maximize outcomes, but rarely consider the influence of violations of emotional expectations (emotional PEs) and how it differs from reward PEs. Here, we conducted a fMRI experiment (n = 43) using a modified Ultimatum Game to examine how reward and emotional PEs affect punishment decisions in terms of rejecting unfair offers. Our results revealed that reward relative to emotional PEs exerted a stronger prediction to punishment decisions. On the neural level, the left dorsomedial prefrontal cortex (dmPFC) was strongly activated during reward receipt whereas the emotions engaged the bilateral anterior insula. Reward and emotional PEs were also encoded differently in brain-wide multivariate patterns, with a more sensitive neural signature observed within fronto-insular circuits for reward PE. We further identified a fronto-insular network encompassing the left anterior cingulate cortex, bilateral insula, left dmPFC and inferior frontal gyrus that encoded punishment decisions. In addition, a stronger fronto-insular pattern expression under reward PE predicted more punishment decisions. These findings underscore that reward and emotional violations interact to shape decisions in complex social interactions, while the underlying neurofunctional PEs computations are distinguishable.

Applications of Behavioral Economics and Neuroeconomics in Mental Health

Objective: The integration of behavioral economics and neuroeconomics into mental health offers innovative perspectives on understanding and addressing psychological disorders. This overview aims to synthesize current knowledge and explore the implications of these interdisciplinary approaches in the context of mental health. Method : In this narrative review, we summarized the current evidence regarding the applications of behavioral economics and neuroeconomics approaches in the field of mental health. Results: Behavioral economics and neuroeconomics provide valuable insights into the cognitive and emotional processes underlying mental health disorders, such as irrational decision-making, impulsivity, and self-control issues. Concepts such as loss aversion, temporal discounting, and framing effects inform the development of innovative interventions and policy initiatives. Behavioral economic interventions, including nudges, incentives, and commitment devices, show promise in promoting treatment adherence, reducing risky behaviors, and enhancing mental well-being. Neuroeconomics contributes by identifying neural markers predictive of treatment response and relapse risk, paving the way for personalized treatment approaches. Conclusion: The integration of behavioral economics and neuroeconomics into mental health research and practice holds significant potential for improving the understanding of psychological disorders and developing more effective, personalized interventions. Further research is needed to elucidate the mechanisms of action, optimize intervention strategies, and address ethical considerations associated with these approaches in mental health settings.

Environmental complexity modulates information processing and the balance between decision-making systems

Behavior in naturalistic scenarios occurs in diverse environments. Adaptive strategies rely on multiple neural circuits and competing decision systems. However, past studies of rodent decision making have largely measured behavior in simple environments. To fill this gap, we recorded neural ensembles from hippocampus (HC), dorsolateral striatum (DLS), and dorsomedial prefrontal cortex (dmPFC) while rats foraged for food under changing rules in environments with varying topological complexity. Environmental complexity increased behavioral variability, lengthened HC nonlocal sequences, and modulated action caching. We found contrasting representations between DLS and HC, supporting a competition between decision systems. dmPFC activity was indicative of setting this balance, in particular predicting the extent of HC non-local coding. Inactivating mPFC impaired short-term behavioral adaptation and produced long-term deficits in balancing decision systems. Our findings reveal the dynamic nature of decision-making systems and how environmental complexity modulates their engagement with implications for behavior in naturalistic environments.

Representational spaces in orbitofrontal and ventromedial prefrontal cortex: task states, values, and beyond

The orbitofrontal cortex (OFC) and ventromedial-prefrontal cortex (vmPFC) play a key role in decision-making and encode task states in addition to expected value. We review evidence suggesting a connection between value and state representations and argue that OFC / vmPFC integrate stimulus, context, and outcome information. Comparable encoding principles emerge in late layers of deep reinforcement learning (RL) models, where single nodes exhibit similar forms of mixed-selectivity, which enables flexible readout of relevant variables by downstream neurons. Based on these lines of evidence, we suggest that outcome-maximization leads to complex representational spaces that are insufficiently characterized by linear value signals that have been the focus of most prior research on the topic. Major outstanding questions concern the role of OFC/ vmPFC in learning across tasks, in encoding of task-irrelevant aspects, and the role of hippocampus-PFC interactions.

Dorsolateral prefrontal cortex drives strategic aborting by optimizing long-run policy extraction

Real world choices often involve balancing decisions that are optimized for the short-vs. long-term. Here, we reason that apparently sub-optimal single trial decisions in macaques may in fact reflect long-term, strategic planning. We demonstrate that macaques freely navigating in VR for sequentially presented targets will strategically abort offers, forgoing more immediate rewards on individual trials to maximize session-long returns. This behavior is highly specific to the individual, demonstrating that macaques reason about their own long-run performance. Reinforcement-learning (RL) models suggest this behavior is algorithmically supported by modular actor-critic networks with a policy module not only optimizing long-term value functions, but also informed of specific state-action values allowing for rapid policy optimization. The behavior of artificial networks suggests that changes in policy for a matched offer ought to be evident as soon as offers are made, even if the aborting behavior occurs much later. We confirm this prediction by demonstrating that single units and population dynamics in macaque dorsolateral prefrontal cortex (dlPFC), but not parietal area 7a or dorsomedial superior temporal area (MSTd), reflect the upcoming reward-maximizing aborting behavior upon offer presentation. These results cast dlPFC as a specialized policy module, and stand in contrast to recent work demonstrating the distributed and recurrent nature of belief-networks.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions, with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, the OFC leads the processing of symbolic RL in the ventral frontostriatal circuitry.

Using machine learning to determine a functional classifier of reward responsiveness and its association with adolescent psychiatric symptomatology

BACKGROUND

Machine learning (ML) has developed classifiers differentiating patient groups despite concerns regarding diagnostic reliability. An alternative strategy, used here, is to develop a functional classifier (hyperplane) (e.g. distinguishing the neural responses to received reward v. received punishment in typically developing (TD) adolescents) and then determine the functional integrity of the response (reward response distance from the hyperplane) in adolescents with externalizing and internalizing conditions and its associations with symptom clusters.

METHODS

Two hundred and ninety nine adolescents (mean age = 15.07 ± 2.30 years, 117 females) were divided into three groups: a training sample of TD adolescents where the Support Vector Machine (SVM) algorithm was applied (N = 65; 32 females), and two test groups- an independent sample of TD adolescents (N = 39; 14 females) and adolescents with a psychiatric diagnosis (major depressive disorder (MDD), generalized anxiety disorder (GAD), attention deficit hyperactivity disorder (ADHD) & conduct disorder (CD); N = 195, 71 females).

RESULTS

SVM ML analysis identified a hyperplane with accuracy = 80.77%, sensitivity = 78.38% and specificity = 88.99% that implicated feature neural regions associated with reward v. punishment (e.g. nucleus accumbens v. anterior insula cortices). Adolescents with externalizing diagnoses were significantly less likely to show a normative and significantly more likely to show a deficient reward response than the TD samples. Deficient reward response was associated with elevated CD, MDD, and ADHD symptoms.

CONCLUSIONS

Distinguishing the response to reward relative to punishment in TD adolescents via ML indicated notable disruptions in this response in patients with CD and ADHD and associations between reward responsiveness and CD, MDD, and ADHD symptom severity.

Childhood neglect is associated with alterations in neural prediction error signaling and the response to novelty

BACKGROUND

One in eight children experience early life stress (ELS), which increases risk for psychopathology. ELS, particularly neglect, has been associated with reduced responsivity to reward. However, little work has investigated the computational specifics of this disrupted reward response - particularly with respect to the neural response to Reward Prediction Errors (RPE) - a critical signal for successful instrumental learning - and the extent to which they are augmented to novel stimuli. The goal of the current study was to investigate the associations of abuse and neglect, and neural representation of RPE to novel and non-novel stimuli.

METHODS

One hundred and seventy-eight participants (aged 10-18, M = 14.9, s.d. = 2.38) engaged in the Novelty task while undergoing functional magnetic resonance imaging. In this task, participants learn to choose novel or non-novel stimuli to win monetary rewards varying from $0 to $0.30 per trial. Levels of abuse and neglect were measured using the Childhood Trauma Questionnaire.

RESULTS

Adolescents exposed to high levels of neglect showed reduced RPE-modulated blood oxygenation level dependent response within medial and lateral frontal cortices particularly when exploring novel stimuli (p < 0.05, corrected for multiple comparisons) relative to adolescents exposed to lower levels of neglect.

CONCLUSIONS

These data expand on previous work by indicating that neglect, but not abuse, is associated with impairments in neural RPE representation within medial and lateral frontal cortices. However, there was no association between neglect and behavioral impairments on the Novelty task, suggesting that these neural differences do not necessarily translate into behavioral differences within the context of the Novelty task.

Understanding dual process cognition via the minimum description length principle

Dual-process theories play a central role in both psychology and neuroscience, figuring prominently in domains ranging from executive control to reward-based learning to judgment and decision making. In each of these domains, two mechanisms appear to operate concurrently, one relatively high in computational complexity, the other relatively simple. Why is neural information processing organized in this way? We propose an answer to this question based on the notion of compression. The key insight is that dual-process structure can enhance adaptive behavior by allowing an agent to minimize the description length of its own behavior. We apply a single model based on this observation to findings from research on executive control, reward-based learning, and judgment and decision making, showing that seemingly diverse dual-process phenomena can be understood as domain-specific consequences of a single underlying set of computational principles.

Contribution of amygdala to dynamic model arbitration under uncertainty

Intrinsic uncertainty in the reward environment requires the brain to run multiple models simultaneously to predict outcomes based on preceding cues or actions, commonly referred to as stimulus- and action-based learning. Ultimately, the brain also must adopt appropriate choice behavior using reliability of these models. Here, we combined multiple experimental and computational approaches to quantify concurrent learning in monkeys performing tasks with different levels of uncertainty about the model of the environment. By comparing behavior in control monkeys and monkeys with bilateral lesions to the amygdala or ventral striatum, we found evidence for dynamic, competitive interaction between stimulus-based and action-based learning, and for a distinct role of the amygdala. Specifically, we demonstrate that the amygdala adjusts the initial balance between the two learning systems, thereby altering the interaction between arbitration and learning that shapes the time course of both learning and choice behaviors. This novel role of the amygdala can account for existing contradictory observations and provides testable predictions for future studies into circuit-level mechanisms of flexible learning and choice under uncertainty.

Timescales of learning in prefrontal cortex

The lateral prefrontal cortex (PFC) in humans and other primates is critical for immediate, goal-directed behaviour and working memory, which are classically considered distinct from the cognitive and neural circuits that support long-term learning and memory. Over the past few years, a reconsideration of this textbook perspective has emerged, in that different timescales of memory-guided behaviour are in constant interaction during the pursuit of immediate goals. Here, we will first detail how neural activity related to the shortest timescales of goal-directed behaviour (which requires maintenance of current states and goals in working memory) is sculpted by long-term knowledge and learning - that is, how the past informs present behaviour. Then, we will outline how learning across different timescales (from seconds to years) drives plasticity in the primate lateral PFC, from single neuron firing rates to mesoscale neuroimaging activity patterns. Finally, we will review how, over days and months of learning, dense local and long-range connectivity patterns in PFC facilitate longer-lasting changes in population activity by changing synaptic weights and recruiting additional neural resources to inform future behaviour. Our Review sheds light on how the machinery of plasticity in PFC circuits facilitates the integration of learned experiences across time to best guide adaptive behaviour.

A Reinforcement Learning approach to study climbing plant behaviour

A plant's structure is the result of constant adaptation and evolution to the surrounding environment. From this perspective, our goal is to investigate the mass and radius distribution of a particular plant organ, namely the searcher shoot, by providing a Reinforcement Learning (RL) environment, that we call Searcher-Shoot, which considers the mechanics due to the mass of the shoot and leaves. We uphold the hypothesis that plants maximize their length, avoiding a maximal stress threshold. To do this, we explore whether the mass distribution along the stem is efficient, formulating a Markov Decision Process. By exploiting this strategy, we are able to mimic and thus study the plant's behavior, finding that shoots decrease their diameters smoothly, resulting in an efficient distribution of the mass. The strong accordance between our results and the experimental data allows us to remark on the strength of our approach in the analysis of biological systems traits.

Thalamocortical architectures for flexible cognition and efficient learning

The brain exhibits a remarkable ability to learn and execute context-appropriate behaviors. How it achieves such flexibility, without sacrificing learning efficiency, is an important open question. Neuroscience, psychology, and engineering suggest that reusing and repurposing computations are part of the answer. Here, we review evidence that thalamocortical architectures may have evolved to facilitate these objectives of flexibility and efficiency by coordinating distributed computations. Recent work suggests that distributed prefrontal cortical networks compute with flexible codes, and that the mediodorsal thalamus provides regularization to promote efficient reuse. Thalamocortical interactions resemble hierarchical Bayesian computations, and their network implementation can be related to existing gating, synchronization, and hub theories of thalamic function. By reviewing recent findings and providing a novel synthesis, we highlight key research horizons integrating computation, cognition, and systems neuroscience.

Non-invasive stimulation of the human striatum disrupts reinforcement learning of motor skills

Reinforcement feedback can improve motor learning, but the underlying brain mechanisms remain underexplored. In particular, the causal contribution of specific patterns of oscillatory activity within the human striatum is unknown. To address this question, we exploited a recently developed non-invasive deep brain stimulation technique called transcranial temporal interference stimulation (tTIS) during reinforcement motor learning with concurrent neuroimaging, in a randomized, sham-controlled, double-blind study. Striatal tTIS applied at 80 Hz, but not at 20 Hz, abolished the benefits of reinforcement on motor learning. This effect was related to a selective modulation of neural activity within the striatum. Moreover, 80 Hz, but not 20 Hz, tTIS increased the neuromodulatory influence of the striatum on frontal areas involved in reinforcement motor learning. These results show that tTIS can non-invasively and selectively modulate a striatal mechanism involved in reinforcement learning, expanding our tools for the study of causal relationships between deep brain structures and human behaviour.

Electrical brain activations in preadolescents during a probabilistic reward-learning task reflect cognitive processes and behavioral strategy

Both adults and children learn through feedback which environmental events and choices are associated with higher probability of reward, an ability thought to be supported by the development of fronto-striatal reward circuits. Recent developmental studies have applied computational models of reward learning to investigate such learning in children. However, tasks and measures effective for assaying the cascade of reward-learning neural processes in children have been limited. Using a child-version of a probabilistic reward-learning task while recording event-related-potential (ERP) measures of electrical brain activity, this study examined key processes of reward learning in preadolescents (8-12 years old; n=30), namely: (1) reward-feedback sensitivity, as measured by the early-latency, reward-related, frontal ERP positivity, (2) rapid attentional shifting of processing toward favored visual stimuli, as measured by the N2pc component, and (3) longer-latency attention-related responses to reward feedback as a function of behavioral strategies (i.e., Win-Stay-Lose-Shift), as measured by the central-parietal P300. Consistent with our prior work in adults, the behavioral findings indicate preadolescents can learn stimulus-reward outcome associations, but at varying levels of performance. Neurally, poor preadolescent learners (those with slower learning rates) showed greater reward-related positivity amplitudes relative to good learners, suggesting greater reward-feedback sensitivity. We also found attention shifting towards to-be-chosen stimuli, as evidenced by the N2pc, but not to more highly rewarded stimuli as we have observed in adults. Lastly, we found the behavioral learning strategy (i.e., Win-Stay-Lose-Shift) reflected by the feedback-elicited parietal P300. These findings provide novel insights into the key neural processes underlying reinforcement learning in preadolescents.

Neural interactions in the human frontal cortex dissociate reward and punishment learning

How human prefrontal and insular regions interact while maximizing rewards and minimizing punishments is unknown. Capitalizing on human intracranial recordings, we demonstrate that the functional specificity toward reward or punishment learning is better disentangled by interactions compared to local representations. Prefrontal and insular cortices display non-selective neural populations to rewards and punishments. Non-selective responses, however, give rise to context-specific interareal interactions. We identify a reward subsystem with redundant interactions between the orbitofrontal and ventromedial prefrontal cortices, with a driving role of the latter. In addition, we find a punishment subsystem with redundant interactions between the insular and dorsolateral cortices, with a driving role of the insula. Finally, switching between reward and punishment learning is mediated by synergistic interactions between the two subsystems. These results provide a unifying explanation of distributed cortical representations and interactions supporting reward and punishment learning.

Reconfigurations of cortical manifold structure during reward-based motor learning

Adaptive motor behavior depends on the coordinated activity of multiple neural systems distributed across the brain. While the role of sensorimotor cortex in motor learning has been well established, how higher-order brain systems interact with sensorimotor cortex to guide learning is less well understood. Using functional MRI, we examined human brain activity during a reward-based motor task where subjects learned to shape their hand trajectories through reinforcement feedback. We projected patterns of cortical and striatal functional connectivity onto a low-dimensional manifold space and examined how regions expanded and contracted along the manifold during learning. During early learning, we found that several sensorimotor areas in the dorsal attention network exhibited increased covariance with areas of the salience/ventral attention network and reduced covariance with areas of the default mode network (DMN). During late learning, these effects reversed, with sensorimotor areas now exhibiting increased covariance with DMN areas. However, areas in posteromedial cortex showed the opposite pattern across learning phases, with its connectivity suggesting a role in coordinating activity across different networks over time. Our results establish the neural changes that support reward-based motor learning and identify distinct transitions in the functional coupling of sensorimotor to transmodal cortex when adapting behavior.

Cortical-striatal brain network distinguishes deepfake from real speaker identity

Deepfakes are viral ingredients of digital environments, and they can trick human cognition into misperceiving the fake as real. Here, we test the neurocognitive sensitivity of 25 participants to accept or reject person identities as recreated in audio deepfakes. We generate high-quality voice identity clones from natural speakers by using advanced deepfake technologies. During an identity matching task, participants show intermediate performance with deepfake voices, indicating levels of deception and resistance to deepfake identity spoofing. On the brain level, univariate and multivariate analyses consistently reveal a central cortico-striatal network that decoded the vocal acoustic pattern and deepfake-level (auditory cortex), as well as natural speaker identities (nucleus accumbens), which are valued for their social relevance. This network is embedded in a broader neural identity and object recognition network. Humans can thus be partly tricked by deepfakes, but the neurocognitive mechanisms identified during deepfake processing open windows for strengthening human resilience to fake information.

Involvement of the claustrum in the cortico-basal ganglia circuitry: connectional study in the non-human primate

The claustrum is an ancient telencephalic subcortical structure displaying extensive, reciprocal connections with much of the cortex and receiving projections from thalamus, amygdala, and hippocampus. This structure has a general role in modulating cortical excitability and is considered to be engaged in different cognitive and motor functions, such as sensory integration and perceptual binding, salience-guided attention, top-down executive functions, as well as in the control of brain states, such as sleep and its interhemispheric integration. The present study is the first to describe in detail a projection from the claustrum to the striatum in the macaque brain. Based on tracer injections in different striatal regions and in different cortical areas, we observed a rough topography of the claustral connectivity, thanks to which a claustral zone projects to both a specific striatal territory and to cortical areas involved in a network projecting to the same striatal territory. The present data add new elements of complexity of the basal ganglia information processing mode in motor and non-motor functions and provide evidence for an influence of the claustrum on both cortical functional domains and cortico-basal ganglia circuits.

Changes in the dopaminergic circuitry and adult neurogenesis linked to reinforcement learning in corvids

The caudolateral nidopallium (NCL, an analog of the prefrontal cortex) is known to be involved in learning, memory, and discrimination in corvids (a songbird), whereas the involvement of other brain regions in these phenomena is not well explored. We used house crows (Corvus splendens) to explore the neural correlates of learning and decision-making by initially training them on a shape discrimination task followed by immunohistochemistry to study the immediate early gene expression (Arc), a dopaminoceptive neuronal marker (DARPP-32, Dopamine- and cAMP-regulated phosphoprotein, Mr 32 kDa) to understand the involvement of the reward pathway and an immature neuronal marker (DCX, doublecortin) to detect learning-induced changes in adult neurogenesis. We performed neuronal counts and neuronal tracing, followed by morphometric analyses. Our present results have demonstrated that besides NCL, other parts of the caudal nidopallium (NC), avian basal ganglia, and intriguingly, vocal control regions in house crows are involved in visual discrimination. We have also found that training on the visual discrimination task can be correlated with neurite pruning in mature dopaminoceptive neurons and immature DCX-positive neurons in the NC of house crows. Furthermore, there is an increase in the incorporation of new neurons throughout NC and the medial striatum which can also be linked to learning. For the first time, our results demonstrate that a combination of structural changes in mature and immature neurons and adult neurogenesis are linked to learning in corvids.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and the mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, OFC leads the process of symbolic reinforcement learning in the ventral frontostriatal circuitry.

Ghrelin decreases sensitivity to negative feedback and increases prediction-error related caudate activity in humans, a randomized controlled trial

The stomach-derived hormone ghrelin plays not only a role in feeding, starvation, and survival, but it has been suggested to also be involved in the stress response, in neuropsychiatric conditions, and in alcohol and drug use disorders. Mechanisms related to reward processing might mediate ghrelin's broader effects on complex behaviors, as indicated by animal studies and mostly correlative human studies. Here, using a within-subject double-blind placebo-controlled design with intravenous ghrelin infusion in healthy volunteers (n = 30), we tested whether ghrelin alters sensitivity to reward and punishment in a reward learning task. Parameters were derived from a computational model of participants' task behavior. The reversal learning task with monetary rewards was performed during functional brain imaging to investigate ghrelin effects on brain signals related to reward prediction errors. Compared to placebo, ghrelin decreased punishment sensitivity (t = -2.448, p = 0.021), while reward sensitivity was unaltered (t = 0.8, p = 0.43). We furthermore found increased prediction-error related activity in the dorsal striatum during ghrelin administration (region of interest analysis: t-values ≥ 4.21, p-values ≤ 0.044). Our results support a role for ghrelin in reward processing that extends beyond food-related rewards. Reduced sensitivity to negative outcomes and increased processing of prediction errors may be beneficial for food foraging when hungry but could also relate to increased risk taking and impulsivity in the broader context of addictive behaviors.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Neuromodulation of Eating Disorders: A Review of Underlying Neural Network Activity and Neuromodulatory Treatments

Eating disorders are a group of psychiatric conditions that involve pathological relationships between patients and food. The most prolific of these disorders are anorexia nervosa, bulimia nervosa, and binge eating disorder. The current standard of care involves psychotherapy, pharmacotherapy, and the management of comorbid conditions, with nutritional rehabilitation reserved for severe cases of anorexia nervosa. Unfortunately, many patients often fail to respond, leaving a concerning treatment gap between the current and requisite treatments for eating disorders. To better understand the neurobiology underlying these eating disorders, investigations have been undertaken to characterize the activity of various neural networks, primarily those activated during tasks of executive inhibition, reward processing, and self-reference. Various neuromodulatory techniques have been proposed to stimulate these networks with the goal of improving patients' BMI and mental health. The aim of this review is to compile a comprehensive summarization of the current literature regarding the underlying neural connectivity of anorexia nervosa, bulimia nervosa, and binge eating disorder as well as the numerous neuromodulatory modalities that have been investigated. Importantly, we aimed to summarize the most significant clinical trials to date as well as to provide an updated assessment of the role of deep brain stimulation, summarizing numerous recently published clinical studies that have greatly contributed to the literature. In this review, we found therapeutic evidence for transcranial magnetic stimulation and transcranial direct current stimulation in treating individuals suffering from anorexia nervosa, bulimia nervosa, and binge eating disorder. We also found significant evidence for the role of deep brain stimulation, particularly as an escalatory therapy option for the those who failed standard therapy. Finally, we hope to provide promising directions for future clinical investigations.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

Changes in Behavior and Neural Dynamics across Adolescent Development

Adolescence is an important developmental period, during which substantial changes occur in brain function and behavior. Several aspects of executive function, including response inhibition, improve during this period. Correspondingly, structural imaging studies have documented consistent decreases in cortical and subcortical gray matter volume, and postmortem histologic studies have found substantial (∼40%) decreases in excitatory synapses in prefrontal cortex. Recent computational modeling work suggests that the change in synaptic density underlie improvements in task performance. These models also predict changes in neural dynamics related to the depth of attractor basins, where deeper basins can underlie better task performance. In this study, we analyzed task-related neural dynamics in a large cohort of longitudinally followed subjects (male and female) spanning early to late adolescence. We found that age correlated positively with behavioral performance in the Eriksen Flanker task. Older subjects were also characterized by deeper attractor basins around task related evoked EEG potentials during specific cognitive operations. Thus, consistent with computational models examining the effects of excitatory synaptic pruning, older adolescents showed stronger attractor dynamics during task performance.SIGNIFICANCE STATEMENT There are well-documented changes in brain and behavior during adolescent development. However, there are few mechanistic theories that link changes in the brain to changes in behavior. Here, we tested a hypothesis, put forward on the basis of computational modeling, that pruning of excitatory synapses in cortex during adolescence changes neural dynamics. We found, consistent with the hypothesis, that variability around event-related potentials shows faster decay dynamics in older adolescent subjects. The faster decay dynamics are consistent with the hypothesis that synaptic pruning during adolescent development leads to stronger attractor basins in task-related neural activity.

Latent profiles of substance use, early life stress, and attention/externalizing problems and their association with neural correlates of reinforcement learning in adolescents

BACKGROUND

Adolescent substance use, externalizing and attention problems, and early life stress (ELS) commonly co-occur. These psychopathologies show overlapping neural dysfunction in the form of reduced recruitment of reward processing neuro-circuitries. However, it is unclear to what extent these psychopathologies show common v. different neural dysfunctions as a function of symptom profiles, as no studies have directly compared neural dysfunctions associated with each of these psychopathologies to each other.

METHODS

In study 1, a latent profile analysis (LPA) was conducted in a sample of 266 adolescents (aged 13-18, 41.7% female, 58.3% male) from a residential youth care facility and the surrounding community to investigate substance use, externalizing and attention problems, and ELS psychopathologies and their co-presentation. In study 2, we examined a subsample of 174 participants who completed the Passive Avoidance learning task during functional magnetic resonance imaging to examine differential and/or common reward processing neuro-circuitry dysfunctions associated with symptom profiles based on these co-presentations.

RESULTS

In study 1, LPA identified profiles of substance use plus rule-breaking behaviors, attention-deficit hyperactivity disorder, and ELS. In study 2, the substance use/rule-breaking profile was associated with reduced recruitment of reward processing and attentional neuro-circuitries during the Passive Avoidance task (p < 0.05, corrected for multiple comparisons).

CONCLUSIONS

Findings indicate that there is reduced responsivity of striato-cortical regions when receiving outcomes on an instrumental learning task within a profile of adolescents with substance use and rule-breaking behaviors. Mitigating reward processing dysfunction specifically may represent a potential intervention target for substance-use psychopathologies accompanied by rule-breaking behaviors.

Perceived financial well-being and its association with frontostriatal functional connectivity, real-life anticipatory experiences, and everyday happiness

Perceived financial well-being (FWB) is an important aspect of life that can affect one's attitude toward future experiences and happiness. However, the relationship between FWB, anticipatory experiences, and happiness, and the brain's functional architecture underlying this relationship remain unknown. Here, we combined an experience sampling method, multilevel modeling, and functional neuroimaging to identify the neural correlates of FWB and their associations with real-world anticipatory experiences and everyday happiness. Behaviorally, we found that individuals with greater FWB felt more positive and more interested when they expected positive events to occur, which in turn resulted in increased everyday happiness. Furthermore, the level of FWB was significantly associated with the strength of functional connectivity (FC) between the nucleus accumbens (NAc) and ventromedial prefrontal cortex (vmPFC) and the local coherence within the vmPFC. The frontostriatal FC and local coherence within the vmPFC were further predictive of everyday happiness via the anticipatory response involving interestedness during positive expectations. Our findings suggest that individual differences in FWB could be reflected in the functional architecture of brain's reward system that may contribute to shaping positive anticipatory experiences and happiness in daily life.

Uncovering the neurophysiology of mood, motivation and behavioral symptoms in Parkinson's disease through intracranial recordings

Neuropsychiatric mood and motivation symptoms (depression, anxiety, apathy, impulse control disorders) in Parkinson's disease (PD) are highly disabling, difficult to treat and exacerbated by current medications and deep brain stimulation therapies. High-resolution intracranial recording techniques have the potential to undercover the network dysfunction and cognitive processes that drive these symptoms, towards a principled re-tuning of circuits. We highlight intracranial recording as a valuable tool for mapping and desegregating neural networks and their contribution to mood, motivation and behavioral symptoms, via the ability to dissect multiplexed overlapping spatial and temporal neural components. This technique can be powerfully combined with behavioral paradigms and emerging computational techniques to model underlying latent behavioral states. We review the literature of intracranial recording studies investigating mood, motivation and behavioral symptomatology with reference to 1) emotional processing, 2) executive control 3) subjective valuation (reward & cost evaluation) 4) motor control and 5) learning and updating. This reveals associations between different frequency specific network activities and underlying cognitive processes of reward decision making and action control. If validated, these signals represent potential computational biomarkers of motivational and behavioural states and could lead to principled therapy development for mood, motivation and behavioral symptoms in PD.

The motivational role of the ventral striatum and amygdala in learning from gains and losses

The ventral striatum (VS) and amygdala are two structures often implicated as essential structures for learning. The literature addressing the contribution of these areas to learning, however, is not entirely consistent. We propose that these inconsistencies are due to learning environments and the effect they have on motivation. To differentiate aspects of learning from environmental factors that affect motivation, we ran a series of experiments with varying task factors. We compared monkeys (Macaca mulatta) with VS lesions, amygdala lesions, and unoperated controls on reinforcement learning (RL) tasks that involve learning from both gains and losses as well as from deterministic and stochastic schedules of reinforcement. We found that for all three groups, performance varied by experiment. All three groups modulated their behavior in the same directions, to varying degrees, across the three experiments. This behavioral modulation is why we find deficits in some experiments, but not others. The amount of effort animals exhibited differed depending on the learning environment. Our results suggest that the VS is important for the amount of effort animals will give in rich deterministic and relatively leaner stochastic learning enivornments. We also showed that monkeys with amygdala lesions can learn stimulus-based RL in stochastic environments and environments with loss and conditioned reinforcers. These results show that learning environments shape motivation and that the VS is essential for distinct aspects of motivated behavior. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Combining noninvasive brain stimulation with behavioral pharmacology methods to study mechanisms of substance use disorder

Psychotropic drugs and transcranial magnetic stimulation (TMS) are effective for treating certain psychiatric conditions. Drugs and TMS have also been used as tools to explore the relationship between brain function and behavior in humans. Combining centrally acting drugs and TMS has proven useful for characterizing the neural basis of movement. This combined intervention approach also holds promise for improving our understanding of the mechanisms underlying disordered behavior associated with psychiatric conditions, including addiction, though challenges exist. For example, altered neocortical function has been implicated in substance use disorder, but the relationship between acute neuromodulation of neocortex with TMS and direct effects on addiction-related behaviors is not well established. We propose that the combination of human behavioral pharmacology methods with TMS can be leveraged to help establish these links. This perspective article describes an ongoing study that combines the administration of delta-9-tetrahydrocannabinol (THC), the main psychoactive compound in cannabis, with neuroimaging-guided TMS in individuals with problematic cannabis use. The study examines the impact of the left dorsolateral prefrontal cortex (DLPFC) stimulation on cognitive outcomes impacted by THC intoxication, including the subjective response to THC and the impairing effects of THC on behavioral performance. A framework for integrating TMS with human behavioral pharmacology methods, along with key details of the study design, are presented. We also discuss challenges, alternatives, and future directions.

Causal role of the dorsolateral prefrontal cortex in modulating the balance between Pavlovian and instrumental systems in the punishment domain

Previous literature suggests that a balance between Pavlovian and instrumental decision-making systems is critical for optimal decision-making. Pavlovian bias (i.e., approach toward reward-predictive stimuli and avoid punishment-predictive stimuli) often contrasts with the instrumental response. Although recent neuroimaging studies have identified brain regions that may be related to Pavlovian bias, including the dorsolateral prefrontal cortex (dlPFC), it is unclear whether a causal relationship exists. Therefore, we investigated whether upregulation of the dlPFC using transcranial current direct stimulation (tDCS) would reduce Pavlovian bias. In this double-blind study, participants were assigned to the anodal or the sham group; they received stimulation over the right dlPFC for 3 successive days. On the last day, participants performed a reinforcement learning task known as the orthogonalized go/no-go task; this was used to assess each participant's degree of Pavlovian bias in reward and punishment domains. We used computational modeling and hierarchical Bayesian analysis to estimate model parameters reflecting latent cognitive processes, including Pavlovian bias, go bias, and choice randomness. Several computational models were compared; the model with separate Pavlovian bias parameters for reward and punishment domains demonstrated the best model fit. When using a behavioral index of Pavlovian bias, the anodal group showed significantly lower Pavlovian bias in the punishment domain, but not in the reward domain, compared with the sham group. In addition, computational modeling showed that Pavlovian bias parameter in the punishment domain was lower in the anodal group than in the sham group, which is consistent with the behavioral findings. The anodal group also showed a lower go bias and choice randomness, compared with the sham group. These findings suggest that anodal tDCS may lead to behavioral suppression or change in Pavlovian bias in the punishment domain, which will help to improve comprehension of the causal neural mechanism.

Neurons in human pre-supplementary motor area encode key computations for value-based choice

Adaptive behaviour in real-world environments requires that choices integrate several variables, including the novelty of the options under consideration, their expected value and uncertainty in value estimation. Here, to probe how integration over decision variables occurs during decision-making, we recorded neurons from the human pre-supplementary motor area (preSMA), ventromedial prefrontal cortex and dorsal anterior cingulate. Unlike the other areas, preSMA neurons not only represented separate pre-decision variables for each choice option but also encoded an integrated utility signal for each choice option and, subsequently, the decision itself. Post-decision encoding of variables for the chosen option was more widely distributed and especially prominent in the ventromedial prefrontal cortex. Our findings position the human preSMA as central to the implementation of value-based decisions.

Enhanced putamen functional connectivity underlies altered risky decision-making in age-related cognitive decline

Risky decision-making is critical to survival and development, which has been compromised in elderly populations. However, the neural substrates of altered financial risk-taking behavior in aging are still under-investigated. Here we examined the intrinsic putamen network in modulating risk-taking behaviors of Balloon Analogue Risk Task in healthy young and older adults using resting-state fMRI. Compared with the young group, the elderly group showed significantly different task performance. Based on the task performance, older adults were further subdivided into two subgroups, showing young-like and over-conservative risk behaviors, regardless of cognitive decline. Compared with young adults, the intrinsic pattern of putamen connectivity was significantly different in over-conservative older adults, but not in young-like older adults. Notably, age-effects on risk behaviors were mediated via the putamen functional connectivity. In addition, the putamen gray matter volume showed significantly different relationships with risk behaviors and functional connectivity in over-conservative older adults. Our findings suggest that reward-based risky behaviors might be a sensitive indicator of brain aging, highlighting the critical role of the putamen network in maintaining optimal risky decision-making in age-related cognitive decline.

Angiotensin blockade enhances motivational reward learning via enhancing striatal prediction error signaling and frontostriatal communication

Adaptive human learning utilizes reward prediction errors (RPEs) that scale the differences between expected and actual outcomes to optimize future choices. Depression has been linked with biased RPE signaling and an exaggerated impact of negative outcomes on learning which may promote amotivation and anhedonia. The present proof-of-concept study combined computational modeling and multivariate decoding with neuroimaging to determine the influence of the selective competitive angiotensin II type 1 receptor antagonist losartan on learning from positive or negative outcomes and the underlying neural mechanisms in healthy humans. In a double-blind, between-subjects, placebo-controlled pharmaco-fMRI experiment, 61 healthy male participants (losartan, n = 30; placebo, n = 31) underwent a probabilistic selection reinforcement learning task incorporating a learning and transfer phase. Losartan improved choice accuracy for the hardest stimulus pair via increasing expected value sensitivity towards the rewarding stimulus relative to the placebo group during learning. Computational modeling revealed that losartan reduced the learning rate for negative outcomes and increased exploitatory choice behaviors while preserving learning for positive outcomes. These behavioral patterns were paralleled on the neural level by increased RPE signaling in orbitofrontal-striatal regions and enhanced positive outcome representations in the ventral striatum (VS) following losartan. In the transfer phase, losartan accelerated response times and enhanced VS functional connectivity with left dorsolateral prefrontal cortex when approaching maximum rewards. These findings elucidate the potential of losartan to reduce the impact of negative outcomes during learning and subsequently facilitate motivational approach towards maximum rewards in the transfer of learning. This may indicate a promising therapeutic mechanism to normalize distorted reward learning and fronto-striatal functioning in depression.

Functional connectivity abnormalities of brain networks in obsessive–compulsive disorder: a systematic review

Obsessive-compulsive disorder (OCD) is characterized by cognitive abnormalities encompassing several executive processes. Neuroimaging studies highlight functional abnormalities of executive fronto-parietal network (FPN) and default-mode network (DMN) in OCD patients, as well as of the prefrontal cortex (PFC) more specifically. We aim at assessing the presence of functional connectivity (FC) abnormalities of intrinsic brain networks and PFC in OCD, possibly underlying specific computational impairments and clinical manifestations. A systematic review of resting-state fMRI studies investigating FC was conducted in unmedicated OCD patients by querying three scientific databases (PubMed, Scopus, PsycInfo) up to July 2022 (search terms: “obsessive–compulsive disorder” AND “resting state” AND “fMRI” AND “function* *connect*” AND “task-positive” OR “executive” OR “central executive” OR “executive control” OR “executive-control” OR “cognitive control” OR “attenti*” OR “dorsal attention” OR “ventral attention” OR “frontoparietal” OR “fronto-parietal” OR “default mode” AND “network*” OR “system*”). Collectively, 20 studies were included. A predominantly reduced FC of DMN – often related to increased symptom severity – emerged. Additionally, intra-network FC of FPN was predominantly increased and often positively related to clinical scores. Concerning PFC, a predominant hyper-connectivity of right-sided prefrontal links emerged. Finally, FC of lateral prefrontal areas correlated with specific symptom dimensions. Several sources of heterogeneity in methodology might have affected results in unpredictable ways and were discussed. Such findings might represent endophenotypes of OCD manifestations, possibly reflecting computational impairments and difficulties in engaging in self-referential processes or in disengaging from cognitive control and monitoring processes.

The thalamus in psychosis spectrum disorder

Psychosis spectrum disorder (PSD) affects 1% of the world population and results in a lifetime of chronic disability, causing devastating personal and economic consequences. Developing new treatments for PSD remains a challenge, particularly those that target its core cognitive deficits. A key barrier to progress is the tenuous link between the basic neurobiological understanding of PSD and its clinical phenomenology. In this perspective, we focus on a key opportunity that combines innovations in non-invasive human neuroimaging with basic insights into thalamic regulation of functional cortical connectivity. The thalamus is an evolutionary conserved region that forms forebrain-wide functional loops critical for the transmission of external inputs as well as the construction and update of internal models. We discuss our perspective across four lines of evidence: First, we articulate how PSD symptomatology may arise from a faulty network organization at the macroscopic circuit level with the thalamus playing a central coordinating role. Second, we discuss how recent animal work has mechanistically clarified the properties of thalamic circuits relevant to regulating cortical dynamics and cognitive function more generally. Third, we present human neuroimaging evidence in support of thalamic alterations in PSD, and propose that a similar "thalamocortical dysconnectivity" seen in pharmacological imaging (under ketamine, LSD and THC) in healthy individuals may link this circuit phenotype to the common set of symptoms in idiopathic and drug-induced psychosis. Lastly, we synthesize animal and human work, and lay out a translational path for biomarker and therapeutic development.

Emotions as computations

Emotions ubiquitously impact action, learning, and perception, yet their essence and role remain widely debated. Computational accounts of emotion aspire to answer these questions with greater conceptual precision informed by normative principles and neurobiological data. We examine recent progress in this regard and find that emotions may implement three classes of computations, which serve to evaluate states, actions, and uncertain prospects. For each of these, we use the formalism of reinforcement learning to offer a new formulation that better accounts for existing evidence. We then consider how these distinct computations may map onto distinct emotions and moods. Integrating extensive research on the causes and consequences of different emotions suggests a parsimonious one-to-one mapping, according to which emotions are integral to how we evaluate outcomes (pleasure & pain), learn to predict them (happiness & sadness), use them to inform our (frustration & content) and others' (anger & gratitude) actions, and plan in order to realize (desire & hope) or avoid (fear & anxiety) uncertain outcomes.

The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors

The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.