Differential encoding of losses and gains in the human striatum

Temporally organized representations of reward and risk in the human brain

The value and uncertainty associated with choice alternatives constitute critical features relevant for decisions. However, the manner in which reward and risk representations are temporally organized in the brain remains elusive. Here we leverage the spatiotemporal precision of intracranial electroencephalography, along with a simple card game designed to elicit the unfolding computation of a set of reward and risk variables, to uncover this temporal organization. Reward outcome representations across wide-spread regions follow a sequential order along the anteroposterior axis of the brain. In contrast, expected value can be decoded from multiple regions at the same time, and error signals in both reward and risk domains reflect a mixture of sequential and parallel encoding. We further highlight the role of the anterior insula in generalizing between reward prediction error and risk prediction error codes. Together our results emphasize the importance of neural dynamics for understanding value-based decisions under uncertainty.

Posterior parietal cortex is causally involved in reward valuation but not in probability weighting during risky choice

This study provides evidence that the posterior parietal cortex is causally involved in risky decision making via the processing of reward values but not reward probabilities. In the within-group experimental design, participants performed a binary lottery choice task following transcranial magnetic stimulation of the right posterior parietal cortex, left posterior parietal cortex, and a right posterior parietal cortex sham (placebo) stimulation. The continuous theta-burst stimulation protocol supposedly downregulating the cortical excitability was used. Both, mean-variance and the prospect theory approach to risky choice showed that the posterior parietal cortex stimulation shifted participants toward greater risk aversion compared with sham. On the behavioral level, after the posterior parietal cortex stimulation, the likelihood of choosing a safer option became more sensitive to the difference in standard deviations between lotteries, compared with sham, indicating greater risk avoidance within the mean-variance framework. We also estimated the shift in prospect theory parameters of risk preferences after posterior parietal cortex stimulation. The hierarchical Bayesian approach showed moderate evidence for a credible change in risk aversion parameter toward lower marginal reward value (and, hence, lower risk tolerance), while no credible change in probability weighting was observed. In addition, we observed anecdotal evidence for a credible increase in the consistency of responses after the left posterior parietal cortex stimulation compared with sham.

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

In the mammalian brain, midbrain dopamine neuron activity is hypothesized to encode reward prediction errors that promote learning and guide behavior by causing rapid changes in dopamine levels in target brain regions. This hypothesis (and alternatives regarding dopamine's role in punishment-learning) has limited direct evidence in humans. We report intracranial, subsecond measurements of dopamine release in human striatum measured, while volunteers (i.e., patients undergoing deep brain stimulation surgery) performed a probabilistic reward and punishment learning choice task designed to test whether dopamine release encodes only reward prediction errors or whether dopamine release may also encode adaptive punishment learning signals. Results demonstrate that extracellular dopamine levels can encode both reward and punishment prediction errors within distinct time intervals via independent valence-specific pathways in the human brain.

Differential effects of reward and punishment on reinforcement-based motor learning and generalization

Reward and punishment have long been recognized as potent modulators of human behavior. Although reinforcement learning is a significant motor learning process, the exact mechanisms underlying how the brain learns movements through reward and punishment are not yet fully understood. Beyond the memory of specific examples, investigating the ability to generalize to new situations offers a better understanding of motor learning. This study hypothesizes that reward and punishment engage qualitatively different motivational systems with different neurochemical and neuroanatomical substrates, which would have differential effects on reinforcement-based motor learning and generalization. To test this hypothesis, two groups of participants learn a motor task in one direction and then relearn the same task in a new direction, receiving only performance-based reward or punishment score feedback. Our findings support our hypothesis, showing that reward led to slower learning but promoted generalization. On the other hand, punishment led to faster learning but impaired generalization. These behavioral differences may be due to different tendencies of movement variability in each group. The punishment group tended to explore more actively than the reward group during the initial learning phase, possibly due to loss aversion. In contrast, the reward group tended to explore more actively than the initial learning phase during the generalization test phase, seemingly recalling the strategy that led to the reward. These results suggest that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization, with important implications for practical applications such as sports training and motor rehabilitation.NEW & NOTEWORTHY Although reinforcement learning is a significant motor learning process, the mechanisms underlying how the brain learns movements through reward and punishment are not fully understood. We modified a well-established motor adaptation task and used savings (faster relearning) to measure generalization. We found reward led to slower learning but promoted generalization, whereas punishment led to faster learning but impaired generalization, suggesting that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization.

Decision-making under ambiguity and risk and executive functions in Parkinson's disease patients: A scoping review of the studies investigating the Iowa Gambling Task and the Game of Dice

Evidence shows that patients affected by Parkinson's disease (PD) display the tendency toward making risky choices. This is due, at least in part, to the pathophysiological characteristics of the disease that affects neural areas underlying decision making (DM), in which a pivotal role is played by nonmotor corticostriatal circuits and dopamine. Executive functions (EFs), which can be impaired by PD as well, may sustain optimal choices in DM processes. However, few studies have investigated whether EFs can support PD patients to make good decisions. Adopting the scoping review approach, the present article is designed to deepen the cognitive mechanisms of DM under conditions of ambiguity and risk (that are conditions common to everyday life decisions) in PD patients without impulse control disorders. We focused our attention on the Iowa Gambling Task and the Game of Dice Task, because they are the most commonly used and reliable tasks to assess DM under ambiguity and under risk, respectively, and analyzed the performances in such tasks and their relationships with EFs tests in PD patients. The analysis supported the relationships between EFs and DM performance, especially when a higher cognitive load is required to make optimal decisions, as it happens under conditions of risk. Possible knowledge gaps and further research directions are suggested to better understand DM mechanisms in PD sustaining patients' cognitive functioning and preventing negative consequences in everyday life derived from suboptimal decisions.

Temporally organized representations of reward and risk in the human brain

The value and uncertainty associated with choice alternatives constitute critical features along which decisions are made. While the neural substrates supporting reward and risk processing have been investigated, the temporal organization by which these computations are encoded remains elusive. Here we leverage the high spatiotemporal precision of intracranial electroencephalography (iEEG) to uncover how representations of decision-related computations unfold in time. We present evidence of locally distributed representations of reward and risk variables that are temporally organized across multiple regions of interest. Reward outcome representations across wide-spread regions follow a temporally cascading order along the anteroposterior axis of the brain. In contrast, expected value can be decoded from multiple regions at the same time, and error signals in both reward and risk domains reflect a mixture of sequential and parallel encoding. We highlight the role of the anterior insula in generalizing between reward prediction error (RePE) and risk prediction error (RiPE), within which the encoding of RePE in the distributed iEEG signal predicts RiPE. Together our results emphasize the utility of uncovering temporal dynamics in the human brain for understanding how computational processes critical for value-based decisions under uncertainty unfold.

Valence-partitioned learning signals drive choice behavior and phenomenal subjective experience in humans

How the human brain generates conscious phenomenal experience is a fundamental problem. In particular, it is unknown how variable and dynamic changes in subjective affect are driven by interactions with objective phenomena. We hypothesize a neurocomputational mechanism that generates valence-specific learning signals associated with 'what it is like' to be rewarded or punished. Our hypothesized model maintains a partition between appetitive and aversive information while generating independent and parallel reward and punishment learning signals. This valence-partitioned reinforcement learning (VPRL) model and its associated learning signals are shown to predict dynamic changes in 1) human choice behavior, 2) phenomenal subjective experience, and 3) BOLD-imaging responses that implicate a network of regions that process appetitive and aversive information that converge on the ventral striatum and ventromedial prefrontal cortex during moments of introspection. Our results demonstrate the utility of valence-partitioned reinforcement learning as a neurocomputational basis for investigating mechanisms that may drive conscious experience.

Computational perspectives on human fear and anxiety

Fear and anxiety are adaptive emotions that serve important defensive functions, yet in excess, they can be debilitating and lead to poor mental health. Computational modelling of behaviour provides a mechanistic framework for understanding the cognitive and neurobiological bases of fear and anxiety, and has seen increasing interest in the field. In this brief review, we discuss recent developments in the computational modelling of human fear and anxiety. Firstly, we describe various reinforcement learning strategies that humans employ when learning to predict or avoid threat, and how these relate to symptoms of fear and anxiety. Secondly, we discuss initial efforts to explore, through a computational lens, approach-avoidance conflict paradigms that are popular in animal research to measure fear- and anxiety-relevant behaviours. Finally, we discuss negative biases in decision-making in the face of uncertainty in anxiety.

Behavioral and neural representation of expected reward and risk

When faced with uncertainty, individuals' value-based decisions are influenced by the expected rewards and risks. Understanding how reward and risk are processed and integrated at the behavioral and neural levels is essential for building up utility theories. Using a modified monetary incentive delay task in which the mean of two possible outcomes (expected reward) and the standard deviation (SD) of the possible outcomes (risk) were parametrically manipulated and orthogonalized, we measured eye movements, response times (RTs), and brain activity when participants seek to secure a reward. We found that RTs varied as a function of the mean but not the SD of the potential reward, suggesting that expected rewards are the main driver of RTs. Moreover, the difference between gazes focused on high vs. low value rewards became smaller when the magnitude of the potential reward (mean of possible outcomes) was larger and when risk (SD of possible outcomes) became smaller, highlighting that reward and risk have different effects on attention deployment. Processing the mean reward activated the striatum. The positive striatal connectivity to the amygdala and negative striatal connectivity to the superior frontal gyrus were correlated with individuals' sensitivity to the expected reward. In contrast, processing risk activated the anterior insula. Its positive connectivity to the ventromedial prefrontal cortex and negative connectivity to the anterior midcingulate cortex were correlated with individual differences in risk sensitivity, further suggesting the functional dissociation of reward and risk at the neural level. Our findings, based on several different measures, delineate the distinct representations of reward and risk in non-decision contexts and provide insight into how these utility parameters modulate attention, motivation, and brain networks.

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

Dorsolateral prefrontal cortex plays causal role in probability weighting during risky choice

In this study, we provide causal evidence that the dorsolateral prefrontal cortex (DLPFC) supports the computation of subjective value in choices under risk via its involvement in probability weighting. Following offline continuous theta-burst transcranial magnetic stimulation (cTBS) of the DLPFC subjects (N = 30, mean age 23.6, 56% females) completed a computerized task consisting of 96 binary lottery choice questions presented in random order. Using the hierarchical Bayesian modeling approach, we then estimated the structural parameters of risk preferences (the degree of risk aversion and the curvature of the probability weighting function) and analyzed the obtained posterior distributions to determine the effect of stimulation on model parameters. On a behavioral level, temporary downregulation of the left DLPFC excitability through cTBS decreased the likelihood of choosing an option with higher expected reward while the probability of choosing a riskier lottery did not significantly change. Modeling the stimulation effects on risk preference parameters showed anecdotal evidence as assessed by Bayes factors that probability weighting parameter increased after the left DLPFC TMS compared to sham.

A prospective investigation of youth alcohol experimentation and reward responsivity in the ABCD study

Rationale

Greater risk-taking behaviors, such as alcohol experimentation, are associated with different patterns of brain functioning in regions implicated in reward (nucleus accumbens, NA) and cognitive control (inferior frontal gyrus, IFG). These neural features have been observed in youth with greater risk-taking tendencies prior to substance use initiation, suggesting NA-IFG disruption may serve as an early marker for subsequent substance use disorders. Prospective studies are needed to determine if NA-IFG neural disruption predicts future substance use in school-age children, including those with minimal use of alcohol (e.g., sipping). The present large-sample prospective study sought to use machine learning to: (1) examine alcohol sipping at ages 9, 10 as a potential behavioral indicator of concurrent underlying altered neural responsivity to reward, and (2) determine if alcohol sipping and NA-IFG activation at ages 9, 10 can be used to predict which youth reported increased alcohol use at ages 11, 12. Additionally, low-level alcohol use and brain functioning at ages 9, 10 were examined as predictors of substance use and brain functioning at ages 11, 12.

Design and methods

This project used data from the baseline (Time 1) and two-year follow-up (Time 2) assessments of the Adolescent Brain Cognitive Development (ABCD) Study (Release 3.0). Support Vector Machine (SVM) learning determined if: (1) NA-IFG neural activity could correctly identify youth who reported alcohol sipping at Time 1 (n = 7409, mean age = 119.34 months, SD = 7.53; 50.27% female), and (2) NA-IFG and alcohol sipping frequency at Time 1 could correctly identify youth who reported drinking alcohol at Time 2 (n = 4000, mean age = 143.25 months, SD = 7.63; 47.53% female). Linear regression was also used to examine the relationship between alcohol sipping and NA-IFG activity at Time 1 and substance use and NA-IFG activity at Time 2. Data were also examined to characterize the environmental context in which youth first tried sips of alcohol (e.g., with or without parental permission, as part of a religious experience).

Results

Approximately 24% of the sample reported having tried sips of alcohol by ages 9, 10. On average, youth reported trying sips of alcohol 4.87 times (SD = 23.19) with age of first sip occurring at 7.36 years old (SD = 1.91). The first SVM model classified youth according to alcohol sipping status at Time 1 no better than chance with an accuracy of 0.35 (balanced accuracy = 0.52, sensitivity = 0.24, specificity = 0.80). The second SVM model classified youth according to alcohol drinking status at Time 2 with an accuracy of 0.76 (balanced accuracy = 0.56, sensitivity = 0.21, specificity = 0.91). Linear regression demonstrated that frequency of alcohol sipping at Time 1 predicted frequency of alcohol use at Time 2 (p < 0.001, adjusted R 2 = 0.075). Alcohol sipping at Time 1 was not linearly associated with NA or IFG activity at Time 2 (all ps > 0.05), and NA activity at Time 1 and Time 2 were not related (all ps > 0.05). Activity in the three subsections of the IFG at Time 1 predicted activity in those same regions at Time 2 (all ps < 0.02).

Conclusions and implications

Early sips of alcohol appear to predict alcohol use in early adolescence. Findings do not provide strong evidence for minimal early alcohol use (sipping) as a behavioral marker of underlying alterations in NA-IFG neural responsivity to reward. Improving our understanding of the neural and behavioral factors that indicate a greater propensity for future substance use is crucial for identifying at-risk youth and potential targets for preventative efforts.

Neural substrates of motivational dysfunction across neuropsychiatric conditions: Evidence from meta-analysis and lesion network mapping

Motivational dysfunction constitutes one of the fundamental dimensions of psychopathology cutting across traditional diagnostic boundaries. However, it is unclear whether there is a common neural circuit responsible for motivational dysfunction across neuropsychiatric conditions. To address this issue, the current study combined a meta-analysis on psychiatric neuroimaging studies of reward/loss anticipation and consumption (4308 foci, 438 contrasts, 129 publications) with a lesion network mapping approach (105 lesion cases). Our meta-analysis identified transdiagnostic hypoactivation in the ventral striatum (VS) for clinical/at-risk conditions compared to controls during the anticipation of both reward and loss. Moreover, the VS subserves a key node in a distributed brain network which encompasses heterogeneous lesion locations causing motivation-related symptoms. These findings do not only provide the first meta-analytic evidence of shared neural alternations linked to anticipatory motivation-related deficits, but also shed novel light on the role of VS dysfunction in motivational impairments in terms of both network integration and psychological functions. Particularly, the current findings suggest that motivational dysfunction across neuropsychiatric conditions is rooted in disruptions of a common brain network anchored in the VS, which contributes to motivational salience processing rather than encoding positive incentive values.

Distributed Neural Systems Support Flexible Attention Updating during Category Learning

To accurately categorize items, humans learn to selectively attend to stimulus dimensions that are most relevant to the task. Models of category learning describe the interconnected cognitive processes that contribute to attentional tuning as labeled stimuli are progressively observed. The Adaptive Attention Representation Model (AARM), for example, provides an account whereby categorization decisions are based on the perceptual similarity of a new stimulus to stored exemplars, and dimension-wise attention is updated on every trial in the direction of a feedback-based error gradient. As such, attention modulation as described by AARM requires interactions among orienting, visual perception, memory retrieval, prediction error, and goal maintenance to facilitate learning across trials. The current study explored the neural bases of attention mechanisms using quantitative predictions from AARM to analyze behavioral and fMRI data collected while participants learned novel categories. Generalized linear model analyses revealed patterns of BOLD activation in the parietal cortex (orienting), visual cortex (perception), medial temporal lobe (memory retrieval), basal ganglia (prediction error), and pFC (goal maintenance) that covaried with the magnitude of model-predicted attentional tuning. Results are consistent with AARM's specification of attention modulation as a dynamic property of distributed cognitive systems.

The neural signature of the decision value of future pain

SignificanceWe often willingly experience pain to reach a goal. However, potential pain can also prevent reckless action. How do we consider future pain when deciding on the best course of action? To date, the precise neural mechanisms underlying the valuation of future pain remain unknown. Using functional MRI, we derive a whole-brain signature of the value of future pain capable of predicting participants' choices to accept pain in exchange for a reward. We show that this signature is characterized by a distributed pattern of activity with clear contributions from structures encoding reward and salience, notably the ventral and dorsal striatum. These findings highlight how the brain assigns value to future pain when choosing the best course of action.

Meta-analysis of human prediction error for incentives, perception, cognition, and action

Prediction errors (PEs) are a keystone for computational neuroscience. Their association with midbrain neural firing has been confirmed across species and has inspired the construction of artificial intelligence that can outperform humans. However, there is still much to learn. Here, we leverage the wealth of human PE data acquired in the functional neuroimaging setting in service of a deeper understanding, using an MKDA (multi-level kernel-based density) meta-analysis. Studies were identified with Google Scholar, and we included studies with healthy adult participants that reported activation coordinates corresponding to PEs published between 1999-2018. Across 264 PE studies that have focused on reward, punishment, action, cognition, and perception, consistent with domain-general theoretical models of prediction error we found midbrain PE signals during cognitive and reward learning tasks, and an insula PE signal for perceptual, social, cognitive, and reward prediction errors. There was evidence for domain-specific error signals--in the visual hierarchy during visual perception, and the dorsomedial prefrontal cortex during social inference. We assessed bias following prior neuroimaging meta-analyses and used family-wise error correction for multiple comparisons. This organization of computation by region will be invaluable in building and testing mechanistic models of cognitive function and dysfunction in machines, humans, and other animals. Limitations include small sample sizes and ROI masking in some included studies, which we addressed by weighting each study by sample size, and directly comparing whole brain vs. ROI-based results.

The human insula processes both modality-independent and pain-selective learning signals

Prediction errors (PEs) are generated when there are differences between an expected and an actual event or sensory input. The insula is a key brain region involved in pain processing, and studies have shown that the insula encodes the magnitude of an unexpected outcome (unsigned PEs). In addition to signaling this general magnitude information, PEs can give specific information on the direction of this deviation-i.e., whether an event is better or worse than expected. It is unclear whether the unsigned PE responses in the insula are selective for pain or reflective of a more general processing of aversive events irrespective of modality. It is also unknown whether the insula can process signed PEs at all. Understanding these specific mechanisms has implications for understanding how pain is processed in the brain in both health and in chronic pain conditions. In this study, 47 participants learned associations between 2 conditioned stimuli (CS) with 4 unconditioned stimuli (US; painful heat or loud sound, of one low and one high intensity each) while undergoing functional magnetic resonance imaging (fMRI) and skin conductance response (SCR) measurements. We demonstrate that activation in the anterior insula correlated with unsigned intensity PEs, irrespective of modality, indicating an unspecific aversive surprise signal. Conversely, signed intensity PE signals were modality specific, with signed PEs following pain but not sound located in the dorsal posterior insula, an area implicated in pain intensity processing. Previous studies have identified abnormal insula function and abnormal learning as potential causes of pain chronification. Our findings link these results and suggest that a misrepresentation of learning relevant PEs in the insular cortex may serve as an underlying factor in chronic pain.

Differential reinforcement learning responses to positive and negative information in unmedicated individuals with depression

Major depressive disorder (MDD) is characterized by behavioral and neural abnormalities in processing both rewarding and aversive stimuli, which may impact motivational and affective symptoms. Learning paradigms have been used to assess reinforcement encoding abnormalities in MDD and their association with dysfunctional incentive-based behavior, but how the valence and context of information modulate this learning is not well understood. To address these gaps, we examined responses to positive and negative reinforcement across multiple temporal phases of information processing. While undergoing functional magnetic resonance imaging (fMRI), 47 participants (23 unmedicated, predominantly medication-naïve participants with MDD and 24 demographically-matched HC participants) completed a probabilistic, feedback-based reinforcement learning task that allowed us to separate neural activation during motor response (choice) from reinforcement feedback and monetary outcome across two independent conditions: pursuing gains and avoiding losses. In the gain condition, MDD participants showed overall blunted learning responses (prediction error) in the dorsal striatum when receiving monetary outcome, and reduced responses in ventral striatum for positive, but not negative, prediction error. The MDD group showed enhanced sensitivity to negative information, and symptom severity was associated with better behavioral performance in the loss condition. These findings suggest that striatal responses during learning are abnormal in individuals with MDD but vary with the valence of information.

Acute stress blunts prediction error signals in the dorsal striatum during reinforcement learning

Acute stress is pervasive in everyday modern life and is thought to affect how people make choices and learn from them. Reinforcement learning, which implicates learning from the unexpected rewarding and punishing outcomes of our choices (i.e., prediction errors), is critical for adjusted behaviour and seems to be affected by acute stress. However, the neural mechanisms by which acute stress disrupts this type of learning are still poorly understood. Here, we investigate whether and how acute stress blunts neural signalling of prediction errors during reinforcement learning using model-based functional magnetic resonance imaging. Male participants completed a well-established reinforcement-learning task involving monetary gains and losses whilst under stress and control conditions. Acute stress impaired participants’ (n = 23) behavioural performance towards obtaining monetary gains (p < 0.001), but not towards avoiding losses (p = 0.57). Importantly, acute stress blunted signalling of prediction errors during gain and loss trials in the dorsal striatum (p = 0.040) — with subsidiary analyses suggesting that acute stress preferentially blunted signalling of positive prediction errors. Our results thus reveal a neurocomputational mechanism by which acute stress may impair reward learning.

Pupil dilation as an index of Pavlovian conditioning. A systematic review and meta-analysis

The use of pupillometry to track emotional learning processes in humans is generating an increasing interest. Here, we provide a first systematic review and meta-analysis on the value of pupil dilation as a marker of Pavlovian conditioning, focusing on the roles of UCS valence (aversive vs. appetitive), the time course across trials and response intervals within trials. Based on data from 39 independent samples (total n = 1303), our results revealed strong evidence for the overall validity of conditioned pupil responses, with a trend for larger effects in aversive (average g = 0.73) vs. appetitive conditioning (g = 0.39). Response differentiation increased over the course of acquisition. Substantial differentiation effects were found in both early and late response windows. Moderator analyses revealed a consistent influence of UCS modality on differential conditioning, while evidence for moderation by contingency instructions and length of acquisition phase was mixed. The results highlight pupil dilation as a sensitive and reliable index of Pavlovian conditioning across valence categories and stimulus modalities. Important implications regarding methodological considerations and research goals are discussed.

Neural reward circuit dysfunction as a risk factor for bipolar spectrum disorders and substance use disorders: A review and integration

Bipolar spectrum disorders (BSDs) and substance use disorders (SUDs) are associated with neural reward dysfunction. However, it is unclear what pattern of neural reward function underlies pre-existing vulnerability to BSDs and SUDs, or whether neural reward function explains their high co-occurrence. The current paper provides an overview of the separate literatures on neural reward sensitivity in BSDs and SUDs. We provide a systematic review of 35 studies relevant to identifying neural reward function vulnerability to BSDs and SUDs. These studies include those examining neural reward processing on a monetary reward task with prospective designs predicting initial onset of SUDs, familial risk studies that examine unaffected offspring or first-degree relatives of family members with BSDs or SUDs, and studies that examine individuals with BSDs or SUDs who are not currently in an episode of the disorder. Findings from the review highlight that aberrant responding and connectivity across neural regions associated with reward and cognitive control confers risk for the development of BSDs and SUDs. Discussion focuses on limitations of the extant literature. We conclude with an integration and theoretical model for understanding how aberrant neural reward responding may constitute a vulnerability to the development of both BSDs and SUDs.

The Neural Correlates of Cued Reward Omission

Compared to our understanding of positive prediction error signals occurring due to unexpected reward outcomes, less is known about the neural circuitry in humans that drives negative prediction errors during omission of expected rewards. While classical learning theories such as Rescorla-Wagner or temporal difference learning suggest that both types of prediction errors result from a simple subtraction, there has been recent evidence suggesting that different brain regions provide input to dopamine neurons which contributes to specific components of this prediction error computation. Here, we focus on the brain regions responding to negative prediction error signals, which has been well-established in animal studies to involve a distinct pathway through the lateral habenula. We examine the activity of this pathway in humans, using a conditioned inhibition paradigm with high-resolution functional MRI. First, participants learned to associate a sensory stimulus with reward delivery. Then, reward delivery was omitted whenever this stimulus was presented simultaneously with a different sensory stimulus, the conditioned inhibitor (CI). Both reward presentation and the reward-predictive cue activated midbrain dopamine regions, insula and orbitofrontal cortex. While we found significant activity at an uncorrected threshold for the CI in the habenula, consistent with our predictions, it did not survive correction for multiple comparisons and awaits further replication. Additionally, the pallidum and putamen regions of the basal ganglia showed modulations of activity for the inhibitor that did not survive the corrected threshold.

Resting-State Functional Connectivity of the Punishment Network Associated With Conformity

Fear of punishment prompts individuals to conform. However, why some people are more inclined than others to conform despite being unaware of any obvious punishment remains unclear, which means the dispositional determinants of individual differences in conformity propensity are poorly understood. Here, we explored whether such individual differences might be explained by individuals' stable neural markers to potential punishment. To do this, we first defined the punishment network (PN) by combining all potential brain regions involved in punishment processing. We subsequently used a voxel-based global brain connectivity (GBC) method based on resting-state functional connectivity (FC) to characterize the hubs in the PN, which reflected an ongoing readiness state (i.e., sensitivity) for potential punishment. Then, we used the within-network connectivity (WNC) of each voxel in the PN of 264 participants to explain their tendency to conform by using a conformity scale. We found that a stronger WNC in the right thalamus, left insula, postcentral gyrus, and dACC was associated with a stronger tendency to conform. Furthermore, the FC among the four hubs seemed to form a three-phase ascending pathway, contributing to conformity propensity at every phase. Thus, our results suggest that task-independent spontaneous connectivity in the PN could predispose individuals to conform.

Putamen volume predicts real-time fMRI neurofeedback learning success across paradigms and neurofeedback target regions

Real-time fMRI guided neurofeedback training has gained increasing interest as a noninvasive brain regulation technique with the potential to modulate functional brain alterations in therapeutic contexts. Individual variations in learning success and treatment response have been observed, yet the neural substrates underlying the learning of self-regulation remain unclear. Against this background, we explored potential brain structural predictors for learning success with pooled data from three real-time fMRI data sets. Our analysis revealed that gray matter volume of the right putamen could predict neurofeedback learning success across the three data sets (n = 66 in total). Importantly, the original studies employed different neurofeedback paradigms during which different brain regions were trained pointing to a general association with learning success independent of specific aspects of the experimental design. Given the role of the putamen in associative learning this finding may reflect an important role of instrumental learning processes and brain structural variations in associated brain regions for successful acquisition of fMRI neurofeedback-guided self-regulation.

Modular deep reinforcement learning from reward and punishment for robot navigation

Modular Reinforcement Learning decomposes a monolithic task into several tasks with sub-goals and learns each one in parallel to solve the original problem. Such learning patterns can be traced in the brains of animals. Recent evidence in neuroscience shows that animals utilize separate systems for processing rewards and punishments, illuminating a different perspective for modularizing Reinforcement Learning tasks. MaxPain and its deep variant, Deep MaxPain, showed the advances of such dichotomy-based decomposing architecture over conventional Q-learning in terms of safety and learning efficiency. These two methods differ in policy derivation. MaxPain linearly unified the reward and punishment value functions and generated a joint policy based on unified values; Deep MaxPain tackled scaling problems in high-dimensional cases by linearly forming a joint policy from two sub-policies obtained from their value functions. However, the mixing weights in both methods were determined manually, causing inadequate use of the learned modules. In this work, we discuss the signal scaling of reward and punishment related to discounting factor γ, and propose a weak constraint for signaling design. To further exploit the learning models, we propose a state-value dependent weighting scheme that automatically tunes the mixing weights: hard-max and softmax based on a case analysis of Boltzmann distribution. We focus on maze-solving navigation tasks and investigate how two metrics (pain-avoiding and goal-reaching) influence each other's behaviors during learning. We propose a sensor fusion network structure that utilizes lidar and images captured by a monocular camera instead of lidar-only and image-only sensing. Our results, both in the simulation of three types of mazes with different complexities and a real robot experiment of an L-maze on Turtlebot3 Waffle Pi, showed the improvements of our methods.

Differing effects of gain and loss feedback on rule-based and information-integration category learning

Although most category-learning studies use feedback for training, little attention has been paid to how individuals use feedback value and framing of feedback as gains or losses to support learning. We compared learning of rule-based (RB) and information-integration (II) categories with point-valued feedback in which participants gained or lost higher point values for more difficult category members (those closer to the decision bound). We implemented point-valued feedback in four different conditions: Gain (earn points for correct answers), Loss (lose points for incorrect answers), Gain+Loss (earn points for correct answers and lose points for incorrect answers), and Control (accuracy feedback only without point gain or loss). Participants were trained until they reached criterion. Overall, point-valued feedback led to better learning than control conditions. However, the patterns differed across category-learning tasks. In the II task participants reached learning criterion fastest when they received both Gains and Losses. This is consistent with the reliance of II learning on reinforcement-based mechanisms and research showing common coding of gains and losses in neural regions underlying II learning. In contrast, in the RB task, participants reached criterion most rapidly when they received either Gains or Losses, but not both Gains and Losses together. The relative impairment in the Gain+Loss condition in RB learning may be due to conflicting executive function demands for interpreting and using the separate Gain and Loss information, and is consistent with reliance of RB learning on explicit hypothesis-testing mechanisms.

Physically active individuals look for more: An eye-tracking study of attentional bias

Attentional capture by exercise-related stimuli is important for the regulation of physical activity. Attentional processing underlying this capture has been investigated with indirect behavioral measures based on reaction times. To investigate more direct measures of visual spatial attention toward physical activity (vs. inactivity) stimuli, we used eye-tracking and a visual dot probe task in 77 young adults with various level of physical activity. Reaction times to detect a dot appearing in the area previously occupied by a physical activity (vs. inactivity) stimulus were an indirect measure of attentional bias. The first picture gaze and viewing time were more direct measures of attentional orienting and attentional engagement, respectively. Pupil dilation was an indicator of arousal. Reaction times revealed a two-way interaction between the location of the dot and participants' usual level of physical activity. Only participants with a high level of physical activity more quickly detected a dot when it appeared in the area previously occupied by a physical activity stimulus. Eye-tracking results showed greater odds of first gazing at physical activity stimuli and for a longer time, and a greater decrease in pupil size when viewing physical activity stimuli when usual level of physical activity was moderate or high, but not low. The variance explained in the outcomes ranged from 13.9% (pupil dilation) to 40% (reaction times). Overall, as hypothesized, compared to less physically active participants, participants who were more physically active demonstrated indirect (reaction times) and direct (first gaze, viewing time) evidence of a more pronounced attentional bias toward physical activity. Physical activity stimuli biased attention, with a pronounced effect when the level of physical activity was higher. These findings suggest that physical activity stimuli are relevant to the current concerns of moderately and highly active individuals.

Do Adolescents Use Substances to Relieve Uncomfortable Sensations? A Preliminary Examination of Negative Reinforcement among Adolescent Cannabis and Alcohol Users

Alcohol and cannabis use are highly prevalent among adolescents and associated with negative consequences. Understanding motivations behind substance use in youth is important for informing prevention and intervention efforts. The present study aims to examine negative reinforcement principles of substance use among adolescent cannabis and alcohol users by pairing a cue reactivity paradigm with an aversive interoceptive stimulus. Adolescents (ages 15–17), classified as controls (CTL; n = 18), cannabis and/or alcohol experimenters (CAN+ALC-EXP; n = 16), or individuals meeting clinical criteria for cannabis and/or alcohol use disorder (CAN+ALC-SUD; n = 13) underwent functional magnetic resonance imaging during which they experienced an aversive interoceptive probe delivered via breathing load while simultaneously performing a cue reactivity paradigm. Participants also provided self-report ratings of how their substance use is positively or negatively reinforced. While experiencing the breathing load, CAN+ALC-SUD exhibited greater (p < 0.05) deactivation in the right amygdala, the left inferior frontal gyrus, and the left parahippocampal gyrus than CAN+ALC-EXP and CTL, who did not differ. Across all substance users, activation during the breathing load within the left parahippocampal gyrus negatively correlated with cannabis and alcohol lifetime use episodes and the left inferior frontal gyrus activity negatively correlated with lifetime alcohol use episodes. CAN+ALC-SUD reported experiencing more positive and negative reinforcement of using their substance of choice than CAN+ALC-EXP; both user groups reported higher levels of positive than negative reinforcement. Adolescents with a cannabis/alcohol use disorder demonstrate an altered response to interoceptive perturbations. However, adolescent cannabis/alcohol use does not appear to be driven by negative reinforcement, as viewing substance images did not dampen this response. Based on self-report data, the experience of positive reinforcement may be stronger for adolescents. Future studies should examine whether positive reinforcement contributes to adolescent substance use.

Signals of anticipation of reward and of mean reward rates in the human brain

Theoretical models of dopamine function stemming from reinforcement learning theory have emphasized the importance of prediction errors, which signal changes in the expectation of impending rewards. Much less is known about the effects of mean reward rates, which may be of motivational significance due to their role in computing the optimal effort put into exploiting reward opportunities. Here, we used a reinforcement learning model to design three functional neuroimaging studies and disentangle the effects of changes in reward expectations and mean reward rates, showing recruitment of specific regions in the brainstem regardless of prediction errors. While changes in reward expectations activated ventral striatal areas as in previous studies, mean reward rates preferentially modulated the substantia nigra/ventral tegmental area, deep layers of the superior colliculi, and a posterior pontomesencephalic region. These brainstem structures may work together to set motivation and attentional efforts levels according to perceived reward opportunities.

Neural Correlates of Learning Pure Tones or Natural Sounds in the Auditory Cortex

Associative learning of pure tones is known to cause tonotopic map expansion in the auditory cortex (ACx), but the function this plasticity sub-serves is unclear. We developed an automated training platform called the “Educage,” which was used to train mice on a go/no-go auditory discrimination task to their perceptual limits, for difficult discriminations among pure tones or natural sounds. Spiking responses of excitatory and inhibitory parvalbumin (PV⁺) L2/3 neurons in mouse ACx revealed learning-induced overrepresentation of the learned frequencies, as expected from previous literature. The coordinated plasticity of excitatory and inhibitory neurons supports a role for PV⁺ neurons in homeostatic maintenance of excitation–inhibition balance within the circuit. Using a novel computational model to study auditory tuning curves, we show that overrepresentation of the learned tones does not necessarily improve discrimination performance of the network to these tones. In a separate set of experiments, we trained mice to discriminate among natural sounds. Perceptual learning of natural sounds induced “sparsening” and decorrelation of the neural response, consequently improving discrimination of these complex sounds. This signature of plasticity in A1 highlights its role in coding natural sounds.

A Neurobehavioral Approach to Addiction: Implications for the Opioid Epidemic and the Psychology of Addiction

Two major questions about addictive behaviors need to be explained by any worthwhile neurobiological theory. First, why do people seek drugs in the first place? Second, why do some people who use drugs seem to eventually become unable to resist drug temptation and so become "addicted"? We will review the theories of addiction that address negative-reinforcement views of drug use (i.e., taking opioids to alleviate distress or withdrawal), positive-reinforcement views (i.e., taking drugs for euphoria), habit views (i.e., growth of automatic drug-use routines), incentive-sensitization views (i.e., growth of excessive "wanting" to take drugs as a result of dopamine-related sensitization), and cognitive-dysfunction views (i.e., impaired prefrontal top-down control), including those involving competing neurobehavioral decision systems (CNDS), and the role of the insula in modulating addictive drug craving. In the special case of opioids, particular attention is paid to whether their analgesic effects overlap with their reinforcing effects and whether the perceived low risk of taking legal medicinal opioids, which are often prescribed by a health professional, could play a role in the decision to use. Specifically, we will address the issue of predisposition or vulnerability to becoming addicted to drugs (i.e., the question of why some people who experiment with drugs develop an addiction, while others do not). Finally, we review attempts to develop novel therapeutic strategies and policy ideas that could help prevent opioid and other substance abuse.

Pupil dilation as an implicit measure of appetitive Pavlovian learning

Appetitive Pavlovian conditioning is a learning mechanism of fundamental biological and pathophysiological significance. Nonetheless, its exploration in humans remains sparse, which is partly attributed to the lack of an established psychophysiological parameter that aptly represents conditioned responding. This study evaluated pupil diameter and other ocular response measures (gaze dwelling time, blink duration and count) as indices of conditioning. Additionally, a learning model was used to infer participants' learning progress on the basis of their pupil dilation. Twenty-nine healthy volunteers completed an appetitive differential delay conditioning paradigm with a primary reward, while the ocular response measures along with other psychophysiological (heart rate, electrodermal activity, postauricular and eyeblink reflex) and behavioral (ratings, contingency awareness) parameters were obtained to examine the relation among different measures. A significantly stronger increase in pupil diameter, longer gaze duration and shorter eyeblink duration was observed in response to the reward-predicting cue compared to the control cue. The Pearce-Hall attention model best predicted the trial-by-trial pupil diameter. This conditioned response was corroborated by a pronounced heart rate deceleration to the reward-predicting cue, while no conditioning effect was observed in the electrodermal activity or startle responses. There was no discernible correlation between the psychophysiological response measures. These results highlight the potential value of ocular response measures as sensitive indices for representing appetitive conditioning.

Reappraisal of incentives ameliorates choking under pressure and is correlated with changes in the neural representations of incentives

It has been observed that the performing for high stakes can, paradoxically, lead to uncharacteristically poor performance. Here we investigate a novel approach to attenuating such 'choking under pressure' by instructing participants performing a demanding motor task that rewards successful performance with a monetary gain, to reappraise this incentive as a monetary loss for unsuccessful performance. We show that when participants applied this simple strategy, choking was significantly reduced. This strategy also influenced participants' neural and physiological activity. When participants reappraised the incentive as a potential monetary loss, the representation of the magnitude of the incentive in the ventral striatum Blood Oxygenation Level Dependent (BOLD) signal was attenuated. In addition, individual differences in the degree of attenuation of the neural response to incentive predicted the effectiveness of the reappraisal strategy in reducing choking. Furthermore, participants' skin conductance changed in proportion to the magnitude of the incentive being played for, and was exaggerated on high incentive trials on which participants failed. Reappraisal of the incentive abolished this exaggerated skin conductance response. This represents the first experimental association of sympathetic arousal with choking. Taken together, these results suggest that reappraisal of the incentive is indeed a promising intervention for attenuating choking under pressure.

Topography of Reward and Aversion Encoding in the Mesolimbic Dopaminergic System

Dopamine (DA) neurons in the VTA play essential roles in adaptive motivated behavior, which requires rapid discrimination between positive and negative motivational signature. However, the precise functional DA circuitry processing reward and aversive information remain elusive. Here, we report that the encoding of reward and aversion by the DA system in the NAc is tightly associated with its anatomical location. By recording the dynamics of DA release with genetically encoded fluorescent DA sensor using in vivo fiber photometry in freely moving male mice, we found that the DA-sensor signal in the dorsomedial NAc shell and dorsolateral NAc shell were increased during rewarding events and decreased during aversive noxious events. In contrast, the release of DA in the ventromedial NAc shell was increased by both rewarding and aversive stimuli, whereas the DA-sensor signal in the central ventromedial NAc shell and ventrolateral NAc shell showed complex dynamics. Furthermore, the activity of DA fibers in different subregions of NAc measured with calcium sensor largely recapitulated the changes of DA-sensor signal in response to rewarding and aversive stimuli. In addition, correlation analysis showed that the response magnitude of DA-sensor or fibers significantly changed along the DV axis of the NAc. These results revealed the distinct role of the mesolimbic DA system in different subregions of NAc in encoding value and salience.SIGNIFICANCE STATEMENT Adaptive motivated behavior requires rapid discrimination between favorable and harmful events and is dynamically modulated by dopamine (DA) neurons in the VTA. However, the precise relationship between distinct DA circuitry and reward/aversion signal encoding is not well understood. Here, by recording the dynamics of DA release and the activity of DA fibers in each subregion of the NAc using in vivo fiber photometry in freely moving animals, we found that the DA system in the dorsomedial/dorsolateral, ventromedial, and ventrolateral NAc shell plays different roles in encoding value and salience. These results extend our knowledge about how the mesolimbic DA system process motivational information at the circuitry level.

Pain-related nucleus accumbens function: modulation by reward and sleep disruption

The nucleus accumbens (NAc) has been implicated in sleep, reward, and pain modulation, but the relationship between these functional roles is unclear. This study aimed to determine whether NAc function at the onset and offset of a noxious thermal stimulus is enhanced by rewarding music, and whether that effect is reversed by experimental sleep disruption. Twenty-one healthy subjects underwent functional magnetic resonance imaging scans on 2 separate days after both uninterrupted sleep and experimental sleep disruption. During functional magnetic resonance imaging scans, participants experienced noxious stimulation while listening to individualized rewarding or neutral music. Behavioral results revealed that rewarding music significantly reduced pain intensity compared with neutral music, and disrupted sleep was associated with decreased pain intensity in the context of listening to music. In whole-brain family-wise error cluster-corrected analysis, the NAc was activated at pain onset, but not during tonic pain or at pain offset. Sleep disruption attenuated NAc activation at pain onset and during tonic pain. Rewarding music altered NAc connectivity with key nodes of the corticostriatal circuits during pain onset. Sleep disruption increased reward-related connectivity between the NAc and the anterior midcingulate cortex at pain onset. This study thus indicates that experimental sleep disruption modulates NAc function during the onset of pain in a manner that may be conditional on the presence of competing reward-related stimuli. These findings point to potential mechanisms for the interaction between sleep, reward, and pain, and suggest that sleep disruption affects both the detection and processing of aversive stimuli that may have important implications for chronic pain.

From Freedom From to Freedom To: New Perspectives on Intentional Action

There are few concepts as relevant as that of intentional action in shaping our sense of self and the interaction with the environment. At the same time, few concepts are so elusive. Indeed, both conceptual and neuroscientific accounts of intentional agency have proven to be problematic. On the one hand, most conceptual views struggle in defining how agents can adequately exert control over their actions. On the other hand, neuroscience settles for definitions by exclusion whereby key features of human intentional actions, including goal-directness, remain underspecified. This paper reviews the existing literature and sketches how this gap might be filled. In particular, we defend a gradualist notion of intentional behavior, which revolves around the following key features: autonomy, flexibility in the integration of causal vectors, and control.

In Cocaine Dependence, Neural Prediction Errors During Loss Avoidance Are Increased With Cocaine Deprivation and Predict Drug Use

BACKGROUND

In substance-dependent individuals, drug deprivation and drug use trigger divergent behavioral responses to environmental cues. These divergent responses are consonant with data showing that short- and long-term adaptations in dopamine signaling are similarly sensitive to state of drug use. The literature suggests a drug state-dependent role of learning in maintaining substance use; evidence linking dopamine to both reinforcement learning and addiction provides a framework to test this possibility.

METHODS

In a randomized crossover design, 22 participants with current cocaine use disorder completed a probabilistic loss-learning task during functional magnetic resonance imaging while on and off cocaine (44 sessions). Another 54 participants without Axis I psychopathology served as a secondary reference group. Within-drug state and paired-subjects' learning effects were assessed with computational model-derived individual learning parameters. Model-based neuroimaging analyses evaluated effects of drug use state on neural learning signals. Relationships among model-derived behavioral learning rates (α+, α-), neural prediction error signals (δ+, δ-), cocaine use, and desire to use were assessed.

RESULTS

During cocaine deprivation, cocaine-dependent individuals exhibited heightened positive learning rates (α+), heightened neural positive prediction error (δ+) responses, and heightened association of α+ with neural δ+ responses. The deprivation-enhanced neural learning signals were specific to successful loss avoidance, comparable to participants without psychiatric conditions, and mediated a relationship between chronicity of drug use and desire to use cocaine.

CONCLUSIONS

Neurocomputational learning signals are sensitive to drug use status and suggest that heightened reinforcement by successful avoidance of negative outcomes may contribute to drug seeking during deprivation. More generally, attention to drug use state is important for delineating substrates of addiction.

Behavioural evidence for parallel outcome-sensitive and outcome-insensitive Pavlovian learning systems in humans

There is a dichotomy in instrumental conditioning between goal-directed actions and habits that are distinguishable on the basis of their relative sensitivity to changes in outcome value. It is less clear whether a similar distinction applies in Pavlovian conditioning, where responses have been found to be predominantly outcome sensitive. To test for both devaluation insensitive and devaluation sensitive Pavlovian conditioning in humans, we conducted four experiments combining Pavlovian conditioning and outcome devaluation procedures while measuring multiple conditioned responses. Our results suggest that Pavlovian conditioning involves two distinct types of learning: one that learns the current value of the outcome which is sensitive to devaluation, and one that learns about the spatial localisation of the outcome which is insensitive to devaluation. Our findings have implications for the mechanistic understanding of Pavlovian conditioning and provide a more nuanced understanding of Pavlovian mechanisms that might contribute to a number of psychiatric disorders.

The Expression and Transfer of Valence Associated with Social Conformity

Consensus seeking – abandoning one’s own judgment to align with a group majority – is a fundamental feature of human social interaction. Notably, such striving for majority affiliation often occurs in the absence of any apparent economic or social gain, suggesting that achieving consensus might have intrinsic value. Here, using a simple gambling task, in which the decisions of ostensible previous gamblers were indicated below available options on each trial, we examined the affective properties of agreeing with a group majority by assessing the trade-off between social and non-social currencies, and the transfer of social valence to concomitant stimuli. In spite of demonstrating near-perfect knowledge of objective reward probabilities, participant’s choices and evaluative judgments reflected a reliable preference for conformity, consistent with the hypothesized value of social alignment.

Gain-loss situation modulates neural responses to self-other decision making under risk

Although self-other behavioral differences in decision making under risk have been observed in some contexts, little is known about the neural mechanisms underlying such differences. Using functional magnetic resonance imaging (fMRI) and the cups task, in which participants choose between risky and sure options for themselves and others in gain and loss situations, we found that people were more risk-taking when making decisions for themselves than for others in loss situations but were equally risk-averse in gain situations. Significantly stronger activations were observed in the dorsomedial prefrontal cortex (dmPFC) and anterior insula (AI) when making decisions for the self than for others in loss situations but not in gain situations. Furthermore, the activation in the dmPFC was stronger when people made sure choices for others than for themselves in gain situations but not when they made risky choices, and was both stronger when people made sure and risky choices for themselves than for others in loss situations. These findings suggest that gain-loss situation modulates self-other differences in decision making under risk, and people are highly likely to differentiate the self from others when making decisions in loss situations.

Selective Role of the Putamen in Serial Reversal Learning in the Marmoset

Fronto-striatal circuitry involving the orbitofrontal cortex has been identified as mediating successful reversal of stimulus-outcome contingencies. The region of the striatum that most contributes to reversal learning remains unclear, with studies in primates implicating both caudate nucleus and putamen. We trained four marmosets on a touchscreen-based serial reversal task and implanted each with cannulae targeting both putamen and caudate bilaterally. This allowed reversible inactivation of the two areas within the same monkeys, but across separate sessions, to directly investigate their respective contributions to reversal performance. Behavioral sensitivity to the GABAA agonist muscimol varied across subjects and between brain regions, so each marmoset received a range of doses. Intermediate doses of intra-putamen muscimol selectively impaired reversal performance, leaving the baseline discrimination phase unchanged. There was no effect of low doses and high doses were generally disruptive. By contrast, low doses of intra-caudate muscimol improved reversal performance, while high doses impaired both reversal and baseline discrimination performance. These data provide evidence for a specific role of the putamen in serial reversal learning, which may reflect the more habitual nature of repeated reversals using the same stimulus pair.

Effects of Positive and Negative Expectations on Human Pain Perception Engage Separate But Interrelated and Dependently Regulated Cerebral Mechanisms

Expectations substantially influence pain perception, but the relationship between positive and negative expectations remains unclear. Recent evidence indicates that the integration between pain-related expectations and prediction errors is crucial for pain perception, which suggests that aversive prediction error-associated regions, such as the anterior insular cortex (aIC) and rostral anterior cingulate cortex (rACC), may play a pivotal role in expectation-induced pain modulation and help to delineate the relationship between positive and negative expectations. In a stimulus expectancy paradigm combining fMRI in healthy volunteers of both sexes, we found that, although positive and negative expectations respectively engaged the right aIC and right rACC to modulate pain, their associated activations and pain rating changes were significantly correlated. When positive and negative expectations modulated pain, the right aIC and rACC exhibited opposite coupling with periaqueductal gray (PAG) and the mismatch between actual and expected pain respectively modulated their coupling with PAG and thalamus across individuals. Participants' certainty about expectations predicted the extent of pain modulation, with positive expectations involving connectivity between aIC and hippocampus, a region regulating anxiety, and negative expectations engaging connectivity between rACC and lateral orbitofrontal cortex, a region reflecting outcome value and certainty. Interestingly, the strength of these certainty-related connectivities was also significantly associated between positive and negative expectations. These findings suggest that aversive prediction-error-related regions interact with pain-processing circuits to underlie stimulus expectancy effects on pain, with positive and negative expectations engaging dissociable but interrelated neural responses that are dependently regulated by individual certainty about expectations.SIGNIFICANCE STATEMENT Positive and negative expectations substantially influence pain perception, but their relationship remains unclear. Using fMRI in a stimulus expectancy paradigm, we found that, although positive and negative expectations engaged separate brain regions encoding the mismatch between actual and expected pain and involved opposite functional connectivities with the descending pain modulatory system, they produced significantly correlated pain rating changes and brain activation. Moreover, participants' certainty about expectations predicted the magnitude of both types of pain modulation, with the underlying functional connectivities significantly correlated between positive and negative expectations. These findings advance current understanding about cognitive modulation of pain, suggesting that both types of pain modulation engage different aversive prediction error signals but are dependently regulated by individual certainty about expectations.

A Neural Circuit Mechanism for Encoding Aversive Stimuli in the Mesolimbic Dopamine System

Ventral tegmental area (VTA) dopamine (DA) neurons play a central role in mediating motivated behaviors, but the circuitry through which they signal positive and negative motivational stimuli is incompletely understood. Using in vivo fiber photometry, we simultaneously recorded activity in DA terminals in different nucleus accumbens (NAc) subnuclei during an aversive and reward conditioning task. We find that DA terminals in the ventral NAc medial shell (vNAcMed) are excited by unexpected aversive outcomes and to cues that predict them, whereas DA terminals in other NAc subregions are persistently depressed. Excitation to reward-predictive cues dominated in the NAc lateral shell and was largely absent in the vNAcMed. Moreover, we demonstrate that glutamatergic (VGLUT2-expressing) neurons in the lateral hypothalamus represent a key afferent input for providing information about aversive outcomes to vNAcMed-projecting DA neurons. Collectively, we reveal the distinct functional contributions of separate mesolimbic DA subsystems and their afferent pathways underlying motivated behaviors. VIDEO ABSTRACT.

Financial gain- and loss-related BOLD signals in the human ventral tegmental area and substantia nigra pars compacta

Neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNC) play central roles in reward-related behaviours. Nonhuman animal studies suggest that these neurons also process aversive events. However, our understanding of how the human VTA and SNC responds to such events is limited and has been hindered by the technical challenge of using functional magnetic resonance imaging (fMRI) to investigate a small structure where the signal is particularly vulnerable to physiological noise. Here we show, using methods optimized specifically for the midbrain (including high-resolution imaging, a novel registration protocol, and physiological noise modelling), a BOLD (blood-oxygen-level dependent) signal to both financial gain and loss in the VTA and SNC, along with a response to nil outcomes that are better or worse than expected in the VTA. Taken together, these findings suggest that the human VTA and SNC are involved in the processing of both appetitive and aversive financial outcomes in humans.

Impact of induced anxiety on neural responses to monetary incentives

Previous research demonstrates that aversive stimuli can interrupt appetitive processing and that brain regions involved with the processing of potential rewards, such as the ventral striatum (VS), also respond to threatening information. Potential losses can likewise activate the VS and, thus, the full extent to which threat can impact neural responses during incentive processing remains unclear. Here, unpredictable threat of shock was used to induce anxiety while participants performed the monetary incentive delay (MID) task during functional magnetic resonance imaging (fMRI). During anticipation, anxiety impacted neural responses within the bilateral VS and distributed regions of the occipital cortex. Anxiety enhanced activity within the VS to both gain and loss trials. Furthermore, anxiety enhanced activity to both gain and loss trials within dorsal areas of BA19. However, anxiety only enhanced activity during gain, but not loss trials, within ventral areas of BA19. These results suggest that during anticipation, induced anxiety enhanced VS activity to incentives generally, which might reflect changes in the subjective salience of gains and losses. Collectively, these results suggest that the impact of induced anxiety on responses to monetary incentives depend on the neural region, type of incentive, and stage of processing.

Microglial dopamine receptor elimination defines sex-specific nucleus accumbens development and social behavior in adolescent rats

Adolescence is a developmental period in which the mesolimbic dopaminergic "reward" circuitry of the brain, including the nucleus accumbens (NAc), undergoes significant plasticity. Dopamine D1 receptors (D1rs) in the NAc are critical for social behavior, but how these receptors are regulated during adolescence is not well understood. In this report, we demonstrate that microglia and complement-mediated phagocytic activity shapes NAc development by eliminating D1rs in male, but not female rats, during adolescence. Moreover, immune-mediated elimination of D1rs is required for natural developmental changes in male social play behavior. These data demonstrate for the first time that microglia and complement-mediated immune signaling (i) participate in adolescent brain development in a sex-specific manner, and (ii) are causally implicated in developmental changes in behavior. These data have broad implications for understanding the adolescent critical period of development, the molecular mechanisms underlying social behavior, and sex differences in brain structure and function.