Valenced action/inhibition learning in humans is modulated by a genetic variant linked to dopamine D2 receptor expression

The neurocognitive role of working memory load when Pavlovian motivational control affects instrumental learning

Research suggests that a fast, capacity-limited working memory (WM) system and a slow, incremental reinforcement learning (RL) system jointly contribute to instrumental learning. Thus, situations that strain WM resources alter instrumental learning: under WM loads, learning becomes slow and incremental, the reliance on computationally efficient learning increases, and action selection becomes more random. It is also suggested that Pavlovian learning influences people's behavior during instrumental learning by providing hard-wired instinctive responses including approach to reward predictors and avoidance of punishment predictors. However, it remains unknown how constraints on WM resources affect instrumental learning under Pavlovian influence. Thus, we conducted a functional magnetic resonance imaging (fMRI) study (N = 49) in which participants completed an instrumental learning task with Pavlovian-instrumental conflict (the orthogonalized go/no-go task) both with and without extra WM load. Behavioral and computational modeling analyses revealed that WM load reduced the learning rate and increased random choice, without affecting Pavlovian bias. Model-based fMRI analysis revealed that WM load strengthened RPE signaling in the striatum. Moreover, under WM load, the striatum showed weakened connectivity with the ventromedial and dorsolateral prefrontal cortex when computing reward expectations. These results suggest that the limitation of cognitive resources by WM load promotes slow and incremental learning through the weakened cooperation between WM and RL; such limitation also makes action selection more random, but it does not directly affect the balance between instrumental and Pavlovian systems.

Dissociable feedback valence effects on frontal midline theta during reward gain versus threat avoidance learning

While frontal midline theta (FMθ) has been associated with threat processing, with cognitive control in the context of anxiety, and with reinforcement learning, most reinforcement learning studies on FMθ have used reward rather than threat-related stimuli as reinforcer. Accordingly, the role of FMθ in threat-related reinforcement learning is largely unknown. Here, n = 23 human participants underwent one reward-, and one punishment-, based reversal learning task, which differed only with regard to the kind of reinforcers that feedback was tied to (i.e., monetary gain vs. loud noise burst, respectively). In addition to single-trial EEG, we assessed single-trial feedback expectations based on both a reinforcement learning computational model and trial-by-trial subjective feedback expectation ratings. While participants' performance and feedback expectations were comparable between the reward and punishment tasks, FMθ was more reliably amplified to negative vs. positive feedback in the reward vs. punishment task. Regressions with feedback valence, computationally derived, and self-reported expectations as predictors and FMθ as criterion further revealed that trial-by-trial variations in FMθ specifically relate to reward-related feedback-valence and not to threat-related feedback or to violated expectations/prediction errors. These findings suggest that FMθ as measured in reinforcement learning tasks may be less sensitive to the processing of events with direct relevance for fear and anxiety.

Motivational learning biases are differentially modulated by genetic determinants of striatal and prefrontal dopamine function

Dopaminergic neurotransmission plays a pivotal role in appetitively motivated behavior in mammals, including humans. Notably, action and valence are not independent in motivated tasks, and it is particularly difficult for humans to learn the inhibition of an action to obtain a reward. We have previously observed that the carriers of the DRD2/ANKK1 TaqIA A1 allele, that has been associated with reduced striatal dopamine D2 receptor expression, showed a diminished learning performance when required to learn response inhibition to obtain rewards, a finding that was replicated in two independent cohorts. With our present study, we followed two aims: first, we aimed to replicate our finding on the DRD2/ANKK1 TaqIA polymorphism in a third independent cohort (N = 99) and to investigate the nature of the genetic effects more closely using trial-by-trial behavioral analysis and computational modeling in the combined dataset (N = 281). Second, we aimed to assess a potentially modulatory role of prefrontal dopamine availability, using the widely studied COMT Val108/158Met polymorphism as a proxy. We first report a replication of the above mentioned finding. Interestingly, after combining all three cohorts, exploratory analyses regarding the COMT Val108/158Met polymorphism suggest that homozygotes for the Met allele, which has been linked to higher prefrontal dopaminergic tone, show a lower learning bias. Our results corroborate the importance of genetic variability of the dopaminergic system in individual learning differences of action-valence interaction and, furthermore, suggest that motivational learning biases are differentially modulated by genetic determinants of striatal and prefrontal dopamine function.

Investigating the effect of losses and gains on effortful engagement during an incentivized Go/NoGo task through anticipatory cortical oscillatory changes

Losses usually have greater subjective value (SV) than gains of equal nominal value but often cause a relative deterioration in effortful performance. Since losses and gains induce differing approach/avoidance behavioral tendencies, we explored whether incentive type interacted with approach/avoidance motor-sets. Alpha- and beta-band event-related desynchronization (ERD) was hypothesized to be weakest when participants expected a loss and prepared an inhibitory motor-set, and strongest when participants expected a gain and prepared an active motor-set. It was also hypothesized that effort would modulate reward and motor-set-related cortical activation patterns. Participants completed a cued Go/NoGo task while expecting a reward (+10p), avoiding a loss (-10p), or receiving no incentive (0p); and while expecting a NoGo cue with a probability of either .75 or .25. Pre-movement alpha- and beta-band EEG power was analyzed using the ERD method, and the SV of effort was evaluated using a cognitive effort discounting task. Gains incentivized faster RTs and stronger preparatory alpha band ERD compared to loss and no incentive conditions, while inhibitory motor-sets resulted in significantly weaker alpha-band ERD. However, there was no interaction between incentive and motor-sets. Participants were more willing to expend effort in losses compared to gain trials, although the SV of effort was not associated with ERD patterns or RTs. Results suggest that incentive and approach/avoidance motor tendencies modulate cortical activations prior to a speeded RT movement independently, and are not associated with the economic value of effort. The present results favor attentional explanations of the effect of incentive modality on effort.

Overcoming Pavlovian bias in semantic space

Action is invigorated in the presence of reward-predicting stimuli and inhibited in the presence of punishment-predicting stimuli. Although valuable as a heuristic, this Pavlovian bias can also lead to maladaptive behaviour and is implicated in addiction. Here we explore whether Pavlovian bias can be overcome through training. Across five experiments, we find that Pavlovian bias is resistant to unlearning under most task configurations. However, we demonstrate that when subjects engage in instrumental learning in a verbal semantic space, as opposed to a motoric space, not only do they exhibit the typical Pavlovian bias, but this Pavlovian bias diminishes with training. Our results suggest that learning within the semantic space is necessary, but not sufficient, for subjects to unlearn their Pavlovian bias, and that other task features, such as gamification and spaced stimulus presentation may also be necessary. In summary, we show that Pavlovian bias, whilst robust, is susceptible to change with experience, but only under specific environmental conditions.

The Role of the Striatum in Learning to Orthogonalize Action and Valence: A Combined PET and 7 T MRI Aging Study

Pavlovian biases influence instrumental learning by coupling reward seeking with action invigoration and punishment avoidance with action suppression. Using a probabilistic go/no-go task designed to orthogonalize action (go/no-go) and valence (reward/punishment), recent studies have shown that the interaction between the two is dependent on the striatum and its key neuromodulator dopamine. Using this task, we sought to identify how structural and neuromodulatory age-related differences in the striatum may influence Pavlovian biases and instrumental learning in 25 young and 31 older adults. Computational modeling revealed a significant age-related reduction in reward and punishment sensitivity and marked (albeit not significant) reduction in learning rate and lapse rate (irreducible noise). Voxel-based morphometry analysis using 7 Tesla MRI images showed that individual differences in learning rate in older adults were related to the volume of the caudate nucleus. In contrast, dopamine synthesis capacity in the dorsal striatum, assessed using [18F]-DOPA positron emission tomography in 22 of these older adults, was not associated with learning performance and did not moderate the relationship between caudate volume and learning rate. This multiparametric approach suggests that age-related differences in striatal volume may influence learning proficiency in old age.

Biasing Actions by Incentive Valence in an Approach/Avoidance Task

The present study investigates interactions between incentive valence and action, which mirror well-known valence-action biases in the emotional domain. In three joystick experiments, incentive valence (win/loss) and action type (approach/avoid) were signaled by distinct orthogonal stimulus features. By combining several design aspects, i.e., the use of bi-directional joystick movements, the inclusion of no-incentive baseline trials, and cue-locked versus target-locked valence and action signals, we tried to bridge between paradigms used in the emotional and motivational domain, and to understand previous, partly inconsistent results. In the first task variant (Experiment 1), we observed performance benefits for compatible mappings (win-approach; loss-avoid) relative to incompatible ones (loss-approach; win-avoid) when valence and action signals were target-locked, consistent with a fairly automatic response activation that can benefit or impair task performance. In contrast, cue-locked valence signals led to response facilitation (relative to a no-incentive baseline) more or less independent of actual valence (win/loss) and action type (approach/avoid), which is reminiscent of general facilitation effects of incentive cues across diverse cognitive tasks. Slight design variations did not change this main result pattern, indicating that it was neither driven by the close proximity between target and performance feedback (Experiment 2), nor by mere temporal coincidence of valence and action signals (Experiment 3), but rather by differences between preparatory (cued) and immediate (non-cued) effects of incentive valence. The present study provides novel insights regarding the nature of valence-action biases in the motivational domain and helps to integrate previous, partly inconsistent findings across domains.

Dorsal striatal dopamine D1 receptor availability predicts an instrumental bias in action learning

The brain’s dopaminergic pathways are crucially important for adaptive behavior. They are thought to enable us to approach rewards and stay away from punishments. During learning, dopaminergic reward prediction errors are thought to reinforce previously rewarded actions, so they become easier to repeat. This dopaminergic activity could lead to a systematic bias by which rewarded actions are more readily learned than rewarded inactions. We present two findings. First, dopamine receptors in cortex, dorsal striatum, and nucleus accumbens provide distinct sources of variance in the human brain. Second, the boost in an individual’s learning rate from previously rewarded actions is dependent on the dopamine receptor density in dorsal striatum, a central structure in the dopaminergic circuit.

Learning to act to obtain reward and inhibit to avoid punishment is easier compared with learning the opposite contingencies. This coupling of action and valence is often thought of as a Pavlovian bias, although recent research has shown it may also emerge through instrumental mechanisms. We measured this learning bias with a rewarded go/no-go task in 60 adults of different ages. Using computational modeling, we characterized the bias as being instrumental. To assess the role of endogenous dopamine (DA) in the expression of this bias, we quantified DA D1 receptor availability using positron emission tomography (PET) with the radioligand [¹¹C]SCH23390. Using principal-component analysis on the binding potentials in a number of cortical and striatal regions of interest, we demonstrated that cortical, dorsal striatal, and ventral striatal areas provide independent sources of variance in DA D1 receptor availability. Interindividual variation in the dorsal striatal component was related to the strength of the instrumental bias during learning. These data suggest at least three anatomical sources of variance in DA D1 receptor availability separable using PET in humans, and we provide evidence that human dorsal striatal DA D1 receptors are involved in the modulation of instrumental learning biases.

Behavioral and Neural Manifestations of Reward Memory in Carriers of Low-Expressing versus High-Expressing Genetic Variants of the Dopamine D2 Receptor

Dopamine is critically important in the neural manifestation of motivated behavior, and alterations in the human dopaminergic system have been implicated in the etiology of motivation-related psychiatric disorders, most prominently addiction. Patients with chronic addiction exhibit reduced dopamine D2 receptor (DRD2) availability in the striatum, and the DRD2 TaqIA (rs1800497) and C957T (rs6277) genetic polymorphisms have previously been linked to individual differences in striatal dopamine metabolism and clinical risk for alcohol and nicotine dependence. Here, we investigated the hypothesis that the variants of these polymorphisms would show increased reward-related memory formation, which has previously been shown to jointly engage the mesolimbic dopaminergic system and the hippocampus, as a potential intermediate phenotype for addiction memory. To this end, we performed functional magnetic resonance imaging (fMRI) in 62 young, healthy individuals genotyped for DRD2 TaqIA and C957T variants. Participants performed an incentive delay task, followed by a recognition memory task 24 h later. We observed effects of both genotypes on the overall recognition performance with carriers of low-expressing variants, namely TaqIA A1 carriers and C957T C homozygotes, showing better performance than the other genotype groups. In addition to the better memory performance, C957T C homozygotes also exhibited a response bias for cues predicting monetary reward. At the neural level, the C957T polymorphism was associated with a genotype-related modulation of right hippocampal and striatal fMRI responses predictive of subsequent recognition confidence for reward-predicting items. Our results indicate that genetic variations associated with DRD2 expression affect explicit memory, specifically for rewarded stimuli. We suggest that the relatively better memory for rewarded stimuli in carriers of low-expressing DRD2 variants may reflect an intermediate phenotype of addiction memory.

The impact of common dopamine D2 receptor gene polymorphisms on D2/3 receptor availability: C957T as a key determinant in putamen and ventral striatum

Dopamine function is broadly implicated in multiple neuropsychiatric conditions believed to have a genetic basis. Although a few positron emission tomography (PET) studies have investigated the impact of single-nucleotide polymorphisms (SNPs) in the dopamine D2 receptor gene (DRD2) on D2/3 receptor availability (binding potential, BP_ND), these studies have often been limited by small sample size. Furthermore, the most commonly studied SNP in D2/3 BP_ND (Taq1A) is not located in the DRD2 gene itself, suggesting that its linkage with other DRD2 SNPs may explain previous PET findings. Here, in the largest PET genetic study to date (n=84), we tested for effects of the C957T and -141C Ins/Del SNPs (located within DRD2) as well as Taq1A on BP_ND of the high-affinity D2 receptor tracer ¹⁸F-Fallypride. In a whole-brain voxelwise analysis, we found a positive linear effect of C957T T allele status on striatal BP_ND bilaterally. The multilocus genetic scores containing C957T and one or both of the other SNPs produced qualitatively similar striatal results to C957T alone. The number of C957T T alleles predicted BP_ND in anatomically defined putamen and ventral striatum (but not caudate) regions of interest, suggesting some regional specificity of effects in the striatum. By contrast, no significant effects arose in cortical regions. Taken together, our data support the critical role of C957T in striatal D2/3 receptor availability. This work has implications for a number of psychiatric conditions in which dopamine signaling and variation in C957T status have been implicated, including schizophrenia and substance use disorders.

Acute effects of cocaine and cannabis on reversal learning as a function of COMT and DRD2 genotype

RATIONALE

Long-term cannabis and cocaine use has been associated with impairments in reversal learning. However, how acute cannabis and cocaine administration affect reversal learning in humans is not known.

OBJECTIVE

In this study, we aimed to establish the acute effects of administration of cannabis and cocaine on valence-dependent reversal learning as a function of DRD2 Taq1A (rs1800497) and COMT Val108/158Met (rs4680) genotype.

METHODS

A double-blind placebo-controlled randomized 3-way crossover design was used. Sixty-one regular poly-drug users completed a deterministic reversal learning task under the influence of cocaine, cannabis, and placebo that enabled assessment of both reward- and punishment-based reversal learning.

RESULTS

Proportion correct on the reversal learning task was increased by cocaine, but decreased by cannabis. Effects of cocaine depended on the DRD2 genotype, as increases in proportion correct were seen only in the A1 carriers, and not in the A2/A2 homozygotes. COMT genotype did not modulate drug-induced effects on reversal learning.

CONCLUSIONS

These data indicate that acute administration of cannabis and cocaine has opposite effects on reversal learning. The effects of cocaine, but not cannabis, depend on interindividual genetic differences in the dopamine D2 receptor gene.