Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release

A Pavlovian, conditioned-reinforcement approach to reducing impulsive choice

Persistent impulsive choice, preference for a smaller-sooner over a larger-later reward, is associated with consequential life outcomes. Procedures that reduce nonhuman impulsive choice often have long training durations that reduce their translational utility. This experiment sought to alter the behavioral function of the stimulus rats encountered during the delay to the larger-later reward. That putatively aversive delay-signaling stimulus was given an appetitive function through Pavlovian conditioning. Forty Long-Evans rats (20 male) were randomly assigned to undergo Pavlovian training (lever-CS precedes food delivery by 8 s), and the other half underwent unpaired training (CS uncorrelated with food). During the test of impulsive choice that followed, choosing the larger-later reward produced 10-s access to the CS during the 20-s delay to that reward. Pavlovian training significantly increased larger-later reward choice relative to rats in the Unpaired group. The large effect size and ease of training are discussed in the context of potential translational research.

Using sign tracking to experimentally increase self-control in rats

Impulsive choice describes a preference for a smaller-sooner reward (SSR) over a larger-later reward (LLR). A large body of research has examined different procedures for decreasing impulsive choice in nonhuman subjects. One limitation of these procedures is the extensive training duration required to achieve the desired results. To address this limitation, the current experiment examined the effects of a brief course of Pavlovian training, designed to establish a conditioned stimulus (CS) that could be strategically used to encourage LLR choices. Forty male Long-Evans rats were randomly assigned to appetitive Pavlovian or unpaired training. A lever insertion signaled an upcoming unconditioned stimulus (i.e., food presentation) for Pavlovian rats and it acquired CS properties. The lever was uncorrelated with the US in the unpaired group, and it did not acquire CS properties. In the subsequent impulsive-choice assessment, the lever from the training phase served as the lever rats pressed to choose the LLR. After an LLR choice, the lever remained in the chamber during the delay to the LLR, just as the SSR lever remained in the chamber until that reward was delivered. Pavlovian-trained rats sign tracked toward the lever CS and made significantly fewer impulsive choices than did rats in the unpaired group.

Conditioning- and reward-related dendritic and presynaptic plasticity of nucleus accumbens neurons in male and female sign-tracker rats

For a subset of individuals known as sign-trackers, discrete Pavlovian cues associated with rewarding stimuli can acquire incentive properties and exert control over behaviour. Because responsiveness to cues is a feature of various neuropsychiatric conditions, rodent models of sign-tracking may prove useful for exploring the neurobiology of individual variation in psychiatric vulnerabilities. Converging evidence points towards the involvement of dopaminergic neurotransmission in the nucleus accumbens core (NAc) in the development of sign-tracking, yet whether this phenotype is associated with specific accumbal postsynaptic properties is unknown. Here, we examined dendritic spine structural organisation, as well as presynaptic and postsynaptic markers of activity, in the NAc core of male and female rats following a Pavlovian-conditioned approach procedure. In contrast to our prediction that cue re-exposure would increase spine density, experiencing the discrete lever-cue without reward delivery resulted in lower spine density than control rats for which the lever was unpaired with reward during training; this effect was tempered in the most robust sign-trackers. Interestingly, this same behavioural test (lever presentation without reward) resulted in increased levels of a marker of presynaptic activity (synaptophysin), and this effect was greatest in female rats. Whilst some behavioural differences were observed in females during initial Pavlovian training, final conditioning scores did not differ from males and were unaffected by the oestrous cycle. This work provides novel insights into how conditioning impacts the neuronal plasticity of the NAc core, whilst highlighting the importance of studying the behaviour and neurobiology of both male and female rats.

Variation in the effectiveness of reinforcement and nonreinforcement in generating different conditioned behaviors

Rat autoshaping procedures generate two readily measurable conditioned responses: During lever presentations that have previously signaled food, rats approach the food well (called goal-tracking) and interact with the lever itself (called sign-tracking). We investigated how reinforced and nonreinforced trials affect the overall and temporal distributions of these two responses across 10-second lever presentations. In two experiments, reinforced trials generated more goal-tracking than sign-tracking, and nonreinforced trials resulted in a larger reduction in goal-tracking than sign-tracking. The effect of reinforced trials was evident as an increase in goal-tracking and reduction in sign-tracking across the duration of the lever presentations, and nonreinforced trials resulted in this pattern transiently reversing and then becoming less evident with further training. These dissociations are consistent with a recent elaboration of the Rescorla-Wagner model, HeiDI (Honey, R.C., Dwyer, D.M., & Iliescu, A.F. (2020a). HeiDI: A model for Pavlovian learning and performance with reciprocal associations. Psychological Review, 127, 829-852.), a model in which responses related to the nature of the unconditioned stimulus (e.g., goal-tracking) have a different origin than those related to the nature of the conditioned stimulus (e.g., sign-tracking).

Human Brain Project Partnering Projects Meeting: Status Quo and Outlook

As the European Flagship Human Brain Project (HBP) ends in September 2023, a meeting dedicated to the Partnering Projects (PPs), a collective of independent research groups that partnered with the HBP, was held on September 4-7, 2022. The purpose of this meeting was to allow these groups to present their results, reflect on their collaboration with the HBP and discuss future interactions with the European Research Infrastructure (RI) EBRAINS that has emerged from the HBP. In this report, we share the tour-de-force that the Partnering Projects that were present in the meeting have made in furthering knowledge concerning various aspects of Brain Research with the HBP. We describe briefly major achievements of the HBP Partnering Projects in terms of a systems-level understanding of the functional architecture of the brain and its possible emulation in artificial systems. We then recapitulate open discussions with EBRAINS representatives about the evolution of EBRAINS as a sustainable Research Infrastructure for the Partnering Projects after the HBP, and also for the wider scientific community.

Effects of inter-trial interval on sign-tracking and conditioned reinforcer efficacy in female rats

Previous nonhuman studies have reported that sign-tracking to a conditioned stimulus (CS) is increased when the intertrial interval (ITI) duration is increased. Separate studies indicate that individual differences in sign-tracking (vs. goal-tracking) at a fixed ITI (and CS duration) is predictive of the conditioned reinforcer efficacy of the CS. The present study evaluates, for the first time, if increasing the ITI increases rats' sign-tracking and the conditioned reinforcing efficacy of the CS. Forty-five female rats were randomly assigned to one of three groups that completed appetitive Pavlovian training with ITIs of 14, 24, or 96 s. Subsequently, they completed tests of conditioned reinforcement. Replicating previous findings, longer ITIs increased sign-tracking to a lever-CS and, extending the literature, conditioned reinforcer efficacy of that CS was highest at the longest ITI used during Pavlovian training. Implications for behavioral interventions using conditioned reinforcement are discussed.

Pavlovian learning and conditioned reinforcement

Conditioned reinforcers are widely used in applied behavior analysis. Basic research evidence reveals that Pavlovian learning plays an important role in the acquisition and efficacy of new conditioned-reinforcer functions. Thus, a better understanding of Pavlovian principles holds the promise of improving the efficacy of conditioned reinforcement in applied research and practice. This paper surveys how (and if) Pavlovian principles are presented in behavior-analytic textbooks; imprecisions and knowledge gaps within contemporary Pavlovian empirical findings are highlighted. Thereafter, six practical principles of Pavlovian conditioning are presented along with empirical support and knowledge gaps that should be filled by applied and translational behavior-analytic researchers. Innovative applications of these principles are outlined for research in language acquisition, token reinforcement, and self-control.

Dopamine in the dorsal bed nucleus of stria terminalis signals Pavlovian sign-tracking and reward violations

Midbrain and striatal dopamine signals have been extremely well characterized over the past several decades, yet novel dopamine signals and functions in reward learning and motivation continue to emerge. A similar characterization of real-time sub-second dopamine signals in areas outside of the striatum has been limited. Recent advances in fluorescent sensor technology and fiber photometry permit the measurement of dopamine binding correlates, which can divulge basic functions of dopamine signaling in non-striatal dopamine terminal regions, like the dorsal bed nucleus of the stria terminalis (dBNST). Here, we record GRABDA signals in the dBNST during a Pavlovian lever autoshaping task. We observe greater Pavlovian cue-evoked dBNST GRABDA signals in sign-tracking (ST) compared to goal-tracking/intermediate (GT/INT) rats and the magnitude of cue-evoked dBNST GRABDA signals decreases immediately following reinforcer-specific satiety. When we deliver unexpected rewards or omit expected rewards, we find that dBNST dopamine signals encode bidirectional reward prediction errors in GT/INT rats, but only positive prediction errors in ST rats. Since sign- and goal-tracking approach strategies are associated with distinct drug relapse vulnerabilities, we examined the effects of experimenter-administered fentanyl on dBNST dopamine associative encoding. Systemic fentanyl injections do not disrupt cue discrimination but generally potentiate dBNST dopamine signals. These results reveal multiple dBNST dopamine correlates of learning and motivation that depend on the Pavlovian approach strategy employed.

Conditioned approach behavior of SHR and SD rats during Pavlovian conditioning

Individual differences in reward-related learning are relevant to many behavioral disorders. Sensory cues that predict reward can become incentive stimuli that adaptively support behavior, or alternatively, cause maladaptive behaviors. The spontaneously hypertensive rat (SHR) expresses a genetically determined elevated sensitivity to delay of reward, and has been extensively studied as a behavioral model for attention deficit hyperactivity disorder (ADHD). We investigated reward-related learning in the SHR, comparing them to Sprague-Dawley (SD) rats as a reference strain. A standard Pavlovian conditioned approach task was used, in which a lever cue was followed by reward. Lever presses could occur while the lever was extended, but had no effect on reward delivery. The behavior of both the SHRs and the SD rats showed that they learnt that the lever cue predicted reward. However, the pattern of behavior differed between the strains. During lever cue presentation, SD rats pressed the lever more often and made fewer magazine entries than SHRs. When lever contacts that did not result in lever presses were analyzed, there was no significant difference between SHRs and SDs. These results suggest that the SHRs attributed less incentive value to the conditioned stimulus than the SD rats. During the presentation of the conditioned cue, cue directed responses are called sign tracking responses, whereas responses directed towards the food magazine are called goal tracking responses. Analysis of behavior using a standard Pavlovian conditioned approach index to quantify sign and goal tracking tendencies showed that both strains had a tendency towards goal tracking in this task. However, the SHRs showed a significantly greater goal tracking tendency than the SD rats. Taken together, these findings suggest that attribution of incentive value to reward predicting cues is attenuated in SHRs, which might explain their elevated sensitivity to delay of reward.

The propensity to sign-track is associated with externalizing behavior and distinct patterns of reward-related brain activation in youth

Externalizing behaviors in childhood often predict impulse control disorders in adulthood; however, the underlying bio-behavioral risk factors are incompletely understood. In animals, the propensity to sign-track, or the degree to which incentive motivational value is attributed to reward cues, is associated with externalizing-type behaviors and deficits in executive control. Using a Pavlovian conditioned approach paradigm, we quantified sign-tracking in 40 healthy 9-12-year-olds. We also measured parent-reported externalizing behaviors and anticipatory neural activations to outcome-predicting cues using the monetary incentive delay fMRI task. Sign-tracking was associated with attentional and inhibitory control deficits and the degree of amygdala, but not cortical, activation during reward anticipation. These findings support the hypothesis that youth with a propensity to sign-track are prone to externalizing tendencies, with an over-reliance on subcortical cue-reactive brain systems. This research highlights sign-tracking as a promising experimental approach delineating the behavioral and neural circuitry of individuals at risk for externalizing disorders.

Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

Nicotine Enhances Goal-Tracking in Ethanol and Food Pavlovian Conditioned Approach Paradigms

Rationale

Nicotine promotes alcohol intake through pharmacological and behavioral interactions. As an example of the latter, nicotine can facilitate approach toward food- and alcohol-associated stimuli ("sign-tracking") in lever-Pavlovian conditioned approach (PavCA) paradigms. However, we recently reported that nicotine can also enhance approach toward locations of reward delivery ("goal-tracking") triggered by ethanol-predictive stimuli when the location of ethanol delivery is non-static (i.e., a retractable sipper bottle).

Objective

To determine whether the non-static nature of the reward location could have biased the development of goal-tracking in our previous study (Loney et al., 2019); we assessed the effect of nicotine in a lever-PavCA paradigm wherein the location of ethanol delivery was static (i.e., a stationary liquid receptacle). Then, to determine whether nicotine's enhancement of goal-tracking is unique to ethanol-predictive stimuli, we assessed the effect of systemic nicotine on approach triggered by food-predictive stimuli in a lever-PavCA paradigm.

Methods

Long-Evans rats were used in two PavCA experiments wherein a lever predicted the receipt of ethanol (15% vol/vol; experiment 1) or food (experiment 2) into a stationary receptacle. Prior to testing, rats were administered nicotine (0.4 mg/kg subcutaneously) or saline systemically.

Results

In both experiments, nicotine increased measures of goal-tracking, but not sign-tracking.

Conclusion

Nicotine can facilitate approach to reward locations without facilitating approach to reward-predictive stimuli. As such, conceptualization of the mechanisms by which nicotine affects behavior must be expanded to explain an enhancement of goal-tracking by nicotine.

Interpreting the role of the striatum during multiple phases of motor learning

The synaptic pathways in the striatum are central to basal ganglia functions including motor control, learning and organization, action selection, acquisition of motor skills, cognitive function, and emotion. Here, we review the role of the striatum and its connections in motor learning and performance. The development of new techniques to record neuronal activity and animal models of motor disorders using neurotoxin, pharmacological, and genetic manipulations are revealing pathways that underlie motor performance and motor learning, as well as how they are altered by pathophysiological mechanisms. We discuss approaches that can be used to analyze complex motor skills, particularly in rodents, and identify specific questions central to understanding how striatal circuits mediate motor learning.

Instrumental and Pavlovian Mechanisms in Alcohol Use Disorder

Purpose of Review

Current theories of alcohol use disorders (AUD) highlight the importance of Pavlovian and instrumental learning processes mainly based on preclinical animal studies. Here, we summarize available evidence for alterations of those processes in human participants with AUD with a focus on habitual versus goal-directed instrumental learning, Pavlovian conditioning, and Pavlovian-to-instrumental transfer (PIT) paradigms.

Recent Findings

The balance between habitual and goal-directed control in AUD participants has been studied using outcome devaluation or sequential decision-making procedures, which have found some evidence of reduced goal-directed/model-based control, but little evidence for stronger habitual responding. The employed Pavlovian learning and PIT paradigms have shown considerable differences regarding experimental procedures, e.g., alcohol-related or conventional reinforcers or stimuli.

Summary

While studies of basic learning processes in human participants with AUD support a role of Pavlovian and instrumental learning mechanisms in the development and maintenance of drug addiction, current studies are characterized by large variability regarding methodology, sample characteristics, and results, and translation from animal paradigms to human research remains challenging. Longitudinal approaches with reliable and ecologically valid paradigms of Pavlovian and instrumental processes, including alcohol-related cues and outcomes, are warranted and should be combined with state-of-the-art imaging techniques, computational approaches, and ecological momentary assessment methods.

Behavioral determinants in the expression of the Kamin blocking effect: Implications for associative learning theory

Associative learning makes important contributions to our behavior and decisions. The Kamin blocking effect is an associative learning phenomenon that plays a central role in understanding of the psychological principles underlying associative learning. However, several recent failures to replicate the blocking effect suggest that the conditions necessary for blocking are poorly understood. To understand the conditions necessary for blocking, here we review studies into the expression of blocking in subjects that either approach and interact with the conditioned cue (sign trackers) or approach and interact with the reward location (goal trackers) during appetitive classical conditioning. Psychological theory and the neurophysiological correlates of appetitive classical conditioning make opposing predictions regarding the expression of blocking in sign and goal trackers. We reconcile these opposing predictions in a qualitative model using two parallel learning processes. Such models offer a better framework for understanding the psychological associative structures acquired during learning, their interactions contributing to the conditioned response, and how they affect subsequent learning and the expression of the Kamin blocking effect.

Individual variation in the vigor and form of Pavlovian conditioned responses: Analysis of a model system

Pavlovian conditioning results in individual variation in the vigor and form of acquired behaviors. Here, we describe a general-process model of associative learning (HeiDI; How excitation and inhibition determine ideo-motion) that provides an analysis for such variation together with a range of other important group-level phenomena. The model takes as its starting point the idea that pairings of a conditioned stimulus (CS) and an unconditioned stimulus (US) result in the formation of reciprocal associations between their central representations. The asymptotic values of these associations and the rate at which these are reached are held to be influenced by the perceived salience of the CS (αCS) and US (βUS). Importantly, whether this associative knowledge is exhibited in behavior that reflects the properties of the CS (e.g., sign-tracking) or US (e.g., goal-tracking) is also influenced by the relative values of αCS and βUS. In this way, HeiDI provides an analysis for both quantitative and qualitative individual differences generated by Pavlovian conditioning procedures.

Context-Dependent Multiplexing by Individual VTA Dopamine Neurons

Dopamine (DA) neurons of the VTA track cues and rewards to generate a reward prediction error signal during Pavlovian conditioning. Here we explored how these neurons respond to a self-paced, operant task in freely moving mice. The animal could trigger a reward-predicting cue by remaining in a specific location of an operant box for a brief time before moving to a spout for reward collection. VTA DA neurons were identified using DAT-Cre male mice that carried an optrode with minimal impact on the behavioral task. In vivo single-unit recordings revealed transient fast spiking responses to the cue and reward in correct trials, while for incorrect ones the activity paused, reflecting positive and negative error signals of a reward prediction. In parallel, a majority of VTA DA neurons simultaneously encoded multiple actions (e.g., movement velocity, acceleration, distance to goal, and licking) in sustained slow firing modulation. Applying a GLM, we show that such multiplexed encoding of rewarding and motor variables by individual DA neurons was only apparent while the mouse was engaged in the task. Downstream targets may exploit such goal-directed multiplexing of VTA DA neurons to adjust actions to optimize the task's outcome.SIGNIFICANCE STATEMENT VTA DA neurons code for multiple functions, including the reward prediction error but also motivation and locomotion. Here we show that about half of the recorded VTA DA neurons perform multiplexing: they exploit the phasic and tonic activity modes to encode, respectively, the cue/reward responses and motor parameters, most prominently when the mouse engages in a self-paced operand task. VTA non-DA neurons, by contrast, encode motor parameters regardless of task engagement.

Mapping sign-tracking and goal-tracking onto human behaviors

As evidenced through classic Pavlovian learning mechanisms, environmental cues can become incentivized and influence behavior. These stimulus-outcome associations are relevant in everyday life but may be particularly important for the development of impulse control disorders including addiction. Rodent studies have elucidated specific learning profiles termed 'sign-tracking' and 'goal-tracking' which map onto individual differences in impulsivity and other behaviors associated with impulse control disorders' etiology, course, and relapse. Whereas goal-trackers are biased toward the outcome, sign-trackers fixate on features that are associated with but not necessary for achieving an outcome; a pattern of behavior that often leads to escalation of reward-seeking that can be maladaptive. The vast majority of the sign- and goal-tracking research has been conducted using rodent models and very few have bridged this concept into the domain of human behavior. In this review, we discuss the attributes of sign- and goal-tracking profiles, how these are manifested neurobiologically, and how these distinct learning styles could be an important tool for clinical interventions in human addiction.

Broadening the scope of PLOS Biology: Short Reports and Methods and Resources

This Editorial discusses the creation of two new scope-broadening article types for PLOS Biology — Methods & Resources articles and Short Reports — acknowledging that important advances in biology come in many shapes and sizes.

Impacts of inter-trial interval duration on a computational model of sign-tracking vs. goal-tracking behaviour

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53-57, 2011). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results.

Hippocampal replays under the scrutiny of reinforcement learning models

Multiple in vivo studies have shown that place cells from the hippocampus replay previously experienced trajectories. These replays are commonly considered to mainly reflect memory consolidation processes. Some data, however, have highlighted a functional link between replays and reinforcement learning (RL). This theory, extensively used in machine learning, has introduced efficient algorithms and can explain various behavioral and physiological measures from different brain regions. RL algorithms could constitute a mechanistic description of replays and explain how replays can reduce the number of iterations required to explore the environment during learning. We review the main findings concerning the different hippocampal replay types and the possible associated RL models (either model-based, model-free, or hybrid model types). We conclude by tying these frameworks together. We illustrate the link between data and RL through a series of model simulations. This review, at the frontier between informatics and biology, paves the way for future work on replays.

Learning what to approach

Most decisions share a common goal: maximize reward and minimize punishment. Achieving this goal requires learning which choices are likely to lead to favorable outcomes. Dopamine is essential for this process, enabling learning by signaling the difference between what we expect to get and what we actually get. Although all animals appear to use this dopamine prediction error circuit, some do so more than others, and this neural heterogeneity correlates with individual variability in behavior. In this issue of PLOS Biology, Lee and colleagues show that manipulating a simple task parameter can bias the animals’ behavioral strategy and modulate dopamine release, implying that how we learn is just as flexible as what we learn.