Blocking of conditioning to a cocaine-paired stimulus: testing the hypothesis that cocaine perpetually produces a signal of larger-than-expected reward

Generalized cue reactivity in dopamine neurons after opioids

Cue reactivity is the maladaptive neurobiological and behavioral response upon exposure to drug cues and is a major driver of relapse. The leading hypothesis is that dopamine release by addictive drugs represents a persistently positive reward prediction error that causes runaway enhancement of dopamine responses to drug cues, leading to their pathological overvaluation compared to non-drug reward alternatives. However, this hypothesis has not been directly tested. Here we developed Pavlovian and operant procedures to measure firing responses, within the same dopamine neurons, to drug versus natural reward cues, which we found to be similarly enhanced compared to cues predicting natural rewards in drug-naïve controls. This enhancement was associated with increased behavioral reactivity to the drug cue, suggesting that dopamine release is still critical to cue reactivity, albeit not as previously hypothesized. These results challenge the prevailing hypothesis of cue reactivity, warranting new models of dopaminergic function in drug addiction, and provide critical insights into the neurobiology of cue reactivity with potential implications for relapse prevention.

Pavlovian learning and conditioned reinforcement

Conditioned reinforcers are widely used in applied behavior analysis. Basic research evidence reveals that Pavlovian learning plays an important role in the acquisition and efficacy of new conditioned-reinforcer functions. Thus, a better understanding of Pavlovian principles holds the promise of improving the efficacy of conditioned reinforcement in applied research and practice. This paper surveys how (and if) Pavlovian principles are presented in behavior-analytic textbooks; imprecisions and knowledge gaps within contemporary Pavlovian empirical findings are highlighted. Thereafter, six practical principles of Pavlovian conditioning are presented along with empirical support and knowledge gaps that should be filled by applied and translational behavior-analytic researchers. Innovative applications of these principles are outlined for research in language acquisition, token reinforcement, and self-control.

Computational models of drug use and addiction: A review

In this brief review, we describe current computational models of drug-use and addiction that fall into 2 broad categories: mathematically based models that rely on computational theories, and brain-based models that link computations to brain areas or circuits. Across categories, many are models of learning and decision-making, which may be compromised in addiction. Several mathematical models take predictive coding approaches, focusing on Bayesian prediction error. Other models focus on learning processes and (traditional) prediction error. Brain-based models have incorporated prefrontal cortex, basal ganglia, and the dopamine system, based on the effects of drugs on dopamine, motivation, and executive control circuits. Several models specifically describe how behavioral control may transition from habitual to goal-directed systems, consistent with computational accounts of compromised "model-based" control. Some brain-based models have linked this to the transition of behavioral control from ventral to dorsal striatum. Overall, we propose that while computational models capture some aspects of addiction and have advanced our thinking, most have focused on the effects of drug use rather than addiction per se, most have not been tested on and/or supported by human data, and few capture multiple stages and symptoms of addiction. We conclude by suggesting a path forward for computational models of addiction. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Addiction beyond pharmacological effects: The role of environment complexity and bounded rationality

Several decision-making vulnerabilities have been identified as underlying causes for addictive behaviours, or the repeated execution of stereotyped actions despite their adverse consequences. These vulnerabilities are mostly associated with brain alterations caused by the consumption of substances of abuse. However, addiction can also happen in the absence of a pharmacological component, such as seen in pathological gambling and videogaming. We use a new reinforcement learning model to highlight a previously neglected vulnerability that we suggest interacts with those already identified, whilst playing a prominent role in non-pharmacological forms of addiction. Specifically, we show that a dual-learning system (i.e. combining model-based and model-free) can be vulnerable to highly rewarding, but suboptimal actions, that are followed by a complex ramification of stochastic adverse effects. This phenomenon is caused by the overload of the capabilities of an agent, as time and cognitive resources required for exploration, deliberation, situation recognition, and habit formation, all increase as a function of the depth and richness of detail of an environment. Furthermore, the cognitive overload can be aggravated due to alterations (e.g. caused by stress) in the bounded rationality, i.e. the limited amount of resources available for the model-based component, in turn increasing the agent's chances to develop or maintain addictive behaviours. Our study demonstrates that, independent of drug consumption, addictive behaviours can arise in the interaction between the environmental complexity and the biologically finite resources available to explore and represent it.

Reward Prediction Errors in Drug Addiction and Parkinson's Disease: from Neurophysiology to Neuroimaging

PURPOSE OF REVIEW

Surprises are important sources of learning. Cognitive scientists often refer to surprises as "reward prediction errors," a parameter that captures discrepancies between expectations and actual outcomes. Here, we integrate neurophysiological and functional magnetic resonance imaging (fMRI) results addressing the processing of reward prediction errors and how they might be altered in drug addiction and Parkinson's disease.

RECENT FINDINGS

By increasing phasic dopamine responses, drugs might accentuate prediction error signals, causing increases in fMRI activity in mesolimbic areas in response to drugs. Chronic substance dependence, by contrast, has been linked with compromised dopaminergic function, which might be associated with blunted fMRI responses to pleasant non-drug stimuli in mesocorticolimbic areas. In Parkinson's disease, dopamine replacement therapies seem to induce impairments in learning from negative outcomes. The present review provides a holistic overview of reward prediction errors across different pathologies and might inform future clinical strategies targeting impulsive/compulsive disorders.

Computational psychiatry as a bridge from neuroscience to clinical applications

Translating advances in neuroscience into benefits for patients with mental illness presents enormous challenges because it involves both the most complex organ, the brain, and its interaction with a similarly complex environment. Dealing with such complexities demands powerful techniques. Computational psychiatry combines multiple levels and types of computation with multiple types of data in an effort to improve understanding, prediction and treatment of mental illness. Computational psychiatry, broadly defined, encompasses two complementary approaches: data driven and theory driven. Data-driven approaches apply machine-learning methods to high-dimensional data to improve classification of disease, predict treatment outcomes or improve treatment selection. These approaches are generally agnostic as to the underlying mechanisms. Theory-driven approaches, in contrast, use models that instantiate prior knowledge of, or explicit hypotheses about, such mechanisms, possibly at multiple levels of analysis and abstraction. We review recent advances in both approaches, with an emphasis on clinical applications, and highlight the utility of combining them.

Cocaine can generate a stronger conditioned reinforcer than food despite being a weaker primary reinforcer

The present study aimed to test the hypothesis that cues associated with drug-taking behavior become extra strong motivators of behavior compared with cues paired with non-drug reinforcers. In experiment 1, rats were trained to lever press for intravenous cocaine infusions and grain pellets. Each reinforcer was paired with a distinct audiovisual cue. When allowed to choose between these alternatives, rats chose grain on ~70-80 percent of trials. However, after extinguishing lever pressing, reintroduction of press-contingent cues during a test for cue-induced reinstatement generated more cocaine seeking than grain seeking (also observed on 3- and 8-week follow-up tests). To examine whether the same pattern of results would occur with two non-drug reinforcers, experiment 2 replicated experiment 1 using grain and sucrose as reinforcement alternatives. Rats chose sucrose over grain on ~70-80 percent of choice trials and also responded more for the sucrose cue than for the grain cue on the reinstatement test. The disconnect between primary and conditioned reinforcements in experiment 1 but not in experiment 2 suggests that drug cues may become exceptionally strong motivators of drug seeking. These results are consistent with cue-focused theories of addiction and may offer insight into the persistent cue-driven drug-seeking behavior observed in addiction.

Computational psychiatry

Psychiatric disorders such as autism and schizophrenia, arise from abnormalities in brain systems that underlie cognitive, emotional, and social functions. The brain is enormously complex and its abundant feedback loops on multiple scales preclude intuitive explication of circuit functions. In close interplay with experiments, theory and computational modeling are essential for understanding how, precisely, neural circuits generate flexible behaviors and their impairments give rise to psychiatric symptoms. This Perspective highlights recent progress in applying computational neuroscience to the study of mental disorders. We outline basic approaches, including identification of core deficits that cut across disease categories, biologically realistic modeling bridging cellular and synaptic mechanisms with behavior, and model-aided diagnosis. The need for new research strategies in psychiatry is urgent. Computational psychiatry potentially provides powerful tools for elucidating pathophysiology that may inform both diagnosis and treatment. To achieve this promise will require investment in cross-disciplinary training and research in this nascent field.

Nonhuman gamblers: lessons from rodents, primates, and robots

The search for neuronal and psychological underpinnings of pathological gambling in humans would benefit from investigating related phenomena also outside of our species. In this paper, we present a survey of studies in three widely different populations of agents, namely rodents, non-human primates, and robots. Each of these populations offers valuable and complementary insights on the topic, as the literature demonstrates. In addition, we highlight the deep and complex connections between relevant results across these different areas of research (i.e., cognitive and computational neuroscience, neuroethology, cognitive primatology, neuropsychiatry, evolutionary robotics), to make the case for a greater degree of methodological integration in future studies on pathological gambling.

Modelling individual differences in the form of Pavlovian conditioned approach responses: a dual learning systems approach with factored representations

Reinforcement Learning has greatly influenced models of conditioning, providing powerful explanations of acquired behaviour and underlying physiological observations. However, in recent autoshaping experiments in rats, variation in the form of Pavlovian conditioned responses (CRs) and associated dopamine activity, have questioned the classical hypothesis that phasic dopamine activity corresponds to a reward prediction error-like signal arising from a classical Model-Free system, necessary for Pavlovian conditioning. Over the course of Pavlovian conditioning using food as the unconditioned stimulus (US), some rats (sign-trackers) come to approach and engage the conditioned stimulus (CS) itself - a lever - more and more avidly, whereas other rats (goal-trackers) learn to approach the location of food delivery upon CS presentation. Importantly, although both sign-trackers and goal-trackers learn the CS-US association equally well, only in sign-trackers does phasic dopamine activity show classical reward prediction error-like bursts. Furthermore, neither the acquisition nor the expression of a goal-tracking CR is dopamine-dependent. Here we present a computational model that can account for such individual variations. We show that a combination of a Model-Based system and a revised Model-Free system can account for the development of distinct CRs in rats. Moreover, we show that revising a classical Model-Free system to individually process stimuli by using factored representations can explain why classical dopaminergic patterns may be observed for some rats and not for others depending on the CR they develop. In addition, the model can account for other behavioural and pharmacological results obtained using the same, or similar, autoshaping procedures. Finally, the model makes it possible to draw a set of experimental predictions that may be verified in a modified experimental protocol. We suggest that further investigation of factored representations in computational neuroscience studies may be useful.

Acquisition of responding with a remifentanil-associated conditioned reinforcer in the rat

RATIONALE

Drug-associated environmental stimuli may serve as conditioned reinforcers to enhance drug self-administration behaviors in humans and laboratory animals. However, it can be difficult to distinguish experimentally the conditioned reinforcing effects of a stimulus from other behavioral processes that can change rates of responding.

OBJECTIVES

To characterize the conditioned reinforcing effects of a stimulus paired with the μ-opioid agonist, remifentanil, using a new-response acquisition procedure in the rat.

METHODS

First, in Pavlovian conditioning (PAV) sessions, rats received response-independent IV injections of remifentanil and presentations of a light-noise compound stimulus. In paired PAV groups, injections and stimulus presentations always co-occurred. In random PAV control groups, injections and stimulus presentations occurred with no consistent relationship. Second, in instrumental acquisition (ACQ) sessions, all animals could respond in an active nose-poke that produced the stimulus alone or in an inactive nose-poke that had no scheduled consequences.

RESULTS

During ACQ, rats made significantly more active nose-pokes than inactive nose-pokes after paired PAV, but not after random PAV. Between groups, rats also made more active nose-pokes after paired PAV than after random PAV. After paired PAV, increased active responding was obtained under different schedules of reinforcement, persisted across multiple ACQ sessions, and depended on the number of PAV sessions conducted.

CONCLUSIONS

The remifentanil-paired stimulus served as a conditioned reinforcer for nose-poking: responding depended on both the contingency between the stimulus and remifentanil and the contingency between the nose-poke and the stimulus. Generally, new-response acquisition procedures may provide valid, flexible models for studying opioid-based conditioned reinforcement.

The error in total error reduction

Most models of human and animal learning assume that learning is proportional to the discrepancy between a delivered outcome and the outcome predicted by all cues present during that trial (i.e., total error across a stimulus compound). This total error reduction (TER) view has been implemented in connectionist and artificial neural network models to describe the conditions under which weights between units change. Electrophysiological work has revealed that the activity of dopamine neurons is correlated with the total error signal in models of reward learning. Similar neural mechanisms presumably support fear conditioning, human contingency learning, and other types of learning. Using a computational modeling approach, we compared several TER models of associative learning to an alternative model that rejects the TER assumption in favor of local error reduction (LER), which assumes that learning about each cue is proportional to the discrepancy between the delivered outcome and the outcome predicted by that specific cue on that trial. The LER model provided a better fit to the reviewed data than the TER models. Given the superiority of the LER model with the present data sets, acceptance of TER should be tempered.

Decision making: from neuroscience to psychiatry

Adaptive behaviors increase the likelihood of survival and reproduction and improve the quality of life. However, it is often difficult to identify optimal behaviors in real life due to the complexity of the decision maker's environment and social dynamics. As a result, although many different brain areas and circuits are involved in decision making, evolutionary and learning solutions adopted by individual decision makers sometimes produce suboptimal outcomes. Although these problems are exacerbated in numerous neurological and psychiatric disorders, their underlying neurobiological causes remain incompletely understood. In this review, theoretical frameworks in economics and machine learning and their applications in recent behavioral and neurobiological studies are summarized. Examples of such applications in clinical domains are also discussed for substance abuse, Parkinson's disease, attention-deficit/hyperactivity disorder, schizophrenia, mood disorders, and autism. Findings from these studies have begun to lay the foundations necessary to improve diagnostics and treatment for various neurological and psychiatric disorders.

Assessment of the impact of pattern of cocaine dosing schedule during conditioning and reconditioning on magnitude of cocaine CPP, extinction, and reinstatement

RATIONALE AND OBJECTIVE

We sought to examine the impact of differing cocaine administration schedules and dosing on the magnitude of cocaine conditioned place preference (CPP), extinction, and stress- and cocaine-induced reinstatement of CPP.

METHODS

First, in C57Bl/6J mice, we investigated whether total cocaine administration or pattern of drug exposure could influence the magnitude of cocaine CPP by conditioning mice with a fixed-low dose (FL; 7.5 mg/kg; total of 30 mg/kg), a fixed-high dose (FH; 16 mg/kg; total of 64 mg/kg), or an ascending dosing schedule (Asc; 2, 4, 8, and 16 mg/kg; total of 30 mg/kg). Next, we investigated if cocaine or saline is more effective at extinguishing preference by reconditioning mice with either a descending dosing schedule (Desc; 8, 4, 2, and 1 mg/kg) or saline. Finally, we examined if prior conditioning and reconditioning history alters stress (~2-3-min forced swim test) or cocaine-induced (3.5 mg/kg) reinstatement.

RESULTS

We replicated and extended findings by Itzhak and Anderson (Addict. Biol. 17(4): 706-16, 2011) demonstrating that Asc conditioning produces a greater CPP than either the FL or FH conditioning schedules. The magnitude of extinction expressed was similar in the Desc reconditioned and saline groups. Moreover, only the saline, and not the Desc reconditioned mice, showed stress and cocaine-induced reinstatement of CPP.

CONCLUSIONS

Our results suggest that the schedule of cocaine administration during conditioning and reconditioning can have a significant influence on the magnitude of CPP and extinction of preference and the ability of cocaine or a stressor to reinstate CPP.

Imbalanced decision hierarchy in addicts emerging from drug-hijacked dopamine spiraling circuit

Despite explicitly wanting to quit, long-term addicts find themselves powerless to resist drugs, despite knowing that drug-taking may be a harmful course of action. Such inconsistency between the explicit knowledge of negative consequences and the compulsive behavioral patterns represents a cognitive/behavioral conflict that is a central characteristic of addiction. Neurobiologically, differential cue-induced activity in distinct striatal subregions, as well as the dopamine connectivity spiraling from ventral striatal regions to the dorsal regions, play critical roles in compulsive drug seeking. However, the functional mechanism that integrates these neuropharmacological observations with the above-mentioned cognitive/behavioral conflict is unknown. Here we provide a formal computational explanation for the drug-induced cognitive inconsistency that is apparent in the addicts' “self-described mistake”. We show that addictive drugs gradually produce a motivational bias toward drug-seeking at low-level habitual decision processes, despite the low abstract cognitive valuation of this behavior. This pathology emerges within the hierarchical reinforcement learning framework when chronic exposure to the drug pharmacologically produces pathologicaly persistent phasic dopamine signals. Thereby the drug hijacks the dopaminergic spirals that cascade the reinforcement signals down the ventro-dorsal cortico-striatal hierarchy. Neurobiologically, our theory accounts for rapid development of drug cue-elicited dopamine efflux in the ventral striatum and a delayed response in the dorsal striatum. Our theory also shows how this response pattern depends critically on the dopamine spiraling circuitry. Behaviorally, our framework explains gradual insensitivity of drug-seeking to drug-associated punishments, the blocking phenomenon for drug outcomes, and the persistent preference for drugs over natural rewards by addicts. The model suggests testable predictions and beyond that, sets the stage for a view of addiction as a pathology of hierarchical decision-making processes. This view is complementary to the traditional interpretation of addiction as interaction between habitual and goal-directed decision systems.

Drug intake is sufficient, but conditioning is not necessary for the emergence of compulsive cocaine seeking after extended self-administration

Compulsive drug seeking, which is characterized by continued instrumental effort despite contingent punishment, has been shown to emerge after extended drug self-administration. Exactly what aspect of drug self-administration drives the appearance of addictive behavior is unclear, but the mechanistic explanations that have been offered differ in one key respect. On one hand, it has been suggested that dysfunctional conditioning during self-administration drives unrealistic reward expectations, ultimately producing resistance to punishment. If this is indeed the pathological process that drives compulsive behavior, then compulsivity should be apparent only in the presence of the pavlovian and instrumental stimuli that underwent frequent pairing with the drug reward. On the other hand, it has also been suggested that extended drug intake produces general changes to reward and decision-making circuits that manifest as compulsive drug seeking. Unfortunately, conditioning history and drug intake are generally intrinsically intertwined. However, here we used an animal model of compulsive cocaine seeking to selectively manipulate drug intake and the degree of conditioning in the test context, to investigate which of the two is more important for the emergence of compulsive cocaine seeking. The results show that extended drug intake alone is sufficient, but extended conditioning in the test context is not necessary for the emergence of compulsive cocaine seeking, resolving a fundamental question in addiction research.

Are computational models of any use to psychiatry?

Mathematically rigorous descriptions of key hypotheses and theories are becoming more common in neuroscience and are beginning to be applied to psychiatry. In this article two fictional characters, Dr. Strong and Mr. Micawber, debate the use of such computational models (CMs) in psychiatry. We present four fundamental challenges to the use of CMs in psychiatry: (a) the applicability of mathematical approaches to core concepts in psychiatry such as subjective experiences, conflict and suffering; (b) whether psychiatry is mature enough to allow informative modelling; (c) whether theoretical techniques are powerful enough to approach psychiatric problems; and (d) the issue of communicating clinical concepts to theoreticians and vice versa. We argue that CMs have yet to influence psychiatric practice, but that they help psychiatric research in two fundamental ways: (a) to build better theories integrating psychiatry with neuroscience; and (b) to enforce explicit, global and efficient testing of hypotheses through more powerful analytical methods. CMs allow the complexity of a hypothesis to be rigorously weighed against the complexity of the data. The paper concludes with a discussion of the path ahead. It points to stumbling blocks, like the poor communication between theoretical and medical communities. But it also identifies areas in which the contributions of CMs will likely be pivotal, like an understanding of social influences in psychiatry, and of the co-morbidity structure of psychiatric diseases.

From reinforcement learning models to psychiatric and neurological disorders

Over the last decade and a half, reinforcement learning models have fostered an increasingly sophisticated understanding of the functions of dopamine and cortico-basal ganglia-thalamo-cortical (CBGTC) circuits. More recently, these models, and the insights that they afford, have started to be used to understand important aspects of several psychiatric and neurological disorders that involve disturbances of the dopaminergic system and CBGTC circuits. We review this approach and its existing and potential applications to Parkinson's disease, Tourette's syndrome, attention-deficit/hyperactivity disorder, addiction, schizophrenia and preclinical animal models used to screen new antipsychotic drugs. The approach's proven explanatory and predictive power bodes well for the continued growth of computational psychiatry and computational neurology.

Females are more vulnerable to drug abuse than males: evidence from preclinical studies and the role of ovarian hormones

Human and animal research indicates the presence of sex differences in drug abuse. These data suggest that females, compared to males, are more vulnerable to key phases of the addiction process that mark transitions in drug use such as initiation, drug bingeing, and relapse. Recent data indicate that the female gonadal hormone estrogen may facilitate drug abuse in women. For example, phases of the menstrual cycle when estrogen levels are high are associated with enhanced positive subjective measures following cocaine and amphetamine administration in women. Furthermore, in animal research, the administration of estrogen increases drug taking and facilitates the acquisition, escalation, and reinstatement of cocaine-seeking behavior. Neurobiological data suggest that estrogen may facilitate drug taking by interacting with reward- and stress-related systems. This chapter discusses sex differences in and hormonal effects on drug-seeking behaviors in animal models of drug abuse. The neurobiological basis of these differences and effects are also discussed.

Learning that a cocaine reward is smaller than expected: A test of Redish's computational model of addiction

The present experiment tested the prediction of Redish's (2004) computational model of addiction that drug reward expectation continues to grow even when the received drug reward is smaller than expected. Initially, rats were trained to press two levers, each associated with a large dose of cocaine. Then, the dose associated with one of the levers was substantially reduced. Thus, when rats first pressed the reduced-dose lever, they expected a large cocaine reward, but received a small one. On subsequent choice tests, preference for the reduced-dose lever was reduced, demonstrating that rats learned to devalue the reduced-dose lever. The finding that rats learned to lower reward expectation when they received a smaller-than-expected cocaine reward is in opposition to the hypothesis that drug reinforcers produce a perpetual and non-correctable positive prediction error that causes the learned value of drug rewards to continually grow. Instead, the present results suggest that standard error-correction learning rules apply even to drug reinforcers.

A neurocomputational model for cocaine addiction

Based on the dopamine hypotheses of cocaine addiction and the assumption of decrement of brain reward system sensitivity after long-term drug exposure, we propose a computational model for cocaine addiction. Utilizing average reward temporal difference reinforcement learning, we incorporate the elevation of basal reward threshold after long-term drug exposure into the model of drug addiction proposed by Redish. Our model is consistent with the animal models of drug seeking under punishment. In the case of nondrug reward, the model explains increased impulsivity after long-term drug exposure. Furthermore, the existence of a blocking effect for cocaine is predicted by our model.