A temporal difference account of avoidance learning

The CogLearn Toolkit for Unity: Validating a virtual reality paradigm for human avoidance learning

Avoidance learning encompasses the acquisition of behaviours that enable individuals to evade or withdraw from potentially harmful stimuli, prior to their occurrence. Maladaptive avoidance is a crucial feature of anxiety and trauma-related disorders. In biological and clinical settings, avoidance behaviours usually involve uninstructed, idiosyncratic and complex motor actions. However, there is a lack of laboratory paradigms that allow investigating how such actions are acquired. To fill this gap, we developed a wireless virtual reality platform to investigate avoidance learning in naturalistic settings, with an uncomfortable sound as unconditioned stimulus (US), a physically plausible avoidance action, and allowing for unconstrained movements. This platform, the CogLearn Toolkit for Unity, is publicly available and allows conducting various types of learning experiments with simple text files as input. We validated this platform in an exploration-confirmation approach with five independent experiments. Overall, participants showed successful acquisition of avoidance behaviour in all experiments. In three exploration experiments, we refined the paradigm and identified mean distance from US location during conditioned stimulus (CS) presentation (before US occurs) as a sensitive measure of avoidance. Two confirmation experiments revealed stronger avoidance for CS+ than CS- during avoidance learning, whether or not this phase was preceded by Pavlovian acquisition. Furthermore, we demonstrated reduced avoidance during extinction with instruction to approach CS, but persistent residual avoidance during this phase. We found evidence of reinstatement in one of two confirmation experiments. Overall, our study provides robust evidence supporting the efficacy of our paradigm in studying avoidance learning in conditions of high ecological relevance.

Slow-rising and fast-falling dopaminergic dynamics jointly adjust negative prediction error in the ventral striatum

The greater the reward expectations are, the more different the brain's physiological response will be. Although it is well-documented that better-than-expected outcomes are encoded quantitatively via midbrain dopaminergic (DA) activity, it has been less addressed experimentally whether worse-than-expected outcomes are expressed quantitatively as well. We show that larger reward expectations upon unexpected reward omissions are associated with the preceding slower rise and following larger decrease (DA dip) in the DA concentration at the ventral striatum of mice. We set up a lever press task on a fixed ratio (FR) schedule requiring five lever presses as an effort for a food reward (FR5). The mice occasionally checked the food magazine without a reward before completing the task. The percentage of this premature magazine entry (PME) increased as the number of lever presses approached five, showing rising expectations with increasing proximity to task completion, and hence greater reward expectations. Fibre photometry of extracellular DA dynamics in the ventral striatum using a fluorescent protein (genetically encoded GPCR activation-based DA sensor: GRABDA2m ) revealed that the slow increase and fast decrease in DA levels around PMEs were correlated with the PME percentage, demonstrating a monotonic relationship between the DA dip amplitude and degree of expectations. Computational modelling of the lever press task implementing temporal difference errors and state transitions replicated the observed correlation between the PME frequency and DA dip amplitude in the FR5 task. Taken together, these findings indicate that the DA dip amplitude represents the degree of reward expectations monotonically, which may guide behavioural adjustment.

When not responding to food changes food value: The role of timing

Establishing behavior change toward appetitive foods can be crucial to improve people's health. Food go/no-go training (GNG), in which people respond to some food items and not to other food items depending on the presentation of a go or no-go cue, is a means to establish behavior change. GNG changes the perceived value of food items and food consumption. After GNG, no-go items are rated as less attractive than go and/or untrained items, an empirical phenomenon called the NoGo-devaluation-effect. This effect is not always found, however. One theory-based explanation for these inconsistent results may be found in the timing of the go and no-go cues, which is also inconsistent across studies. Hence, in the present work we conducted two experiments to examine the possible role of go and no-go cue presentation timing in eliciting the NoGo-devaluation-effect. In Experiment 1, we presented the food items before the presentation of go/no-go cues, whereas we reversed this order in Experiment 2. As predicted, the NoGo-devaluation-effect was obtained in Experiment 1. This effect was absent in Experiment 2. Moreover, recognition memory for stimulus-action contingencies moderated the devaluation effect in Experiment 1, but not in Experiment 2. These results show that NoGo devaluation is dependent on the timing of the NoGo cue, which has theoretical and applied implications for understanding how and when go/no-go training influences food consumption. We propose that the value of food items is updated during go/no-go training to minimize prediction errors, and that this updating process is boosted by attention.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Impaired action-safety learning and excessive relief during avoidance in patients with anxiety disorders

Anxiety-related disorders are characterized by high levels of avoidance, but experimental research into avoidance learning in patients is scarce. To fill this gap, we compared healthy controls (HC, n = 47) with patients with obsessive-compulsive disorder (OCD, n = 33), panic disorder with agoraphobia (PDA, n = 40), and post-traumatic stress disorder (PTSD, n = 66) in a computer-based avoidance learning task, in order to examine (1) differences in rates of avoidance responses, (2) differences in action-safety learning during avoidance, and (3) differences in subjective relief following successful avoidance. The task comprised aversive negative pictures (unconditional stimulus, US) that followed pictures of two colored lamps (conditional stimuli, CS+), but not a third colored lamp (safety stimulus, CS-), and could be avoided by pressing a button during one CS+ (CS+ avoidable) but not the other (CS+ unavoidable). Participants rated their US-expectancy and level of relief on a trial-by-trial basis. Compared to the HC group, patient groups displayed higher levels of avoidance to the safety stimulus, and higher levels of US-expectancy and relief following the safety and avoidable danger stimulus. We propose that patients with anxiety disorders have low confidence in the safety consequences of avoidance actions, which induces increased relief during US omissions that reinforce the avoidance action.

Pain, from perception to action: A computational perspective

Pain is driven by sensation and emotion, and in turn, it motivates decisions and actions. To fully appreciate the multidimensional nature of pain, we formulate the study of pain within a closed-loop framework of sensory-motor prediction. In this closed-loop cycle, prediction plays an important role, as the interaction between prediction and actual sensory experience shapes pain perception and subsequently, action. In this Perspective, we describe the roles of two prominent computational theories-Bayesian inference and reinforcement learning-in modeling adaptive pain behaviors. We show that prediction serves as a common theme between these two theories, and that each of these theories can explain unique aspects of the pain perception-action cycle. We discuss how these computational theories and models can improve our mechanistic understandings of pain-centered processes such as anticipation, attention, placebo hypoalgesia, and pain chronification.

Why do valence asymmetries emerge in value learning? A reinforcement learning account

The Value Learning Task (VLT; e.g., Raymond & O'Brien, 2009) is widely used to investigate how acquired value impacts how we perceive and process stimuli. The task consists of a series of trials in which participants attempt to maximize accumulated winnings as they make choices from a pair of presented images associated with probabilistic win, loss, or no-change outcomes. The probabilities and outcomes are initially unknown to the participant and thus the task involves decision making and learning under uncertainty. Despite the symmetric outcome structure for win and loss pairs, people learn win associations better than loss associations (Lin, Cabrera-Haro, & Reuter-Lorenz, 2020). This learning asymmetry could lead to differences when the stimuli are probed in subsequent tasks, compromising inferences about how acquired value affects downstream processing. We investigate the nature of the asymmetry using a standard error-driven reinforcement learning model with a softmax choice rule. Despite having no special role for valence, the model yields the learning asymmetry observed in human behavior, whether the model parameters are set to maximize empirical fit, or task payoff. The asymmetry arises from an interaction between a neutral initial value estimate and a choice policy that exploits while exploring, leading to more poorly discriminated value estimates for loss stimuli. We also show how differences in estimated individual learning rates help to explain individual differences in the observed win-loss asymmetries, and how the final value estimates produced by the model provide a simple account of a post-learning explicit value categorization task.

Different brain systems support learning from received and avoided pain during human pain-avoidance learning

Both unexpected pain and unexpected pain absence can drive avoidance learning, but whether they do so via shared or separate neural and neurochemical systems is largely unknown. To address this issue, we combined an instrumental pain-avoidance learning task with computational modeling, functional magnetic resonance imaging (fMRI), and pharmacological manipulations of the dopaminergic (100 mg levodopa) and opioidergic (50 mg naltrexone) systems (N = 83). Computational modeling provided evidence that untreated participants learned more from received than avoided pain. Our dopamine and opioid manipulations negated this learning asymmetry by selectively increasing learning rates for avoided pain. Furthermore, our fMRI analyses revealed that pain prediction errors were encoded in subcortical and limbic brain regions, whereas no-pain prediction errors were encoded in frontal and parietal cortical regions. However, we found no effects of our pharmacological manipulations on the neural encoding of prediction errors. Together, our results suggest that human pain-avoidance learning is supported by separate threat- and safety-learning systems, and that dopamine and endogenous opioids specifically regulate learning from successfully avoided pain.

Temporal dynamics of fMRI signal changes during conditioned interoceptive pain-related fear and safety acquisition and extinction

Associative learning and memory mechanisms drive interoceptive signaling along the gut-brain axis, thus shaping affective-emotional reactions and behavior. Specifically, learning to predict potentially harmful, visceral pain is assumed to succeed within very few trials. However, the temporal dynamics of cerebellar and cerebral fMRI signal changes underlying early acquisition and extinction of learned fear signals and the concomitant evolvement of safety learning remain incompletely understood. 3T fMRI data of healthy individuals from three studies were uniformly processed across the whole brain and the cerebellum including an advanced normalizing method of the cerebellum. All studies employed differential delay conditioning (N=94) with one visual cue (CS⁺) being repeatedly paired with visceral pain as unconditioned stimulus (US) while a second cue remained unpaired (CS^-). During subsequent extinction (N=51), all CS were presented without US. Behavioral results revealed increased CS⁺-aversiveness and CS^--pleasantness after conditioning and diminished valence ratings for both CS following extinction. During early acquisition, the CS^- induced linearly increasing neural activation in the insula, midcingulate cortex, hippocampus, precuneus as well as cerebral and cerebellar somatomotor regions. The comparison between acquisition and extinction phases yielded a CS^--induced linear increase in the posterior cingulate cortex and precuneus during early acquisition, while there was no evidence for linear fMRI signal changes for the CS⁺ during acquisition and for both CS during extinction. Based on theoretical accounts of discrimination and temporal difference learning, these results suggest a gradual evolvement of learned safety cues that engage emotional arousal, memory, and cortical modulatory networks. As safety signals are presumably more difficult to learn and to discriminate from learned threat cues, the underlying temporal dynamics may reflect enhanced salience and prediction processing as well as increasing demands for attentional resources and the integration of multisensory information. Maladaptive responses to learned safety signals are a clinically relevant phenotype in multiple conditions, including chronic visceral pain, and can be exceptionally resistant to modification or extinction. Through sustained hypervigilance, safety seeking constitutes one key component in pain and stress-related avoidance behavior, calling for future studies targeting the mechanisms of safety learning and extinction to advance current cognitive-behavioral treatment approaches.

Reliability of web-based affective auditory stimulus presentation

Web-based experimental testing has seen exponential growth in psychology and cognitive neuroscience. However, paradigms involving affective auditory stimuli have yet to adapt to the online approach due to concerns about the lack of experimental control and other technical challenges. In this study, we assessed whether sounds commonly used to evoke affective responses in-lab can be used online. Using recent developments to increase sound presentation quality, we selected 15 commonly used sound stimuli and assessed their impact on valence and arousal states in a web-based experiment. Our results reveal good inter-rater and test-retest reliabilities, with results comparable to in-lab studies. Additionally, we compared a variety of previously used unpleasant stimuli, allowing us to identify the most aversive among these sounds. Our findings demonstrate that affective sounds can be reliably delivered through web-based platforms, which help facilitate the development of new auditory paradigms for affective online experiments.

Simulating the computational mechanisms of cognitive and behavioral psychotherapeutic interventions: insights from active inference

Cognitive-behavioral therapy (CBT) leverages interactions between thoughts, feelings, and behaviors. To deepen understanding of these interactions, we present a computational (active inference) model of CBT that allows formal simulations of interactions between cognitive interventions (i.e., cognitive restructuring) and behavioral interventions (i.e., exposure) in producing adaptive behavior change (i.e., reducing maladaptive avoidance behavior). Using spider phobia as a concrete example of maladaptive avoidance more generally, we show simulations indicating that when conscious beliefs about safety/danger have strong interactions with affective/behavioral outcomes, behavioral change during exposure therapy is mediated by changes in these beliefs, preventing generalization. In contrast, when these interactions are weakened, and cognitive restructuring only induces belief uncertainty (as opposed to strong safety beliefs), behavior change leads to generalized learning (i.e., "over-writing" the implicit beliefs about action-outcome mappings that directly produce avoidance). The individual is therefore equipped to face any new context, safe or dangerous, remaining in a content state without the need for avoidance behavior-increasing resilience from a CBT perspective. These results show how the same changes in behavior during CBT can be due to distinct underlying mechanisms; they predict lower rates of relapse when cognitive interventions focus on inducing uncertainty and on reducing the effects of automatic negative thoughts on behavior.

Avoidance learning as predictor of posttraumatic stress in firefighters

BACKGROUND

Avoidance is a well-established maintenance factor in anxiety-related psychopathology. Individuals prone to anxiety show more maladaptive avoidance responses in conditioning paradigms aimed at avoidance learning, which indicates impairments in safety learning. To what extent avoidance learning is associated with posttraumatic stress disorder (PTSD) is still unclear, despite the logical relevance to the symptomatology. In this prospective study, we investigate avoidance learning responses in first responders, a population at high risk for traumatic exposure and thus PTSD development, and studied whether avoidance learning was associated with concurrent and future PTSD symptoms.

METHOD

Firefighters (N = 502) performed an avoidance learning task at baseline assessment in which they first learned that two conditioned stimuli (CS+) were followed by an aversive stimulus (US) and one conditioned stimulus (CS-) was not. After that, they could learn to which CS avoidance of the US was effective, ineffective or unnecessary. Self-reported PTSD symptoms were assessed at baseline, and at 6, 12, 18 and 24 months.

RESULTS

Participants exhibited comparable avoidance patterns to low anxiety individuals from previous studies. Avoidance learning responses were not associated with PTSD symptoms at baseline nor at follow-up.

DISCUSSION

Our study found no evidence that avoidance learning was related to PTSD symptom severity in a high-risk, yet low symptomatic population, nor did it predict the development of PTSD symptoms at a later point in time. Future research should focus on studying avoidance learning in a clinical or high symptomatic sample to further clarify the role of avoidance learning in PTSD development.

Hierarchical models of pain: Inference, information-seeking, and adaptive control

Computational models of pain consider how the brain processes nociceptive information and allow mapping neural circuits and networks to cognition and behaviour. To date, they have generally have assumed two largely independent processes: perceptual inference, typically modelled as an approximate Bayesian process, and action control, typically modelled as a reinforcement learning process. However, inference and control are intertwined in complex ways, challenging the clarity of this distinction. Here, we consider how they may comprise a parallel hierarchical architecture that combines inference, information-seeking, and adaptive value-based control. This sheds light on the complex neural architecture of the pain system, and takes us closer to understanding from where pain 'arises' in the brain.

A reinforcement-learning model of active avoidance behavior: Differences between Sprague Dawley and Wistar-Kyoto rats

Avoidance behavior is a typically adaptive response performed by an organism to avert harmful situations. Individuals differ remarkably in their tendency to acquire and perform new avoidance behaviors, as seen in anxiety disorders where avoidance becomes pervasive and inappropriate. In rodent models of avoidance, the inbred Wistar-Kyoto (WKY) rat demonstrates increased learning and expression of avoidance compared to the outbred Sprague Dawley (SD) rat. However, underlying mechanisms that contribute to these differences are unclear. Computational modeling techniques can help identify factors that may not be easily decipherable from behavioral data alone. Here, we utilize a reinforcement learning (RL) model approach to better understand strain differences in avoidance behavior. An actor-critic model, with separate learning rates for action selection (in the actor) and state evaluation (in the critic), was applied to individual data of avoidance acquisition from a large cohort of WKY and SD rats. Latent parameters were extracted, such as learning rate and subjective reinforcement value of foot shock, that were then compared across groups. The RL model was able to accurately represent WKY and SD avoidance behavior, demonstrating that the model could simulate individual performance. The model determined that the perceived negative value of foot shock was significantly higher in WKY than SD rats, whereas learning rate in the actor was lower in WKY than SD rats. These findings demonstrate the utility of computational modeling in identifying underlying processes that could promote strain differences in behavioral performance.

Pain: A Precision Signal for Reinforcement Learning and Control

Since noxious stimulation usually leads to the perception of pain, pain has traditionally been considered sensory nociception. But its variability and sensitivity to a broad array of cognitive and motivational factors have meant it is commonly viewed as inherently imprecise and intangibly subjective. However, the core function of pain is motivational-to direct both short- and long-term behavior away from harm. Here, we illustrate that a reinforcement learning model of pain offers a mechanistic understanding of how the brain supports this, illustrating the underlying computational architecture of the pain system. Importantly, it explains why pain is tuned by multiple factors and necessarily supported by a distributed network of brain regions, recasting pain as a precise and objectifiable control signal.

Understanding the role of dopamine in conditioned and unconditioned fear

Pharmacological and molecular imaging studies in anxiety disorders have primarily focused on the serotonin system. In the meantime, dopamine has been known as the neurotransmitter of reward for 60 years, particularly for its action in the nervous terminals of the mesocorticolimbic system. Interest in the mediation by dopamine of the well-known brain aversion system has grown recently, particularly given recent evidence obtained on the role of D2 dopamine receptors in unconditioned fear. However, it has been established that excitation of the mesocorticolimbic pathway, originating from dopaminergic (DA) neurons from the ventral tegmental area (VTA), is relevant for the development of anxiety. Among the forebrain regions innervated by this pathway, the amygdala is an essential component of the neural circuitry of conditioned fear. Current findings indicate that the dopamine D2 receptor-signaling pathway connecting the VTA to the basolateral amygdala modulates fear and anxiety, whereas neural circuits in the midbrain tectum underlie the expression of innate fear. The A13 nucleus of the zona incerta is proposed as the origin of these DA neurons projecting to caudal structures of the brain aversion system. In this article we review data obtained in studies showing that DA receptor-mediated mechanisms on ascending or descending DA pathways play opposing roles in fear/anxiety processes. Dopamine appears to mediate conditioned fear by acting at rostral levels of the brain and regulate unconditioned fear at the midbrain level.

Pavlovian-instrumental interactions in active avoidance: The bark of neutral trials

In active avoidance tasks, subjects have to learn to execute particular actions in order to avoid an aversive stimulus, such as a shock. Such paradigms pose a number of psychological and neural enigmas, and so have attracted substantial computational interest. However, the ratio of conjecture to confirmation remains high. Here, we perform a theoretical inquiry into a recent experiment by Gentry, Lee, and Roesch ('Phasic dopamine release in the rat nucleus accumbens predicts approach and avoidance performance', Nat. Commun., 7:13154) who measured phasic dopamine concentrations in the nucleus accumbens core of rats whilst they avoided shocks, acquired food, or acted to gain no programmed outcome. These last, neutral, trials turned out to be a perfect probe for the workings of avoidance, partly because of the substantial differences between subjects and sessions revealed in the experiment. We suggest a way to interpret this probe, gaining support for opponency-, safety-, and Pavlovian-influenced treatments of avoidance.

Computation in Psychotherapy, or How Computational Psychiatry Can Aid Learning-Based Psychological Therapies

Learning-based therapies, such as cognitive-behavioral therapy, are used worldwide, and their efficacy is endorsed by health and research funding agencies. However, the mechanisms behind both their strengths and their weaknesses are inadequately understood. Here we describe how advances in computational modeling may help formalize and test hypotheses regarding how patients make inferences, which are core postulates of these therapies. Specifically, we highlight the relevance of computations with regard to the development, maintenance, and therapeutic change in psychiatric disorders. A Bayesian approach helps delineate which apparent inferential biases and aberrant beliefs are in fact near-normative, given patients' current concerns, and which are not. As examples, we formalize three hypotheses. First, high-level dysfunctional beliefs should be treated as beliefs over models of the world. There is a need to test how, and whether, people apply these high-level beliefs to guide the formation of lower level beliefs important for real-life decision making, conditional on their experiences. Second, during the genesis of a disorder, maladaptive beliefs grow because more benign alternative schemas are discounted during belief updating. Third, we propose that when patients learn within therapy but fail to benefit in real life, this can be accounted for by a mechanism that we term overaccommodation, similar to that used to explain fear reinstatement. Beyond these specifics, an ambitious collaborative research program between computational psychiatry researchers, therapists, and experts-by-experience needs to form testable predictions out of factors claimed to be important for therapy.

A Formal Valuation Framework for Emotions and Their Control

Computational psychiatry aims to apply mathematical and computational techniques to help improve psychiatric care. To achieve this, the phenomena under scrutiny should be within the scope of formal methods. As emotions play an important role across many psychiatric disorders, such computational methods must encompass emotions. Here, we consider formal valuation accounts of emotions. We focus on the fact that the flexibility of emotional responses and the nature of appraisals suggest the need for a model-based valuation framework for emotions. However, resource limitations make plain model-based valuation impossible and require metareasoning strategies to apportion cognitive resources adaptively. We argue that emotions may implement such metareasoning approximations by restricting the range of behaviors and states considered. We consider the processes that guide the deployment of the approximations, discerning between innate, model-free, heuristic, and model-based controllers. A formal valuation and metareasoning framework may thus provide a principled approach to examining emotions.

Vigour in active avoidance

It would be maladaptive to learn about catastrophes by trial and error alone. Investment in planning and effort are necessary. Devoting too many resources to averting disaster, however, can impair quality of life, as in anxiety and paranoia. Here, we developed a novel task to explore how people adjust effort expenditure (vigor) so as to avoid negative consequences. Our novel paradigm is immersive, enabling us to measure vigor in the context of (simulated) disaster. We found that participants (N = 118) exerted effort to avoid disaster-associated states, adjusting their effort expenditure according to the baseline probability of catastrophe, in agreement with theoretical predictions. Furthermore, negative subjective emotional states were associated both with threat level and with increasing vigor in the face of disaster. We describe for the first time effort expenditure in the context of irreversible losses, with important implications for disorders marked by excessive avoidance.

Modeling Trait Anxiety: From Computational Processes to Personality

Computational methods are increasingly being applied to the study of psychiatric disorders. Often, this involves fitting models to the behavior of individuals with subclinical character traits that are known vulnerability factors for the development of psychiatric conditions. Anxiety disorders can be examined with reference to the behavior of individuals high in "trait" anxiety, which is a known vulnerability factor for the development of anxiety and mood disorders. However, it is not clear how this self-report measure relates to neural and behavioral processes captured by computational models. This paper reviews emerging computational approaches to the study of trait anxiety, specifying how interacting processes susceptible to analysis using computational models could drive a tendency to experience frequent anxious states and promote vulnerability to the development of clinical disorders. Existing computational studies are described in the light of this perspective and appropriate targets for future studies are discussed.

Partial Adaptation of Obtained and Observed Value Signals Preserves Information about Gains and Losses

Given that the range of rewarding and punishing outcomes of actions is large but neural coding capacity is limited, efficient processing of outcomes by the brain is necessary. One mechanism to increase efficiency is to rescale neural output to the range of outcomes expected in the current context, and process only experienced deviations from this expectation. However, this mechanism comes at the cost of not being able to discriminate between unexpectedly low losses when times are bad versus unexpectedly high gains when times are good. Thus, too much adaptation would result in disregarding information about the nature and absolute magnitude of outcomes, preventing learning about the longer-term value structure of the environment. Here we investigate the degree of adaptation in outcome coding brain regions in humans, for directly experienced outcomes and observed outcomes. We scanned participants while they performed a social learning task in gain and loss blocks. Multivariate pattern analysis showed two distinct networks of brain regions adapt to the most likely outcomes within a block. Frontostriatal areas adapted to directly experienced outcomes, whereas lateral frontal and temporoparietal regions adapted to observed social outcomes. Critically, in both cases, adaptation was incomplete and information about whether the outcomes arose in a gain block or a loss block was retained. Univariate analysis confirmed incomplete adaptive coding in these regions but also detected nonadapting outcome signals. Thus, although neural areas rescale their responses to outcomes for efficient coding, they adapt incompletely and keep track of the longer-term incentives available in the environment.

SIGNIFICANCE STATEMENT

Optimal value-based choice requires that the brain precisely and efficiently represents positive and negative outcomes. One way to increase efficiency is to adapt responding to the most likely outcomes in a given context. However, too strong adaptation would result in loss of precise representation (e.g., when the avoidance of a loss in a loss-context is coded the same as receipt of a gain in a gain-context). We investigated an intermediate form of adaptation that is efficient while maintaining information about received gains and avoided losses. We found that frontostriatal areas adapted to directly experienced outcomes, whereas lateral frontal and temporoparietal regions adapted to observed social outcomes. Importantly, adaptation was intermediate, in line with influential models of reference dependence in behavioral economics.

Improving Accuracy and Temporal Resolution of Learning Curve Estimation for within- and across-Session Analysis

Estimation of learning curves is ubiquitously based on proportions of correct responses within moving trial windows. Thereby, it is tacitly assumed that learning performance is constant within the moving windows, which, however, is often not the case. In the present study we demonstrate that violations of this assumption lead to systematic errors in the analysis of learning curves, and we explored the dependency of these errors on window size, different statistical models, and learning phase. To reduce these errors in the analysis of single-subject data as well as on the population level, we propose adequate statistical methods for the estimation of learning curves and the construction of confidence intervals, trial by trial. Applied to data from an avoidance learning experiment with rodents, these methods revealed performance changes occurring at multiple time scales within and across training sessions which were otherwise obscured in the conventional analysis. Our work shows that the proper assessment of the behavioral dynamics of learning at high temporal resolution can shed new light on specific learning processes, and, thus, allows to refine existing learning concepts. It further disambiguates the interpretation of neurophysiological signal changes recorded during training in relation to learning.

Safety out of control: dopamine and defence

We enjoy a sophisticated understanding of how animals learn to predict appetitive outcomes and direct their behaviour accordingly. This encompasses well-defined learning algorithms and details of how these might be implemented in the brain. Dopamine has played an important part in this unfolding story, appearing to embody a learning signal for predicting rewards and stamping in useful actions, while also being a modulator of behavioural vigour. By contrast, although choosing correct actions and executing them vigorously in the face of adversity is at least as important, our understanding of learning and behaviour in aversive settings is less well developed. We examine aversive processing through the medium of the role of dopamine and targets such as D2 receptors in the striatum. We consider critical factors such as the degree of control that an animal believes it exerts over key aspects of its environment, the distinction between 'better' and 'good' actual or predicted future states, and the potential requirement for a particular form of opponent to dopamine to ensure proper calibration of state values.

A Computational Analysis of Aberrant Delay Discounting in Psychiatric Disorders

Impatience for reward is a facet of many psychiatric disorders. We draw attention to a growing literature finding greater discounting of delayed reward, an important aspect of impatience, across a range of psychiatric disorders. We propose these findings are best understood by considering the goals and motivation for discounting future reward. We characterize these as arising from either the opportunity costs of waiting or the uncertainty associated with delayed reward. We link specific instances of higher discounting in psychiatric disorder to heightened subjective estimates of either of these factors. We propose these costs are learned and represented based either on a flexible cognitive model of the world, an accumulation of previous experience, or through evolutionary specification. Any of these can be considered suboptimal for the individual if the resulting behavior results in impairments in personal and social functioning and/or in distress. By considering the neurochemical and neuroanatomical implementation of these processes, we illustrate how this approach can in principle unite social, psychological and biological conceptions of impulsive choice.

Prospective and Pavlovian mechanisms in aversive behaviour

Studying aversive behaviour is critical for understanding negative emotions and associated psychopathologies. However a comprehensive picture of the mechanisms underlying aversion is lacking, with associative learning theories focusing on Pavlovian reactions and decision-making theoretic approaches on prospective functions. We propose a computational model of aversion that combines goal-directed and Pavlovian forms of control into a unifying framework in which their relative importance is regulated by factors such as threat distance and controllability. Using simulations, we test whether the model can reproduce available empirical findings and discuss its relevance to understanding factors underlying negative emotions such as fear and anxiety. Furthermore, the specific method used to construct the model permits a natural mapping from its components to brain structure and function. Our model provides a basis for a unifying account of aversion that can guide empirical and interventional study contexts.

Contextual modulation of value signals in reward and punishment learning

Compared with reward seeking, punishment avoidance learning is less clearly understood at both the computational and neurobiological levels. Here we demonstrate, using computational modelling and fMRI in humans, that learning option values in a relative—context-dependent—scale offers a simple computational solution for avoidance learning. The context (or state) value sets the reference point to which an outcome should be compared before updating the option value. Consequently, in contexts with an overall negative expected value, successful punishment avoidance acquires a positive value, thus reinforcing the response. As revealed by post-learning assessment of options values, contextual influences are enhanced when subjects are informed about the result of the forgone alternative (counterfactual information). This is mirrored at the neural level by a shift in negative outcome encoding from the anterior insula to the ventral striatum, suggesting that value contextualization also limits the need to mobilize an opponent punishment learning system.

In contrast to predictions from learning theory, humans learn to seek rewards and avoid punishments equally well. Here the authors offer an elegant solution to this problem by demonstrating that humans learn option values relative to a reference point subserved by a common neural substrate.

Using signals associated with safety in avoidance learning: computational model of sex differences

Avoidance behavior involves learning responses that prevent upcoming aversive events; these responses typically extinguish when the aversive events stop materializing. Stimuli that signal safety from aversive events can paradoxically inhibit extinction of avoidance behavior. In animals, males and females process safety signals differently. These differences help explain why women are more likely to be diagnosed with an anxiety disorder and exhibit differences in symptom presentation and course compared to men. In the current study, we extend an existing model of strain differences in avoidance behavior to simulate sex differences in rats. The model successfully replicates data showing that the omission of a signal associated with a period of safety can facilitate extinction in females, but not males, and makes novel predictions that this effect should depend on the duration of the period, the duration of the signal itself, and its occurrence within that period. Non-reinforced responses during the safe period were also found to be important in the expression of these patterns. The model also allowed us to explore underlying mechanisms for the observed sex effects, such as whether safety signals serve as occasion setters for aversive events, to determine why removing them can facilitate extinction of avoidance. The simulation results argue against this account, and instead suggest the signal may serve as a conditioned reinforcer of avoidance behavior.

Testing the role of reward and punishment sensitivity in avoidance behavior: a computational modeling approach

Exaggerated avoidance behavior is a predominant symptom in all anxiety disorders and its degree often parallels the development and persistence of these conditions. Both human and non-human animal studies suggest that individual differences as well as various contextual cues may impact avoidance behavior. Specifically, we have recently shown that female sex and inhibited temperament, two anxiety vulnerability factors, are associated with greater duration and rate of the avoidance behavior, as demonstrated on a computer-based task closely related to common rodent avoidance paradigms. We have also demonstrated that avoidance is attenuated by the administration of explicit visual signals during "non-threat" periods (i.e., safety signals). Here, we use a reinforcement-learning network model to investigate the underlying mechanisms of these empirical findings, with a special focus on distinct reward and punishment sensitivities. Model simulations suggest that sex and inhibited temperament are associated with specific aspects of these sensitivities. Specifically, differences in relative sensitivity to reward and punishment might underlie the longer avoidance duration demonstrated by females, whereas higher sensitivity to punishment might underlie the higher avoidance rate demonstrated by inhibited individuals. Simulations also suggest that safety signals attenuate avoidance behavior by strengthening the competing approach response. Lastly, several predictions generated by the model suggest that extinction-based cognitive-behavioral therapies might benefit from the use of safety signals, especially if given to individuals with high reward sensitivity and during longer safe periods. Overall, this study is the first to suggest cognitive mechanisms underlying the greater avoidance behavior observed in healthy individuals with different anxiety vulnerabilities.

Absence of "Warm-Up" during Active Avoidance Learning in a Rat Model of Anxiety Vulnerability: Insights from Computational Modeling

Avoidance behaviors, in which a learned response causes omission of an upcoming punisher, are a core feature of many psychiatric disorders. While reinforcement learning (RL) models have been widely used to study the development of appetitive behaviors, less attention has been paid to avoidance. Here, we present a RL model of lever-press avoidance learning in Sprague-Dawley (SD) rats and in the inbred Wistar Kyoto (WKY) rat, which has been proposed as a model of anxiety vulnerability. We focus on “warm-up,” transiently decreased avoidance responding at the start of a testing session, which is shown by SD but not WKY rats. We first show that a RL model can correctly simulate key aspects of acquisition, extinction, and warm-up in SD rats; we then show that WKY behavior can be simulated by altering three model parameters, which respectively govern the tendency to explore new behaviors vs. exploit previously reinforced ones, the tendency to repeat previous behaviors regardless of reinforcement, and the learning rate for predicting future outcomes. This suggests that several, dissociable mechanisms may contribute independently to strain differences in behavior. The model predicts that, if the “standard” inter-session interval is shortened from 48 to 24 h, SD rats (but not WKY) will continue to show warm-up; we confirm this prediction in an empirical study with SD and WKY rats. The model further predicts that SD rats will continue to show warm-up with inter-session intervals as short as a few minutes, while WKY rats will not show warm-up, even with inter-session intervals as long as a month. Together, the modeling and empirical data indicate that strain differences in warm-up are qualitative rather than just the result of differential sensitivity to task variables. Understanding the mechanisms that govern expression of warm-up behavior in avoidance may lead to better understanding of pathological avoidance, and potential pathways to modify these processes.

From adversity to psychosis: pathways and mechanisms from specific adversities to specific symptoms

PURPOSE

Although there is considerable evidence that adversities in childhood such as social deprivation, sexual abuse, separation from parents, neglect and exposure to deviant parental communication are associated with psychosis in later life, most studies have considered broad diagnoses as outcomes. In this review we consider evidence for pathways between specific types of adversity and specific symptoms of psychosis.

METHODS

We present theoretical arguments for expecting some degree of specificity (although by no means perfect specificity) between different kinds of adversity and different symptoms of psychosis. We review studies that have investigated social-environmental risk factors for thought disorder, auditory-verbal hallucinations and paranoid delusions, and consider how these risk factors may impact on specific psychological and biological mechanisms.

RESULTS

Communication deviance in parents has been implicated in the development of thought disorder in offspring, childhood sexual abuse has been particularly implicated in auditory-verbal hallucinations, and attachment-disrupting events (e.g. neglect, being brought up in an institution) may have particular potency for the development of paranoid symptoms. Current research on psychological mechanisms underlying these symptoms suggests a number of symptom-specific mechanisms that may explain these associations.

CONCLUSIONS

Few studies have considered symptoms, underlying mechanisms and different kinds of adversity at the same time. Future research along these lines will have the potential to elucidate the mechanisms that lead to severe mental illness, and may have considerable clinical implications.

Action versus valence in decision making

•

Pavlovian responses couple action and valence.

•

This coupling interferes with instrumental learning and performance.

•

Action dominates valence in the striatum and dopaminergic midbrain.

•

Boosting dopamine enhances the dominance of action over valence in the striatum.

•

Boosting dopamine decreases the extent of the behavioral coupling between action and valence.

The selection of actions, and the vigor with which they are executed, are influenced by the affective valence of predicted outcomes. This interaction between action and valence significantly influences appropriate and inappropriate choices and is implicated in the expression of psychiatric and neurological abnormalities, including impulsivity and addiction. We review a series of recent human behavioral, neuroimaging, and pharmacological studies whose key design feature is an orthogonal manipulation of action and valence. These studies find that the interaction between the two is subject to the critical influence of dopamine. They also challenge existing views that neural representations in the striatum focus on valence, showing instead a dominance of the anticipation of action.

On the role of subsecond dopamine release in conditioned avoidance

Using shock avoidance procedures to study conditioned behavioral responses has a rich history within the field of experimental psychology. Such experiments led to the formulation of the general concept of negative reinforcement and specific theories attempting to explain escape and avoidance behavior, or why animals choose to either terminate or prevent the presentation of an aversive event. For example, the two-factor theory of avoidance holds that cues preceding an aversive event begin to evoke conditioned fear responses, and these conditioned fear responses reinforce the instrumental avoidance response. Current neuroscientific advances are providing new perspectives into this historical literature. Due to its well-established role in reinforcement processes and behavioral control, the mesolimbic dopamine system presented itself as a logical starting point in the search for neural correlates of avoidance and escape behavior. We recently demonstrated that phasic dopamine release events are inhibited by stimuli associated with aversive events but increased by stimuli preceding the successful avoidance of the aversive event. The latter observation is inconsistent with the second component of the two-factor theory of avoidance and; therefore, led us propose a new theoretical explanation of conditioned avoidance: (1) fear is initially conditioned to the warning signal and dopamine computes this fear association as a decrease in release, (2) the warning signal, now capable of producing a negative emotional state, suppresses dopamine release and behavior, (3) over repeated trials the warning signal becomes associated with safety rather than fear; dopaminergic neurons already compute safety as an increase in release and begin to encode the warning signal as the earliest predictor of safety (4) the warning signal now promotes conditioned avoidance via dopaminergic modulation of the brain's incentive-motivational circuitry.

Instrumental vigour in punishment and reward

Recent notions about the vigour of responding in operant conditioning suggest that the long-run average rate of reward should control the alacrity of action in cases in which the actual cost of speed is balanced against the opportunity cost of sloth. The average reward rate is suggested as being reported by tonic activity in the dopamine system and thereby influencing all actions, including ones that do not themselves lead directly to the rewards. This idea is syntactically problematical for the case of punishment. Here, we broaden the scope of the original suggestion, providing a two-factor analysis of obviated punishment in a variety of operant circumstances. We also consider the effects of stochastically successful actions, which turn out to differ rather markedly between appetitive and aversive cases. Finally, we study how to fit these ideas into nascent treatments that extend concepts of opponency between dopamine and serotonin from valence to invigoration.

Twenty-five lessons from computational neuromodulation

Neural processing faces three rather different, and perniciously tied, communication problems. First, computation is radically distributed, yet point-to-point interconnections are limited. Second, the bulk of these connections are semantically uniform, lacking differentiation at their targets that could tag particular sorts of information. Third, the brain's structure is relatively fixed, and yet different sorts of input, forms of processing, and rules for determining the output are appropriate under different, and possibly rapidly changing, conditions. Neuromodulators address these problems by their multifarious and broad distribution, by enjoying specialized receptor types in partially specific anatomical arrangements, and by their ability to mold the activity and sensitivity of neurons and the strength and plasticity of their synapses. Here, I offer a computationally focused review of algorithmic and implementational motifs associated with neuromodulators, using decision making in the face of uncertainty as a running example.

A model of amygdala-hippocampal-prefrontal interaction in fear conditioning and extinction in animals

Empirical research has shown that the amygdala, hippocampus, and ventromedial prefrontal cortex (vmPFC) are involved in fear conditioning. However, the functional contribution of each brain area and the nature of their interactions are not clearly understood. Here, we extend existing neural network models of the functional roles of the hippocampus in classical conditioning to include interactions with the amygdala and prefrontal cortex. We apply the model to fear conditioning, in which animals learn physiological (e.g. heart rate) and behavioral (e.g. freezing) responses to stimuli that have been paired with a highly aversive event (e.g. electrical shock). The key feature of our model is that learning of these conditioned responses in the central nucleus of the amygdala is modulated by two separate processes, one from basolateral amygdala and signaling a positive prediction error, and one from the vmPFC, via the intercalated cells of the amygdala, and signaling a negative prediction error. In addition, we propose that hippocampal input to both vmPFC and basolateral amygdala is essential for contextual modulation of fear acquisition and extinction. The model is sufficient to account for a body of data from various animal fear conditioning paradigms, including acquisition, extinction, reacquisition, and context specificity effects. Consistent with studies on lesioned animals, our model shows that damage to the vmPFC impairs extinction, while damage to the hippocampus impairs extinction in a different context (e.g., a different conditioning chamber from that used in initial training in animal experiments). We also discuss model limitations and predictions, including the effects of number of training trials on fear conditioning.

Action controls dopaminergic enhancement of reward representations

Dopamine is widely observed to signal anticipation of future rewards and thus thought to be a key contributor to affectively charged decision making. However, the experiments supporting this view have not dissociated rewards from the actions that lead to, or are occasioned by, them. Here, we manipulated dopamine pharmacologically and examined the effect on a task that explicitly dissociates action and reward value. We show that dopamine enhanced the neural representation of rewarding actions, without significantly affecting the representation of reward value as such. Thus, increasing dopamine levels with levodopa selectively boosted striatal and substantia nigra/ventral tegmental representations associated with actions leading to reward, but not with actions leading to the avoidance of punishment. These findings highlight a key role for dopamine in the generation of appetitively motivated actions.

Go and no-go learning in reward and punishment: interactions between affect and effect

Decision-making invokes two fundamental axes of control: affect or valence, spanning reward and punishment, and effect or action, spanning invigoration and inhibition. We studied the acquisition of instrumental responding in healthy human volunteers in a task in which we orthogonalized action requirements and outcome valence. Subjects were much more successful in learning active choices in rewarded conditions, and passive choices in punished conditions. Using computational reinforcement-learning models, we teased apart contributions from putatively instrumental and Pavlovian components in the generation of the observed asymmetry during learning. Moreover, using model-based fMRI, we showed that BOLD signals in striatum and substantia nigra/ventral tegmental area (SN/VTA) correlated with instrumentally learnt action values, but with opposite signs for go and no-go choices. Finally, we showed that successful instrumental learning depends on engagement of bilateral inferior frontal gyrus. Our behavioral and computational data showed that instrumental learning is contingent on overcoming inherent and plastic Pavlovian biases, while our neuronal data showed this learning is linked to unique patterns of brain activity in regions implicated in action and inhibition respectively.

Negative symptoms and the failure to represent the expected reward value of actions: behavioral and computational modeling evidence

CONTEXT

Negative symptoms are a core feature of schizophrenia, but their pathogenesis remains unclear. Negative symptoms are defined by the absence of normal function. However, there must be a productive mechanism that leads to this absence.

OBJECTIVE

To test a reinforcement learning account suggesting that negative symptoms result from a failure in the representation of the expected value of rewards coupled with preserved loss-avoidance learning.

DESIGN

Participants performed a probabilistic reinforcement learning paradigm involving stimulus pairs in which choices resulted in reward or in loss avoidance. Following training, participants indicated their valuation of the stimuli in a transfer test phase. Computational modeling was used to distinguish between alternative accounts of the data.

SETTING

A tertiary care research outpatient clinic.

PATIENTS

In total, 47 clinically stable patients with a diagnosis of schizophrenia or schizoaffective disorder and 28 healthy volunteers participated in the study. Patients were divided into a high-negative symptom group and a low-negative symptom group.

MAIN OUTCOME MEASURES

The number of choices leading to reward or loss avoidance, as well as performance in the transfer test phase. Quantitative fits from 3 different models were examined.

RESULTS

Patients in the high-negative symptom group demonstrated impaired learning from rewards but intact loss-avoidance learning and failed to distinguish rewarding stimuli from loss-avoiding stimuli in the transfer test phase. Model fits revealed that patients in the high-negative symptom group were better characterized by an "actor-critic" model, learning stimulus-response associations, whereas control subjects and patients in the low-negative symptom group incorporated expected value of their actions ("Q learning") into the selection process.

CONCLUSIONS

Negative symptoms in schizophrenia are associated with a specific reinforcement learning abnormality: patients with high-negative symptoms do not represent the expected value of rewards when making decisions but learn to avoid punishments through the use of prediction errors. This computational framework offers the potential to understand negative symptoms at a mechanistic level.

Roles of D1-like dopamine receptors in the nucleus accumbens and dorsolateral striatum in conditioned avoidance responses

RATIONALE

Aversively motivated learning is more poorly understood than appetitively motivated learning in many aspects, including the role of dopamine receptors in different regions of the striatum.

OBJECTIVES

The present study investigated the roles of the D1-like DA receptors in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) on learning and performance of conditioned avoidance responses (CARs).

METHODS

Adult male Wistar rats received intraperitoneal (i.p.), intra-NAc, or intra-DLS injections of the D1 dopamine receptor agonist SKF 81297 or the D1 receptor antagonist SCH 23390 20 min before or immediately after a training session in the CAR task two-way active avoidance, carried out 24 h before a test session.

RESULTS

Pre-training administration of SCH 23390, but not SKF 81297, caused a significant decrease in the number of CARs in the test, but not in the training session, when injected into the DLS, or in either session when injected into the NAc. It also caused a significant increase in the number of escape failures in the training session when injected into the NAc. Systemic administration caused a combination of these effects. Post-training administrations of these drugs caused no significant effect.

CONCLUSIONS

The results suggest that the D1-like receptors in the NAc and DLS play important, though different, roles in learning and performance of CAR.

Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain

The acquisition of reward and the avoidance of punishment could logically be contingent on either emitting or withholding particular actions. However, the separate pathways in the striatum for go and no-go appear to violate this independence, instead coupling affect and effect. Respect for this interdependence has biased many studies of reward and punishment, so potential action-outcome valence interactions during anticipatory phases remain unexplored. In a functional magnetic resonance imaging study with healthy human volunteers, we manipulated subjects' requirement to emit or withhold an action independent from subsequent receipt of reward or avoidance of punishment. During anticipation, in the striatum and a lateral region within the substantia nigra/ventral tegmental area (SN/VTA), action representations dominated over valence representations. Moreover, we did not observe any representation associated with different state values through accumulation of outcomes, challenging a conventional and dominant association between these areas and state value representations. In contrast, a more medial sector of the SN/VTA responded preferentially to valence, with opposite signs depending on whether action was anticipated to be emitted or withheld. This dominant influence of action requires an enriched notion of opponency between reward and punishment.

Social world interactions: how company connects to paranoia

BACKGROUND

Experimental studies have indicated that social contact, even when it is neutral, triggers paranoid thinking in people who score high on clinical or subclinical paranoia. We investigated whether contextual variables are predictive of momentary increases in the intensity of paranoid thinking in a sample of participants ranging across a psychometric paranoia continuum.

METHOD

The sample (n=154) consisted of 30 currently paranoid patients, 34 currently non-paranoid patients, 15 remitted psychotic patients, 38 high-schizotypy participants, and 37 control subjects. Based on their total score on Fenigstein's Paranoia Scale (PS), three groups with different degrees of paranoia were defined. The Experience Sampling Method (ESM), a structured diary technique, was used to assess momentary social context, perceived social threat and paranoia in daily life.

RESULTS

There were differences in the effect of social company on momentary levels of paranoia and perceived social threat across the range of trait paranoia. The low and medium paranoia groups reported higher levels of perceived social threat when they were with less-familiar compared to familiar individuals. The medium paranoia group reported more paranoia in less-familiar company. The high paranoia group reported no difference in the perception of social threat or momentary paranoia between familiar and unfamiliar contacts.

CONCLUSIONS

Paranoid thinking is context dependent in individuals with medium or at-risk levels of trait paranoia. Perceived social threat seems to be context dependent in the low paranoia group. However, at high levels of trait paranoia, momentary paranoia and momentary perceived social threat become autonomous and independent of social reality.

Are computational models of any use to psychiatry?

Mathematically rigorous descriptions of key hypotheses and theories are becoming more common in neuroscience and are beginning to be applied to psychiatry. In this article two fictional characters, Dr. Strong and Mr. Micawber, debate the use of such computational models (CMs) in psychiatry. We present four fundamental challenges to the use of CMs in psychiatry: (a) the applicability of mathematical approaches to core concepts in psychiatry such as subjective experiences, conflict and suffering; (b) whether psychiatry is mature enough to allow informative modelling; (c) whether theoretical techniques are powerful enough to approach psychiatric problems; and (d) the issue of communicating clinical concepts to theoreticians and vice versa. We argue that CMs have yet to influence psychiatric practice, but that they help psychiatric research in two fundamental ways: (a) to build better theories integrating psychiatry with neuroscience; and (b) to enforce explicit, global and efficient testing of hypotheses through more powerful analytical methods. CMs allow the complexity of a hypothesis to be rigorously weighed against the complexity of the data. The paper concludes with a discussion of the path ahead. It points to stumbling blocks, like the poor communication between theoretical and medical communities. But it also identifies areas in which the contributions of CMs will likely be pivotal, like an understanding of social influences in psychiatry, and of the co-morbidity structure of psychiatric diseases.

From reinforcement learning models to psychiatric and neurological disorders

Over the last decade and a half, reinforcement learning models have fostered an increasingly sophisticated understanding of the functions of dopamine and cortico-basal ganglia-thalamo-cortical (CBGTC) circuits. More recently, these models, and the insights that they afford, have started to be used to understand important aspects of several psychiatric and neurological disorders that involve disturbances of the dopaminergic system and CBGTC circuits. We review this approach and its existing and potential applications to Parkinson's disease, Tourette's syndrome, attention-deficit/hyperactivity disorder, addiction, schizophrenia and preclinical animal models used to screen new antipsychotic drugs. The approach's proven explanatory and predictive power bodes well for the continued growth of computational psychiatry and computational neurology.

Opponency revisited: competition and cooperation between dopamine and serotonin

Affective valence lies on a spectrum ranging from punishment to reward. The coding of such spectra in the brain almost always involves opponency between pairs of systems or structures. There is ample evidence for the role of dopamine in the appetitive half of this spectrum, but little agreement about the existence, nature, or role of putative aversive opponents such as serotonin. In this review, we consider the structure of opponency in terms of previous biases about the nature of the decision problems that animals face, the conflicts that may thus arise between Pavlovian and instrumental responses, and an additional spectrum joining invigoration to inhibition. We use this analysis to shed light on aspects of the role of serotonin and its interactions with dopamine.