Dopamine selectively remediates 'model-based' reward learning: a computational approach

Understanding disrupted motivation in Parkinson's disease through a value-based decision-making lens

Neurobehavioural disturbances such as loss of motivation have profound effects on the lives of many people living with Parkinson's disease (PD), as well as other brain disorders. The field of decision-making neuroscience, underpinned by a plethora of work across species, provides an important framework within which to investigate apathy in clinical populations. Here we review how changes in a number of different processes underlying value-based decision making may lead to the common phenotype of apathy in PD. The application of computational models to probe both behaviour and neurophysiology show promise in elucidating these cognitive processes crucial for motivated behaviour. However, observations from the clinical management of PD demand an expanded view of this relationship, which we aim to delineate. Ultimately, effective treatment of apathy may depend on identifying the pattern in which decision making and related mechanisms have been disrupted in individuals living with PD.

Exploring Habits in Anorexia Nervosa: Promise, Pitfalls, and Progress

PURPOSE OF REVIEW

Habits, characterized by automaticity and insensitivity to outcomes, may be key to the persistence of maladaptive behaviors in anorexia nervosa (AN). This review examines the status of habit research in AN, focusing on insights from task-based assessments.

RECENT FINDINGS

Findings indicate dysfunction in the frontostriatal circuits associated with habitual and goal-directed behaviors, with some studies linking neural disturbances to habit measures or clinically relevant behaviors. Heightened habitual tendencies in AN have consistently been reported using self-reports, while research utilizing experimental paradigms has yielded mixed results and efforts to capture real-world habits in AN remain limited. Some experimental paradigms appear more sensitive than others, but all face challenges associated with studying habits in the lab. Promising new approaches will need to be adopted and efforts made to capture real-world habits. Understanding which habits are problematic, when in illness and for whom they dominate, could make good on the promise of habit-focused treatments for AN.

Proactive and reactive construction of memory-based preferences

We are often faced with decisions we have never encountered before, requiring us to infer possible outcomes before making a choice. Computational theories suggest that one way to make these types of decisions is by accessing and linking related experiences stored in memory. Past work has shown that such memory-based preference construction can occur at a number of different timepoints relative to the moment a decision is made. Some studies have found that memories are integrated at the time a decision is faced (reactively) while others found that memory integration happens earlier, when memories were initially encoded (proactively). Here we offer a resolution to this inconsistency, demonstrating that these two strategies tradeoff rationally as a function of the associative structure of memory. We use fMRI to decode patterns of brain responses unique to categories of images in memory and find that proactive memory access is more common and allows more efficient inference. However, we also find that participants use reactive access when choice options are linked to a larger number of memory associations. Together, these results indicate that the brain judiciously conducts proactive inference by accessing memories ahead of time when conditions make this strategy more favorable.

Proactive and reactive construction of memory-based preferences

We are often faced with decisions we have never encountered before, requiring us to infer possible outcomes before making a choice. Computational theories suggest that one way to make these types of decisions is by accessing and linking related experiences stored in memory. Past work has shown that such memory-based preference construction can occur at a number of different timepoints relative to the moment a decision is made. Some studies have found that memories are integrated at the time a decision is faced (reactively) while others found that memory integration happens earlier, when memories were initially encoded (proactively). Here we offer a resolution to this inconsistency, demonstrating that these two strategies tradeoff rationally as a function of the associative structure of memory. We use fMRI to decode patterns of brain responses unique to categories of images in memory and find that proactive memory access is more common and allows more efficient inference. However, we also find that participants use reactive access when choice options are linked to a larger number of memory associations. Together, these results indicate that the brain judiciously conducts proactive inference by accessing memories ahead of time when conditions make this strategy more favorable.

Two routes to value-based decisions in Parkinson's disease: differentiating incremental reinforcement learning from episodic memory

Patients with Parkinson's disease are impaired at incremental reward-based learning. It is typically assumed that this impairment reflects a loss of striatal dopamine. However, many open questions remain about the nature of reward-based learning deficits in Parkinson's. Recent studies have found that a combination of different cognitive and computational strategies contribute even to simple reward-based learning tasks, suggesting a possible role for episodic memory. These findings raise critical questions about how incremental learning and episodic memory interact to support learning from past experience and what their relative contributions are to impaired decision-making in Parkinson's disease. Here we addressed these questions by asking patients with Parkinson's disease (n=26) both on and off their dopamine replacement medication and age- and education-matched healthy controls (n=26) to complete a task designed to isolate the contributions of incremental learning and episodic memory to reward-based learning and decision-making. We found that Parkinson's patients performed as well as healthy controls when using episodic memory, but were impaired at incremental reward-based learning. Dopamine replacement medication remediated this deficit while enhancing subsequent episodic memory for the value of motivationally relevant stimuli. These results demonstrate that Parkinson's patients are impaired at learning about reward from trial-and-error when episodic memory is properly controlled for, and that learning based on the value of single experiences remains intact in patients with Parkinson's disease.

A novel free-operant framework enables experimental habit induction in humans

Habits are a prominent feature of both adaptive and maladaptive behavior. Yet, despite substantial research efforts, there are currently no well-established experimental procedures for habit induction in humans. It is likely that laboratory experimental settings, as well as the session-based structure typically used in controlled experiments (also outside the lab), impose serious constraints on studying habits and other effects that are sensitive to context, motivation, and training duration and frequency. To overcome these challenges, we devised a unique real-world free-operant task structure, implemented through a novel smartphone application, whereby participants could freely enter the app (24 hours a day, 7 days a week) to win rewards. This procedure is free of typical laboratory constraints, yet well controlled. Using the canonical sensitivity to outcome devaluation criterion, we successfully demonstrated habit formation as a function of training duration, a long-standing challenge in the field. Additionally, we show a positive relationship between multiple facets of engagement/motivation and goal-directedness. We suggest that our novel paradigm can be used to study the neurobehavioral and psychological mechanism underlying habits in humans. Moreover, the real-world free-operant framework can potentially be used to examine other instrumental behavior-related questions, with greater face validity in naturalistic conditions.

Dopamine projections to the basolateral amygdala drive the encoding of identity-specific reward memories

To make adaptive decisions, we build an internal model of the associative relationships in an environment and use it to make predictions and inferences about specific available outcomes. Detailed, identity-specific cue-reward memories are a core feature of such cognitive maps. Here we used fiber photometry, cell-type and pathway-specific optogenetic manipulation, Pavlovian cue-reward conditioning and decision-making tests in male and female rats, to reveal that ventral tegmental area dopamine (VTADA) projections to the basolateral amygdala (BLA) drive the encoding of identity-specific cue-reward memories. Dopamine is released in the BLA during cue-reward pairing; VTADA→BLA activity is necessary and sufficient to link the identifying features of a reward to a predictive cue but does not assign general incentive properties to the cue or mediate reinforcement. These data reveal a dopaminergic pathway for the learning that supports adaptive decision-making and help explain how VTADA neurons achieve their emerging multifaceted role in learning.

Dopamine Increases Accuracy and Lengthens Deliberation Time in Explicit Motor Skill Learning

Although animal research implicates a central role for dopamine in motor skill learning, a direct causal link has yet to be established in neurotypical humans. Here, we tested if a pharmacological manipulation of dopamine alters motor learning, using a paradigm which engaged explicit, goal-directed strategies. Participants (27 females; 11 males; aged 18-29 years) first consumed either 100 mg of levodopa (n = 19), a dopamine precursor that increases dopamine availability, or placebo (n = 19). Then, during training, participants learnt the explicit strategy of aiming away from presented targets by instructed angles of varying sizes. Targets jumped mid-movement by the instructed aiming angle. Task success was thus contingent upon aiming accuracy and not speed. The effect of the dopamine manipulations on skill learning was assessed during training and after an overnight follow-up. Increasing dopamine availability at training improved aiming accuracy and lengthened reaction times, particularly for larger, more difficult aiming angles, both at training and, importantly, at follow-up, despite prominent session-by-session performance improvements in both accuracy and speed. Exogenous dopamine thus seems to result in a learnt, persistent propensity to better adhere to task goals. Results support the proposal that dopamine is important in engagement of instrumental motivation to optimize adherence to task goals, particularly when learning to execute goal-directed strategies in motor skill learning.

Erotic cue exposure increases physiological arousal, biases choices toward immediate rewards, and attenuates model-based reinforcement learning

Computational psychiatry focuses on identifying core cognitive processes that appear altered across distinct psychiatric disorders. Temporal discounting of future rewards and model-based control during reinforcement learning have proven as two promising candidates. Despite its trait-like stability, temporal discounting may be at least partly under contextual control. Highly arousing cues were shown to increase discounting, although evidence to date remains somewhat mixed. Whether model-based reinforcement learning is similarly affected by arousing cues remains unclear. Here, we tested cue-reactivity effects (erotic pictures) on subsequent temporal discounting and model-based reinforcement learning in a within-subjects design in n = 39 healthy heterosexual male participants. Self-reported and physiological arousal (cardiac activity and pupil dilation) were assessed before and during cue exposure. Arousal was increased during exposure of erotic versus neutral cues both on the subjective and autonomic level. Erotic cue exposure increased discounting as reflected by more impatient choices. Hierarchical drift diffusion modeling (DDM) linked increased discounting to a shift in the starting point bias of evidence accumulation toward immediate options. Model-based control during reinforcement learning was reduced following erotic cues according to model-agnostic analysis. Notably, DDM linked this effect to attenuated forgetting rates of unchosen options, leaving the model-based control parameter unchanged. Our findings replicate previous work on cue-reactivity effects in temporal discounting and for the first time show similar effects in model-based reinforcement learning in a heterosexual male sample. This highlights how environmental cues can impact core human decision processes and reveal that comprehensive modeling approaches can yield novel insights in reward-based decision processes.

Dopamine Alters the Effect of Brain Stimulation on Decision-Making

Noninvasive brain stimulation techniques, such as transcranial direct current stimulation (tDCS), show promise in treating a range of psychiatric and neurologic conditions. However, optimization of such applications requires a better understanding of how tDCS alters cognition and behavior. Existing evidence implicates dopamine in tDCS alterations of brain activity and plasticity; however, there is as yet no causal evidence for a role of dopamine in tDCS effects on cognition and behavior. Here, in a preregistered, double-blinded study, we examined how pharmacologically manipulating dopamine altered the effect of tDCS on the speed-accuracy trade-off, which taps ubiquitous strategic operations. Cathodal tDCS was delivered over the left prefrontal cortex and the superior medial frontal cortex before participants (N = 62, 24 males, 38 females) completed a dot-motion task, making judgments on the direction of a field of moving dots under instructions to emphasize speed, accuracy, or both. We leveraged computational modeling to uncover how our interventions altered latent decisional processes driving the speed-accuracy trade-off. We show that dopamine in combination with tDCS (but not tDCS alone nor dopamine alone) not only impaired decision accuracy but also impaired discriminability, which suggests that these manipulations altered the encoding or representation of discriminative evidence. This is, to the best of our knowledge, the first direct evidence implicating dopamine in the way tDCS affects cognition and behavior.SIGNIFICANCE STATEMENT tDCS can improve cognitive and behavioral impairments in clinical conditions; however, a better understanding of its mechanisms is required to optimize future clinical applications. Here, using a pharmacological approach to manipulate brain dopamine levels in healthy adults, we demonstrate a role for dopamine in the effects of tDCS in the speed-accuracy trade-off, a strategic cognitive process ubiquitous in many contexts. In doing so, we provide direct evidence implicating dopamine in the way tDCS affects cognition and behavior.

Impulsivity Traits in Parkinson's Disease: A Systematic Review and Meta-Analysis

Background

In Parkinson's disease (PD), impulsivity as a personality trait may be linked to the risk of developing impulse control disorders (ICDs) during dopaminergic therapy. However, studies evaluating differences in trait impulsivity between patients with PD and healthy controls or between patients with PD with and without ICDs reported partly inconsistent findings.

Objectives

We conducted a systematic review and meta-analysis (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) of studies comparing Barratt Impulsiveness Scale (BIS-11) scores between patients with PD and healthy controls and between patients with PD with and without ICDs.

Methods

Eligible studies were identified through a systematic search in 3 databases. Mean differences with 95% confidence intervals (CIs) for BIS-11 total and subscale scores were separately calculated for studies comparing patients with PD and healthy controls and patients with PD with and without ICDs. Meta-regressions were performed to explore sources of heterogeneity (percentage of men, age, disease duration, and levodopa equivalent daily dose).

Results

A total of 40 studies were included in the quantitative analyses. BIS-11 total scores were significantly higher in patients with PD compared with healthy controls (mean difference 2.43; 95% CI, 1.03, 3.83), and in patients with PD with active ICDs compared with patients without ICDs (6.62; 95% CI, 5.01, 8.23). No significant moderators emerged by meta-regression analyses.

Conclusions

The present meta-analysis supports that impulsivity, as a personality trait, may characterize patients with PD, even in the absence of ICDs. Moreover, these data corroborate findings of clinical studies reporting higher levels of trait impulsivity in PD patients with ICDs compared with patients without ICDs.

Rethinking model-based and model-free influences on mental effort and striatal prediction errors

A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.

Distinct reinforcement learning profiles distinguish between language and attentional neurodevelopmental disorders

BACKGROUND

Theoretical models posit abnormalities in cortico-striatal pathways in two of the most common neurodevelopmental disorders (Developmental dyslexia, DD, and Attention deficit hyperactive disorder, ADHD), but it is still unclear what distinct cortico-striatal dysfunction might distinguish language disorders from others that exhibit very different symptomatology. Although impairments in tasks that depend on the cortico-striatal network, including reinforcement learning (RL), have been implicated in both disorders, there has been little attempt to dissociate between different types of RL or to compare learning processes in these two types of disorders. The present study builds upon prior research indicating the existence of two learning manifestations of RL and evaluates whether these processes can be differentiated in language and attention deficit disorders. We used a two-step RL task shown to dissociate model-based from model-free learning in human learners.

RESULTS

Our results show that, relative to neurotypicals, DD individuals showed an impairment in model-free but not in model-based learning, whereas in ADHD the ability to use both model-free and model-based learning strategies was significantly compromised.

CONCLUSIONS

Thus, learning impairments in DD may be linked to a selective deficit in the ability to form action-outcome associations based on previous history, whereas in ADHD some learning deficits may be related to an incapacity to pursue rewards based on the tasks' structure. Our results indicate how different patterns of learning deficits may underlie different disorders, and how computation-minded experimental approaches can differentiate between them.

Negative symptoms, striatal dopamine and model-free reward decision-making in schizophrenia

Negative symptoms, such as lack of motivation or social withdrawal, are highly prevalent and debilitating in patients with schizophrenia. Underlying mechanisms of negative symptoms are incompletely understood, thereby preventing the development of targeted treatments. We hypothesized that in patients with schizophrenia during psychotic remission, impaired influences of both model-based and model-free reward predictions on decision-making ('reward prediction influence', RPI) underlie negative symptoms. We focused on psychotic remission, because psychotic symptoms might confound reward-based decision-making. Moreover, we hypothesized that impaired model-based/model-free RPIs depend on alterations of both associative striatum dopamine synthesis and storage (DSS) and executive functioning. Both factors influence RPI in healthy subjects and are typically impaired in schizophrenia. Twenty-five patients with schizophrenia with pronounced negative symptoms during psychotic remission and 24 healthy controls were included in the study. Negative symptom severity was measured by the Positive and Negative Syndrome Scale negative subscale, model-based/model-free RPI by the two-stage decision task, associative striatum DSS by 18F-DOPA positron emission tomography and executive functioning by the symbol coding task. Model-free RPI was selectively reduced in patients and associated with negative symptom severity as well as with reduced associative striatum DSS (in patients only) and executive functions (both in patients and controls). In contrast, model-based RPI was not altered in patients. Results provide evidence for impaired model-free reward prediction influence as a mechanism for negative symptoms in schizophrenia as well as for reduced associative striatum dopamine and executive dysfunction as relevant factors. Data suggest potential treatment targets for patients with schizophrenia and pronounced negative symptoms.

Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task

Humans form complex societies in which we routinely engage in social decision-making regarding the allocation of resources among ourselves and others. One dimension that characterizes social decision-making in particular is whether to prioritize self-interest or respect for others-proself or prosocial. What causes this individual difference in social value orientation? Recent developments in the social dual-process theory argue that social decision-making is characterized by its underlying domain-general learning systems: the model-free and model-based systems. In line with this "learning" approach, we propose and experimentally test the hypothesis that differences in social preferences stem from which learning system is dominant in an individual. Here, we used a non-social state transition task that allowed us to assess the balance between model-free/model-based learning and investigate its relation to the social value orientations. The results showed that proselfs depended more on model-based learning, whereas prosocials depended more on model-free learning. Reward amount and reaction time analyses showed that proselfs learned the task structure earlier in the session than prosocials, reflecting their difference in model-based/model-free learning dependence. These findings support the learning hypothesis on what makes differences in social preferences and have implications for understanding the mechanisms of prosocial behavior.

Effects of dopamine D2/3 and opioid receptor antagonism on the trade-off between model-based and model-free behaviour in healthy volunteers

Human behaviour requires flexible arbitration between actions we do out of habit and actions that are directed towards a specific goal. Drugs that target opioid and dopamine receptors are notorious for inducing maladaptive habitual drug consumption; yet, how the opioidergic and dopaminergic neurotransmitter systems contribute to the arbitration between habitual and goal-directed behaviour is poorly understood. By combining pharmacological challenges with a well-established decision-making task and a novel computational model, we show that the administration of the dopamine D2/3 receptor antagonist amisulpride led to an increase in goal-directed or 'model-based' relative to habitual or 'model-free' behaviour, whereas the non-selective opioid receptor antagonist naltrexone had no appreciable effect. The effect of amisulpride on model-based/model-free behaviour did not scale with drug serum levels in the blood. Furthermore, participants with higher amisulpride serum levels showed higher explorative behaviour. These findings highlight the distinct functional contributions of dopamine and opioid receptors to goal-directed and habitual behaviour and support the notion that even small doses of amisulpride promote flexible application of cognitive control.

The catecholamine precursor Tyrosine reduces autonomic arousal and decreases decision thresholds in reinforcement learning and temporal discounting

Supplementation with the catecholamine precursor L-Tyrosine might enhance cognitive performance, but overall findings are mixed. Here, we investigate the effect of a single dose of tyrosine (2g) vs. placebo on two catecholamine-dependent trans-diagnostic traits: model-based control during reinforcement learning (2-step task) and temporal discounting, using a double-blind, placebo-controlled, within-subject design (n = 28 healthy male participants). We leveraged drift diffusion models in a hierarchical Bayesian framework to jointly model participants' choices and response times (RTS) in both tasks. Furthermore, comprehensive autonomic monitoring (heart rate, heart rate variability, pupillometry, spontaneous eye blink rate) was performed both pre- and post-supplementation, to explore potential physiological effects of supplementation. Across tasks, tyrosine consistently reduced participants' RTs without deteriorating task-performance. Diffusion modeling linked this effect to attenuated decision-thresholds in both tasks and further revealed increased model-based control (2-step task) and (if anything) attenuated temporal discounting. On the physiological level, participants' pupil dilation was predictive of the individual degree of temporal discounting. Tyrosine supplementation reduced physiological arousal as revealed by increases in pupil dilation variability and reductions in heart rate. Supplementation-related changes in physiological arousal predicted individual changes in temporal discounting. Our findings provide first evidence that tyrosine supplementation might impact psychophysiological parameters, and suggest that modeling approaches based on sequential sampling models can yield novel insights into latent cognitive processes modulated by amino-acid supplementation.

Time estimation and arousal responses in dopa-responsive dystonia

Dopa-responsive dystonia (DRD) is caused by an impaired dopamine biosynthesis due to a GTP-cyclohydrolase-1 (GCH1) deficiency, resulting in a combination of dystonia and parkinsonism. However, the effect of GCH1 mutations and levodopa treatment on motor control beyond simple movements, such as timing, action preparation and feedback processing, have not been investigated so far. In an active time estimation task with trial-by-trial feedback, participants indicated a target interval (1200 ms) by a motor response. We compared 12 patients tested (in fixed order) under their current levodopa medication ("ON") and after levodopa withdrawal ("OFF") to matched healthy controls (HC), measured twice to control for repetition effects. We assessed time estimation accuracy, trial-to-trial adjustment, as well as task- and feedback-related pupil-linked arousal responses. Patients showed comparable time estimation accuracy ON medication as HC but reduced performance OFF medication. Task-related pupil responses showed the reverse pattern. Trial-to-trial adjustments of response times were reduced in DRD, particularly OFF medication. Our results indicate differential alterations of time estimation accuracy and task-related arousal dynamics in DRD patients as a function of dopaminergic medication state. A medication-independent alteration of task repetition effects in DRD cannot be ruled out with certainty but is discussed as less likely.

What Can Reinforcement Learning Models of Dopamine and Serotonin Tell Us about the Action of Antidepressants?

Although evidence suggests that antidepressants are effective at treating depression, the mechanisms behind antidepressant action remain unclear, especially at the cognitive/computational level. In recent years, reinforcement learning (RL) models have increasingly been used to characterise the roles of neurotransmitters and to probe the computations that might be altered in psychiatric disorders like depression. Hence, RL models might present an opportunity for us to better understand the computational mechanisms underlying antidepressant effects. Moreover, RL models may also help us shed light on how these computations may be implemented in the brain (e.g., in midbrain, striatal, and prefrontal regions) and how these neural mechanisms may be altered in depression and remediated by antidepressant treatments. In this paper, we evaluate the ability of RL models to help us understand the processes underlying antidepressant action. To do this, we review the preclinical literature on the roles of dopamine and serotonin in RL, draw links between these findings and clinical work investigating computations altered in depression, and appraise the evidence linking modification of RL processes to antidepressant function. Overall, while there is no shortage of promising ideas about the computational mechanisms underlying antidepressant effects, there is insufficient evidence directly implicating these mechanisms in the response of depressed patients to antidepressant treatment. Consequently, future studies should investigate these mechanisms in samples of depressed patients and assess whether modifications in RL processes mediate the clinical effect of antidepressant treatments.

Decreased sensitivity to loss of options in patients with Parkinson's disease

Humans prefer to have many options when making decisions. When there is a threat of options disappearing, humans invest more to keep these options available, indicating that they are sensitive to the loss of options. This study examined whether patients with Parkinson's disease (PD), a disease characterized by dopamine depletion, try to keep options available when options are disappearing. Twenty-seven PD patients without dementia and 27 healthy controls (HCs) performed the door game, in which participants were presented with multiple alternatives in the form of three doors, each associated with a different point distribution. The participants were asked to maximize their point earnings by finding the best door. The task included two conditions. In the shutter condition, shutters gradually closed on doors that were not chosen; once the shutters completely closed, the door was no longer available. There were no shutters in the control condition. The results revealed that the HCs switched doors more often in the shutter condition than in the control condition, indicating a tendency to keep options available. However, the PD patients did not show such differences between the two conditions. The difference in the number of switches between the shutter and control conditions in the PD patients was significantly positively correlated with the distribution of dopamine transporters in the left striatum, as measured by ¹²³I-ioflupane-SPECT (DaTSCAN) images. These results suggest that PD patients are less sensitive to the loss of options, and this decreased sensitivity may be caused by a decline in dopaminergic neurotransmission.

Dopaminergic medication increases motivation to exert cognitive control by reducing subjective effort costs in Parkinson's patients

Engaging in demanding mental activities requires the allocation of cognitive control, which can be effortful and aversive. Individuals thus tend to avoid exerting cognitive effort if less demanding behavioral options are available. Recent accounts propose a key role for dopamine in motivating behavior by increasing the sensitivity to rewards associated with effort exertion. Whether dopamine additionally plays a specific role in modulating the sensitivity to the costs of cognitive effort, even in the absence of any incentives, is much less clear. To address this question, we assessed cognitive effort avoidance in patients (n = 38) with Parkinson's disease, a condition characterized by loss of midbrain dopaminergic neurons, both ON and OFF dopaminergic medication and compared them to healthy controls (n = 24). Effort avoidance was assessed using the Demand Selection Task (DST), in which participants could freely choose between performing a high-demand or a low-demand version of a task-switching paradigm. Critically, participants were not offered any incentives to choose the more effortful option, nor for good performance. While healthy controls and patients OFF their dopaminergic medications consistently preferred the low-demand option, effort avoidance in patients ON dopaminergic medications was reduced compared to patients OFF, a difference which seems to lessen over trials. These differences in preference could not be explained by altered task-switching performance. Although patients ON were less accurate at detecting the different effort levels, as measured during instructed forced-choice blocks, their detection ability was not associated with effort avoidance, unlike in the healthy controls and the patients OFF. Our findings provide evidence that dopamine replacement in Parkinson's patients increases the willingness to engage in cognitively demanding behavior, and that this cannot be explained by possible effects of dopamine replacement on performance nor on the ability to detect effort demands. These results suggest that dopamine plays a role in reducing the sensitivity to effort costs that is independent of its role in enhancing the sensitivity to the benefits of effort exertion.

A Neuroeconomics Approach to Obesity

Obesity is a heterogeneous condition that is affected by physiological, behavioral, and environmental factors. Value-based decision making is a useful framework for integrating these factors at the individual level. The disciplines of behavioral economics and reinforcement learning provide tools for identifying specific cognitive and motivational processes that may contribute to the development and maintenance of obesity. Neuroeconomics complements these disciplines by studying the neural mechanisms underlying these processes. We surveyed recent literature on individual decision characteristics that are most frequently implicated in obesity: discounting the value of future outcomes, attitudes toward uncertainty, and learning from rewards and punishments. Our survey highlighted both consistent and inconsistent behavioral findings. These findings underscore the need to examine multiple processes within individuals to identify unique behavioral profiles associated with obesity. Such individual characterization will inform future studies on the neurobiology of obesity as well as the design of effective interventions that are individually tailored.

Disrupted reward processing in Parkinson's disease and its relationship with dopamine state and neuropsychiatric syndromes: a systematic review and meta-analysis

BACKGROUND

Neuropsychiatric symptoms are common in Parkinson's disease (PD) and predict poorer outcomes. Reward processing dysfunction is a candidate mechanism for the development of psychiatric symptoms including depression and impulse control disorders (ICDs). We aimed to determine whether reward processing is impaired in PD and its relationship with neuropsychiatric syndromes and dopamine replacement therapy.

METHODS

The Ovid MEDLINE/PubMed, Embase and PsycInfo databases were searched for articles published up to 5 November 2020. Studies reporting reward processing task performance by patients with PD and healthy controls were included. Summary statistics comparing reward processing between groups were converted to standardised mean difference (SMD) scores and meta-analysed using a random effects model.

RESULTS

We identified 55 studies containing 2578 participants (1638 PD and 940 healthy controls). Studies assessing three subcomponent categories of reward processing tasks were included: option valuation (n=12), reinforcement learning (n=37) and reward response vigour (n=6). Across all studies, patients with PD on medication exhibited a small-to-medium impairment versus healthy controls (SMD=0.34; 95% CI 0.14 to 0.53), with greater impairments observed off dopaminergic medication in within-subjects designs (SMD=0.43, 95% CI 0.29 to 0.57). Within-subjects subcomponent analysis revealed impaired processing off medication on option valuation (SMD=0.57, 95% CI 0.39 to 0.75) and reward response vigour (SMD=0.36, 95% CI 0.13 to 0.59) tasks. However, the opposite applied for reinforcement learning, which relative to healthy controls was impaired on-medication (SMD=0.45, 95% CI 0.25 to 0.65) but not off-medication (SMD=0.28, 95% CI -0.03 to 0.59). ICD was the only neuropsychiatric syndrome with sufficient studies (n=13) for meta-analysis, but no significant impairment was identified compared tonon-ICD patients (SMD=-0.02, 95% CI -0.43 to 0.39).

CONCLUSION

Reward processing disruption in PD differs according to subcomponent and dopamine medication state, and warrants further study as a potential treatment target and mechanism underlying associated neuropsychiatric syndromes.

Role of dopamine and clinical heterogeneity in cognitive dysfunction in Parkinson's disease

Parkinson's disease (PD) is commonly treated with dopaminergic medication, which enhances some, while impairing other cognitive functions. It can even contribute to impulse control disorder and addiction. We describe the history of research supporting the dopamine overdose hypothesis, which accounts for the large within-patient variability in dopaminergic medication effects across different tasks by referring to the spatially non-uniform pattern of dopamine depletion in dorsal versus ventral striatum. However, there is tremendous variability in dopaminergic medication effects not just within patients across distinct tasks, but also across different patients. In the second part of this chapter we review recent studies addressing the large individual variability in the negative side effects of dopaminergic medication on functions that implicate dopamine, such as value-based learning and choice. These studies begin to unravel the mechanisms of dopamine overdosing, thus revising the strict version of the overdose hypothesis. For example, the work shows that the canonical boosting of reward-versus punishment-based choice by medication is greater in patients with depression and a non-tremor phenotype, which both implicate, among other pathology, more rather than less severe dysregulation of the mesolimbic dopamine system. Future longitudinal cohort studies are needed to identify how to optimally combine different clinical, personality, cognitive, neural, genetic and molecular predictors of detrimental medication effects in order to account for as much of the relevant variability as possible. This will provide a useful tool for precision neurology, allowing individual and contextual tailoring of (the dose of) dopaminergic medication in order to maximize its cognitive benefits, yet minimize its side effects.

The prediction-error hypothesis of schizophrenia: new data point to circuit-specific changes in dopamine activity

Schizophrenia is a severe psychiatric disorder affecting 21 million people worldwide. People with schizophrenia suffer from symptoms including psychosis and delusions, apathy, anhedonia, and cognitive deficits. Strikingly, schizophrenia is characterised by a learning paradox involving difficulties learning from rewarding events, whilst simultaneously 'overlearning' about irrelevant or neutral information. While dysfunction in dopaminergic signalling has long been linked to the pathophysiology of schizophrenia, a cohesive framework that accounts for this learning paradox remains elusive. Recently, there has been an explosion of new research investigating how dopamine contributes to reinforcement learning, which illustrates that midbrain dopamine contributes in complex ways to reinforcement learning, not previously envisioned. This new data brings new possibilities for how dopamine signalling contributes to the symptomatology of schizophrenia. Building on recent work, we present a new neural framework for how we might envision specific dopamine circuits contributing to this learning paradox in schizophrenia in the context of models of reinforcement learning. Further, we discuss avenues of preclinical research with the use of cutting-edge neuroscience techniques where aspects of this model may be tested. Ultimately, it is hoped that this review will spur to action more research utilising specific reinforcement learning paradigms in preclinical models of schizophrenia, to reconcile seemingly disparate symptomatology and develop more efficient therapeutics.

Adaptive Empathy: A Model for Learning Empathic Responses in Response to Feedback

Empathy is usually deployed in social interactions. Nevertheless, common measures and examinations of empathy study this construct in isolation from the person in distress. In this article we seek to extend the field of examination to include both empathizer and target to determine whether and how empathic responses are affected by feedback and learned through interaction. Building on computational approaches in feedback-based adaptations (e.g., no feedback, model-free and model-based learning), we propose a framework for understanding how empathic responses are learned on the basis of feedback. In this framework, adaptive empathy, defined as the ability to adapt one's empathic responses, is a central aspect of empathic skills and can provide a new dimension to the evaluation and investigation of empathy. By extending existing neural models of empathy, we suggest that adaptive empathy may be mediated by interactions between the neural circuits associated with valuation, shared distress, observation-execution, and mentalizing. Finally, we propose that adaptive empathy should be considered a prominent facet of empathic capabilities with the potential to explain empathic behavior in health and psychopathology.

Influences of dopaminergic system dysfunction on late-life depression

Deficits in cognition, reward processing, and motor function are clinical features relevant to both aging and depression. Individuals with late-life depression often show impairment across these domains, all of which are moderated by the functioning of dopaminergic circuits. As dopaminergic function declines with normal aging and increased inflammatory burden, the role of dopamine may be particularly salient for late-life depression. We review the literature examining the role of dopamine in the pathogenesis of depression, as well as how dopamine function changes with aging and is influenced by inflammation. Applying a Research Domain Criteria (RDoC) Initiative perspective, we then review work examining how dopaminergic signaling affects these domains, specifically focusing on Cognitive, Positive Valence, and Sensorimotor Systems. We propose a unified model incorporating the effects of aging and low-grade inflammation on dopaminergic functioning, with a resulting negative effect on cognition, reward processing, and motor function. Interplay between these systems may influence development of a depressive phenotype, with an initial deficit in one domain reinforcing decline in others. This model extends RDoC concepts into late-life depression while also providing opportunities for novel and personalized interventions.

Dopamine enhances model-free credit assignment through boosting of retrospective model-based inference

Dopamine is implicated in representing model-free (MF) reward prediction errors a as well as influencing model-based (MB) credit assignment and choice. Putative cooperative interactions between MB and MF systems include a guidance of MF credit assignment by MB inference. Here, we used a double-blind, placebo-controlled, within-subjects design to test an hypothesis that enhancing dopamine levels boosts the guidance of MF credit assignment by MB inference. In line with this, we found that levodopa enhanced guidance of MF credit assignment by MB inference, without impacting MF and MB influences directly. This drug effect correlated negatively with a dopamine-dependent change in purely MB credit assignment, possibly reflecting a trade-off between these two MB components of behavioural control. Our findings of a dopamine boost in MB inference guidance of MF learning highlight a novel DA influence on MB-MF cooperative interactions.

Dissociable effects of age and Parkinson's disease on instruction-based learning

The cognitive deficits associated with Parkinson's disease vary across individuals and change across time, with implications for prognosis and treatment. Key outstanding challenges are to define the distinct behavioural characteristics of this disorder and develop diagnostic paradigms that can assess these sensitively in individuals. In a previous study, we measured different aspects of attentional control in Parkinson's disease using an established fMRI switching paradigm. We observed no deficits for the aspects of attention the task was designed to examine; instead those with Parkinson's disease learnt the operational requirements of the task more slowly. We hypothesized that a subset of people with early-to-mid stage Parkinson's might be impaired when encoding rules for performing new tasks. Here, we directly test this hypothesis and investigate whether deficits in instruction-based learning represent a characteristic of Parkinson's Disease. Seventeen participants with Parkinson's disease (8 male; mean age: 61.2 years), 18 older adults (8 male; mean age: 61.3 years) and 20 younger adults (10 males; mean age: 26.7 years) undertook a simple instruction-based learning paradigm in the MRI scanner. They sorted sequences of coloured shapes according to binary discrimination rules that were updated at two-minute intervals. Unlike common reinforcement learning tasks, the rules were unambiguous, being explicitly presented; consequently, there was no requirement to monitor feedback or estimate contingencies. Despite its simplicity, a third of the Parkinson's group, but only one older adult, showed marked increases in errors, 4 SD greater than the worst performing young adult. The pattern of errors was consistent, reflecting a tendency to misbind discrimination rules. The misbinding behaviour was coupled with reduced frontal, parietal and anterior caudate activity when rules were being encoded, but not when attention was initially oriented to the instruction slides or when discrimination trials were performed. Concomitantly, Magnetic Resonance Spectroscopy showed reduced gamma-Aminobutyric acid levels within the mid-dorsolateral prefrontal cortices of individuals who made misbinding errors. These results demonstrate, for the first time, that a subset of early-to-mid stage people with Parkinson's show substantial deficits when binding new task rules in working memory. Given the ubiquity of instruction-based learning, these deficits are likely to impede daily living. They will also confound clinical assessment of other cognitive processes. Future work should determine the value of instruction-based learning as a sensitive early marker of cognitive decline and as a measure of responsiveness to therapy in Parkinson's disease.

α-Synuclein Selectively Impairs Motor Sequence Learning and Value Sensitivity: Reversal by the Adenosine A2A Receptor Antagonists

Parkinson's disease (PD) is characterized pathologically by alpha-synuclein (α-Syn) aggregates and clinically by the motor as well as cognitive deficits, including impairments in sequence learning and habit learning. Using intracerebral injection of WT and A53T mutant α-Syn fibrils, we investigate the behavioral mechanism of α-Syn for procedure-learning deficit in PD by critically determining the α-Syn-induced effects on model-based goal-directed behavior, model-free (probability-based) habit learning, and hierarchically organized sequence learning. 1) Contrary to the widely held view of habit-learning deficit in early PD, α-Syn aggregates in the dorsomedial striatum (DMS) and dorsolateral striatum (DLS) did not affect acquisition of habit learning, but selectively impaired goal-directed behavior with reduced value sensitivity. 2) α-Syn in the DLS (but not DMS) and SNc selectively impaired the sequence learning by affecting sequence initiation with the reduced first-step accuracy. 3) Adenosine A2A receptor (A2AR) antagonist KW6002 selectively improved sequence learning by preferentially improving sequence initiation and shift of sequence learning as well as behavioral reactivity. These findings established a casual role of α-Syn in the SN-DLS pathway in sequence-learning deficit and DMS α-Syn in goal-directed behavior deficit and suggest a novel therapeutic strategy to improve sequence-learning deficit in PD with enhanced sequence initiation by A2AR antagonists.

The Straw That Broke the Camel's Back: Natural Variations in 17β-Estradiol and COMT-Val158Met Genotype Interact in the Modulation of Model-Free and Model-Based Control

The sex hormone estradiol has recently gained attention in human decision-making research. Animal studies have already shown that estradiol promotes dopaminergic transmission and thus supports reward-seeking behavior and aspects of addiction. In humans, natural variations of estradiol across the menstrual cycle modulate the ability to learn from direct performance feedback ("model-free" learning). However, it remains unclear whether estradiol also influences more complex "model-based" contributions to reinforcement learning. Here, 41 women were tested twice - in the low and high estradiol state of the follicular phase of their menstrual cycle - with a Two-Step decision task designed to separate model-free from model-based learning. The results showed that in the high estradiol state women relied more heavily on model-free learning, and accomplished reduced performance gains, particularly during the more volatile periods of the task that demanded increased learning effort. In contrast, model-based control remained unaltered by the influence of hormonal state across the group. Yet, when accounting for individual differences in the genetic proxy of the COMT-Val158Met polymorphism (rs4680), we observed that only the participants homozygote for the methionine allele (n = 12; with putatively higher prefrontal dopamine) experienced a decline in model-based control when facing volatile reward probabilities. This group also showed the increase in suboptimal model-free control, while the carriers of the valine allele remained unaffected by the rise in endogenous estradiol. Taken together, these preliminary findings suggest that endogenous estradiol may affect the balance between model-based and model-free control, and particularly so in women with a high prefrontal baseline dopamine capacity and in situations of increased environmental volatility.

Null effects of levodopa on reward- and error-based motor adaptation, savings, and anterograde interference

Dopamine signaling is thought to mediate reward-based learning. We tested for a role of dopamine in motor adaptation by administering the dopamine precursor levodopa to healthy participants in two experiments involving reaching movements. Levodopa has been shown to impair reward-based learning in cognitive tasks. Thus, we hypothesized that levodopa would selectively impair aspects of motor adaptation that depend on the reinforcement of rewarding actions. In the first experiment, participants performed two separate tasks in which adaptation was driven either by visual error-based feedback of the hand position or binary reward feedback. We used EEG to measure event-related potentials evoked by task feedback. We hypothesized that levodopa would specifically diminish adaptation and the neural responses to feedback in the reward learning task. However, levodopa did not affect motor adaptation in either task nor did it diminish event-related potentials elicited by reward outcomes. In the second experiment, participants learned to compensate for mechanical force field perturbations applied to the hand during reaching. Previous exposure to a particular force field can result in savings during subsequent adaptation to the same force field or interference during adaptation to an opposite force field. We hypothesized that levodopa would diminish savings and anterograde interference, as previous work suggests that these phenomena result from a reinforcement learning process. However, we found no reliable effects of levodopa. These results suggest that reward-based motor adaptation, savings, and interference may not depend on the same dopaminergic mechanisms that have been shown to be disrupted by levodopa during various cognitive tasks.NEW & NOTEWORTHY Motor adaptation relies on multiple processes including reinforcement of successful actions. Cognitive reinforcement learning is impaired by levodopa-induced disruption of dopamine function. We administered levodopa to healthy adults who participated in multiple motor adaptation tasks. We found no effects of levodopa on any component of motor adaptation. This suggests that motor adaptation may not depend on the same dopaminergic mechanisms as cognitive forms or reinforcement learning that have been shown to be impaired by levodopa.

What is dopamine doing in model-based reinforcement learning?

Experiments have implicated dopamine in model-based reinforcement learning (RL). These findings are unexpected as dopamine is thought to encode a reward prediction error (RPE), which is the key teaching signal in model-free RL. Here we examine two possible accounts for dopamine's involvement in model-based RL: the first that dopamine neurons carry a prediction error used to update a type of predictive state representation called a successor representation, the second that two well established aspects of dopaminergic activity, RPEs and surprise signals, can together explain dopamine's involvement in model-based RL.

Reliance on model-based and model-free control in obesity

Consuming more energy than is expended may reflect a failure of control over eating behaviour in obesity. Behavioural control arises from a balance between two dissociable strategies of reinforcement learning: model-free and model-based. We hypothesized that weight status relates to an imbalance in reliance on model-based and model-free control, and that it may do so in a linear or quadratic manner. To test this, 90 healthy participants in a wide BMI range [normal-weight (n = 31), overweight (n = 29), obese (n = 30)] performed a sequential decision-making task. The primary analysis indicated that obese participants relied less on model-based control than overweight and normal-weight participants, with no difference between overweight and normal-weight participants. In line, secondary continuous analyses revealed a negative linear, but not quadratic, relationship between BMI and model-based control. Computational modelling of choice behaviour suggested that a mixture of both strategies was shifted towards less model-based control in obese participants. Our findings suggest that obesity may indeed be related to an imbalance in behavioural control as expressed in a phenotype of less model-based control potentially resulting from enhanced reliance on model-free computations.

Deficient Goal-Directed Control in a Population Characterized by Extreme Goal Pursuit

Research in computational psychiatry has sought to understand the basis of compulsive behavior by relating it to basic psychological and neural mechanisms: specifically, goal-directed versus habitual control. These psychological categories have been further identified with formal computational algorithms, model-based and model-free learning, which helps to provide quantitative tools to distinguish them. Computational psychiatry may be particularly useful for examining phenomena in individuals with anorexia nervosa (AN), whose self-starvation appears both excessively goal directed and habitual. However, these laboratory-based studies have not aimed to examine complex behavior, as seen outside the laboratory, in contexts that extend beyond monetary rewards. We therefore assessed (1) whether behavior in AN was characterized by enhanced or diminished model-based behavior, (2) the domain specificity of any abnormalities by comparing learning in a food-specific (i.e., illness-relevant) context as well as in a monetary context, and (3) whether impairments were secondary to starvation by comparing learning before and after initial treatment. Across all conditions, individuals with AN, relative to healthy controls, showed an impairment in model-based, but not model-free, learning, suggesting a general and persistent contribution of habitual over goal-directed control, across domains and time points. Thus, eating behavior in individuals with AN that appears very goal-directed may be under more habitual than goal-directed control, and this is not remediated by achieving weight restoration.

Is impaired dopaminergic function associated with mobility capacity in older adults?

The capacity to move is essential for independence and declines with age. Slow movement speed, in particular, is strongly associated with negative health outcomes. Prior research on mobility (herein defined as movement slowness) and aging has largely focused on musculoskeletal mechanisms and processes. More recent work has provided growing evidence for a significant role of the nervous system in contributing to reduced mobility in older adults. In this article, we report four pieces of complementary evidence from behavioral, genetic, and neuroimaging experiments that, we believe, provide theoretical support for the assertion that the basal ganglia and its dopaminergic function are responsible, in part, for age-related reductions in mobility. We report four a posteriori findings from an existing dataset: (1) slower central activation of ballistic force development is associated with worse mobility among older adults; (2) older adults with the Val/Met intermediate catecholamine-O-methyl-transferase (COMT) genotype involved in dopamine degradation exhibit greater mobility than their homozygous counterparts; (3) there are moderate relationships between performance times from a series of lower and upper extremity tasks supporting the notion that movement speed in older adults is a trait-like attribute; and (4) there is a relationship of functional connectivity within the medial orbofrontal (mOFC) cortico-striatal network and measures of mobility, suggesting that a potential neural mechanism for impaired mobility with aging is the deterioration of the integrity of key regions within the mOFC cortico-striatal network. These findings align with recent basic and clinical science work suggesting that the basal ganglia and its dopaminergic function are mechanistically linked to age-related reductions in mobility capacity.

Dopamine promotes instrumental motivation, but reduces reward-related vigour

We can be motivated when reward depends on performance, or merely by the prospect of a guaranteed reward. Performance-dependent (contingent) reward is instrumental, relying on an internal action-outcome model, whereas motivation by guaranteed reward may minimise opportunity cost in reward-rich environments. Competing theories propose that each type of motivation should be dependent on dopaminergic activity. We contrasted these two types of motivation with a rewarded saccade task, in patients with Parkinson’s disease (PD). When PD patients were ON dopamine, they had greater response vigour (peak saccadic velocity residuals) for contingent rewards, whereas when PD patients were OFF medication, they had greater vigour for guaranteed rewards. These results support the view that reward expectation and contingency drive distinct motivational processes, and can be dissociated by manipulating dopaminergic activity. We posit that dopamine promotes goal-directed motivation, but dampens reward-driven vigour, contradictory to the prediction that increased tonic dopamine amplifies reward expectation.

Parallel model-based and model-free reinforcement learning for card sorting performance

The Wisconsin Card Sorting Test (WCST) is considered a gold standard for the assessment of cognitive flexibility. On the WCST, repeating a sorting category following negative feedback is typically treated as indicating reduced cognitive flexibility. Therefore such responses are referred to as 'perseveration' errors. Recent research suggests that the propensity for perseveration errors is modulated by response demands: They occur less frequently when their commitment repeats the previously executed response. Here, we propose parallel reinforcement-learning models of card sorting performance, which assume that card sorting performance can be conceptualized as resulting from model-free reinforcement learning at the level of responses that occurs in parallel with model-based reinforcement learning at the categorical level. We compared parallel reinforcement-learning models with purely model-based reinforcement learning, and with the state-of-the-art attentional-updating model. We analyzed data from 375 participants who completed a computerized WCST. Parallel reinforcement-learning models showed best predictive accuracies for the majority of participants. Only parallel reinforcement-learning models accounted for the modulation of perseveration propensity by response demands. In conclusion, parallel reinforcement-learning models provide a new theoretical perspective on card sorting and it offers a suitable framework for discerning individual differences in latent processes that subserve behavioral flexibility.

Dopaminergic Modulation of Human Intertemporal Choice: A Diffusion Model Analysis Using the D2-Receptor Antagonist Haloperidol

The neurotransmitter dopamine is implicated in diverse functions, including reward processing, reinforcement learning, and cognitive control. The tendency to discount future rewards over time has long been discussed in the context of potential dopaminergic modulation. Here we examined the effect of a single dose of the D2 receptor antagonist haloperidol (2 mg) on temporal discounting in healthy female and male human participants. Our approach extends previous pharmacological studies in two ways. First, we applied combined temporal discounting drift diffusion models to examine choice dynamics. Second, we examined dopaminergic modulation of reward magnitude effects on temporal discounting. Hierarchical Bayesian parameter estimation revealed that the data were best accounted for by a temporal discounting drift diffusion model with nonlinear trialwise drift rate scaling. This model showed good parameter recovery, and posterior predictive checks revealed that it accurately reproduced the relationship between decision conflict and response times in individual participants. We observed reduced temporal discounting and substantially faster nondecision times under haloperidol compared with placebo. Discounting was steeper for low versus high reward magnitudes, but this effect was largely unaffected by haloperidol. Results were corroborated by model-free analyses and modeling via more standard approaches. We previously reported elevated caudate activation under haloperidol in this sample of participants, supporting the idea that haloperidol elevated dopamine neurotransmission (e.g., by blocking inhibitory feedback via presynaptic D2 auto-receptors). The present results reveal that this is associated with an augmentation of both lower-level (nondecision time) and higher-level (temporal discounting) components of the decision process.SIGNIFICANCE STATEMENT Dopamine is implicated in reward processing, reinforcement learning, and cognitive control. Here we examined the effects of a single dose of the D2 receptor antagonist haloperidol on temporal discounting and choice dynamics during the decision process. We extend previous studies by applying computational modeling using the drift diffusion model, which revealed that haloperidol reduced the nondecision time and reduced impulsive choice compared with placebo. These findings are compatible with a haloperidol-induced increase in striatal dopamine (e.g., because of a presynaptic mechanism). Our data provide novel insights into the contributions of dopamine to value-based decision-making and highlight how comprehensive model-based analyses using sequential sampling models can inform the effects of pharmacological modulation on choice processes.

A Computational Study of Executive Dysfunction in Amyotrophic Lateral Sclerosis

Executive dysfunction is a well-documented, yet nonspecific corollary of various neurological diseases and psychiatric disorders. Here, we applied computational modeling of latent cognition for executive control in amyotrophic lateral sclerosis (ALS) patients. We utilized a parallel reinforcement learning model of trial-by-trial Wisconsin Card Sorting Test (WCST) behavior. Eighteen ALS patients and 21 matched healthy control participants were assessed on a computerized variant of the WCST (cWCST). ALS patients showed latent cognitive symptoms, which can be characterized as bradyphrenia and haphazard responding. A comparison with results from a recent computational Parkinson's disease (PD) study (Steinke et al., 2020, J Clin Med) suggests that bradyphrenia represents a disease-nonspecific latent cognitive symptom of ALS and PD patients alike. Haphazard responding seems to be a disease-specific latent cognitive symptom of ALS, whereas impaired stimulus-response learning seems to be a disease-specific latent cognitive symptom of PD. These data were obtained from the careful modeling of trial-by-trial behavior on the cWCST, and they suggest that computational cognitive neuropsychology provides nosologically specific indicators of latent facets of executive dysfunction in ALS (and PD) patients, which remain undiscoverable for traditional behavioral cognitive neuropsychology. We discuss implications for neuropsychological assessment, and we discuss opportunities for confirmatory computational brain imaging studies.

Dopamine is associated with prioritization of reward-associated memories in Parkinson's disease

Patients with Parkinson's disease have reduced reward sensitivity related to dopaminergic neuron loss, which is associated with impairments in reinforcement learning. Increasingly, however, dopamine-dependent reward signals are recognized to play an important role beyond reinforcement learning. In particular, it has been shown that reward signals mediated by dopamine help guide the prioritization of events for long-term memory consolidation. Meanwhile, studies of memory in patients with Parkinson's disease have focused on overall memory capacity rather than what is versus what isn't remembered, leaving open questions about the effect of dopamine replacement on the prioritization of memories by reward and the time-dependence of this effect. The current study sought to fill this gap by testing the effect of reward and dopamine on memory in patients with Parkinson's disease. We tested the effect of dopamine modulation and reward on two forms of long-term memory: episodic memory for neutral objects and memory for stimulus-value associations. We measured both forms of memory in a single task, adapting a standard task of reinforcement learning with incidental episodic encoding events of trial-unique objects. Objects were presented on each trial at the time of feedback, which was either rewarding or not. Memory for the trial-unique images and for the stimulus-value associations, and the influence of reward on both, was tested immediately after learning and 2 days later. We measured performance in Parkinson's disease patients tested either ON or OFF their dopaminergic medications and in healthy older control subjects. We found that dopamine was associated with a selective enhancement of memory for reward-associated images, but that it did not influence overall memory capacity. Contrary to predictions, this effect did not differ between the immediate and delayed memory tests. We also found that while dopamine had an effect on reward-modulated episodic memory, there was no effect of dopamine on memory for stimulus-value associations. Our results suggest that impaired prioritization of cognitive resource allocation may contribute to the early cognitive deficits of Parkinson's disease.

Combined model-free and model-sensitive reinforcement learning in non-human primates

Contemporary reinforcement learning (RL) theory suggests that potential choices can be evaluated by strategies that may or may not be sensitive to the computational structure of tasks. A paradigmatic model-free (MF) strategy simply repeats actions that have been rewarded in the past; by contrast, model-sensitive (MS) strategies exploit richer information associated with knowledge of task dynamics. MF and MS strategies should typically be combined, because they have complementary statistical and computational strengths; however, this tradeoff between MF/MS RL has mostly only been demonstrated in humans, often with only modest numbers of trials. We trained rhesus monkeys to perform a two-stage decision task designed to elicit and discriminate the use of MF and MS methods. A descriptive analysis of choice behaviour revealed directly that the structure of the task (of MS importance) and the reward history (of MF and MS importance) significantly influenced both choice and response vigour. A detailed, trial-by-trial computational analysis confirmed that choices were made according to a combination of strategies, with a dominant influence of a particular form of model sensitivity that persisted over weeks of testing. The residuals from this model necessitated development of a new combined RL model which incorporates a particular credit assignment weighting procedure. Finally, response vigor exhibited a subtly different collection of MF and MS influences. These results provide new illumination onto RL behavioural processes in non-human primates.

We routinely solve planning problems in which present decisions have consequences in the future. These pose complex computational and statistical problems and are addressed by multiple systems in the brain which use different solutions to these problems, and which may compete and cooperate. We trained two rhesus monkeys on a paradigmatic planning task that transparently reveals canonical aspects of different strategies. We performed a detailed behavioral analysis using methods of reinforcement learning on choice and reaction time to reveal conjoint influences and structural interactions of different sources of information. We show the strengths and limitations of these analyses, at the same time as we provide a novel perspective on how different learning systems interact for choice in non-human primates.

The cortical oscillatory patterns associated with varying levels of reward during an effortful vigilance task

We explored how reward and value of effort shapes performance in a sustained vigilance, reaction time (RT) task. It was posited that reward and value would hasten RTs and increase cognitive effort by boosting activation in the sensorimotor cortex and inhibition in the frontal cortex, similar to the horse-race model of motor actions. Participants performed a series of speeded responses while expecting differing monetary rewards (0 pence (p), 1 p, and 10 p) if they responded faster than their median RT. Amplitudes of cortical alpha, beta, and theta oscillations were analysed using the event-related desynchronization method. In experiment 1 (N = 29, with 12 females), reward was consistent within block, while in experiment 2 (N = 17, with 12 females), reward amount was displayed before each trial. Each experiment evaluated the baseline amplitude of cortical oscillations differently. The value of effort was evaluated using a cognitive effort discounting task (COGED). In both experiments, RTs decreased significantly with higher rewards. Reward level sharpened the increased amplitudes of beta oscillations during fast responses in experiment 1. In experiment 2, reward decreased the amplitudes of beta oscillations in the ipsilateral sensorimotor cortex. Individual effort values did not significantly correlate with oscillatory changes in either experiment. Results suggest that reward level and response speed interacted with the task- and baseline-dependent patterns of cortical inhibition in the frontal cortex and with activation in the sensorimotor cortex during the period of motor preparation in a sustained vigilance task. However, neither the shortening of RT with increasing reward nor the value of effort correlated with oscillatory changes. This implies that amplitudes of cortical oscillations may shape upcoming motor responses but do not translate higher-order motivational factors into motor performance.

Age-related variability in decision-making: Insights from neurochemistry

Despite dopamine's significant role in models of value-based decision-making and findings demonstrating loss of dopamine function in aging, evidence of systematic changes in decision-making over the life span remains elusive. Previous studies attempting to resolve the neural basis of age-related alteration in decision-making have typically focused on physical age, which can be a poor proxy for age-related effects on neural systems. There is growing appreciation that aging has heterogeneous effects on distinct components of the dopamine system within subject in addition to substantial variability between subjects. We propose that some of the conflicting findings in age-related effects on decision-making may be reconciled if we can observe the underlying dopamine components within individuals. This can be achieved by incorporating in vivo imaging techniques including positron emission tomography (PET) and neuromelanin-sensitive MR. Further, we discuss how affective factors may contribute to individual differences in decision-making performance among older adults. Specifically, we propose that age-related shifts in affective attention ("positivity effect") can, in some cases, counteract the impact of altered dopamine function on specific decision-making processes, contributing to variability in findings. In an effort to provide clarity to the field and advance productive hypothesis testing, we propose ways in which in vivo dopamine imaging can be leveraged to disambiguate dopaminergic influences on decision-making, and suggest strategies for assessing individual differences in the contribution of affective attentional focus.

Computational Modeling for Neuropsychological Assessment of Bradyphrenia in Parkinson's Disease

The neural mechanisms of cognitive dysfunctions in neurological diseases remain poorly understood. Here, we conjecture that this unsatisfying state-of-the-art is in part due to the non-specificity of the typical behavioral indicators for cognitive dysfunctions. Our study addresses the topic by advancing the assessment of cognitive dysfunctions through computational modeling. We investigate bradyphrenia in Parkinson's disease (PD) as an exemplary case of cognitive dysfunctions in neurological diseases. Our computational model conceptualizes trial-by-trial behavioral data as resulting from parallel cognitive and sensorimotor reinforcement learning. We assessed PD patients 'on' and 'off' their dopaminergic medication and matched healthy control (HC) participants on a computerized version of the Wisconsin Card Sorting Test. PD patients showed increased retention of learned cognitive information and decreased retention of learned sensorimotor information from previous trials in comparison to HC participants. Systemic dopamine replacement therapy did not remedy these cognitive dysfunctions in PD patients but incurred non-desirable side effects such as decreasing cognitive learning from positive feedback. Our results reveal novel insights into facets of bradyphrenia that are indiscernible by observable behavioral indicators of cognitive dysfunctions. We discuss how computational modeling may contribute to the advancement of future research on brain-behavior relationships and neuropsychological assessment.

Substance use is associated with reduced devaluation sensitivity

Substance use has been linked to impairments in reward processing and decision-making, yet empirical research on the relationship between substance use and devaluation of reward in humans is limited. We report findings from two studies that tested whether individual differences in substance use behavior predicted reward learning strategies and devaluation sensitivity in a nonclinical sample. Participants in Experiment 1 (N = 66) and Experiment 2 (N = 91) completed subscales of the Externalizing Spectrum Inventory and then performed a two-stage reinforcement learning task that included a devaluation procedure. Spontaneous eye blink rate was used as an indirect proxy for dopamine functioning. In Experiment 1, correlational analysis revealed a negative relationship between substance use and devaluation sensitivity. In Experiment 2, regression modeling revealed that while spontaneous eyeblink rate moderated the relationship between substance use and reward learning strategies, substance use alone was related to devaluation sensitivity. These results suggest that once reward-action associations are established during reinforcement learning, substance use predicted reduced sensitivity to devaluation independently of variation in eyeblink rate. Thus, substance use is not only related to increased habit formation but also to difficulty disengaging from learned habits. Implications for the role of the dopaminergic system in habitual responding in individuals with substance use problems are discussed.

The neurochemical substrates of habitual and goal-directed control

Our daily decisions are governed by the arbitration between goal-directed and habitual strategies. However, the neurochemical basis of this arbitration is unclear. We assessed the contribution of dopaminergic, serotonergic, and opioidergic systems to this balance across reward and loss domains. Thirty-nine participants (17 healthy controls, 15 patients with pathological gambling, and 7 with binge eating disorder) underwent positron emission tomography (PET) imaging with [¹⁸F]FDOPA, [¹¹C]MADAM and [¹¹C]carfentanil to assess presynaptic dopamine, and serotonin transporter and mu-opioid receptor binding potential. Separately, participants completed a modified two-step task, which quantifies the degree to which decision-making is influenced by goal-directed or habitual strategies. All participants completed a version with reward outcomes; healthy controls additionally completed a version with loss outcomes. In the context of rewarding outcomes, we found that greater serotonin transporter binding potential in prefrontal regions was associated with habitual control, while greater serotonin transporter binding potential in the putamen was marginally associated with goal-directed control; however, the findings were no longer significant when controlling for the opposing valence (loss). In blocks with loss outcomes, we found that the opioidergic system, specifically greater [¹¹C]carfentanil binding potential, was positively associated with goal-directed control and negatively associated with habit-directed control. Our findings illuminate the complex neurochemical basis of goal-directed and habitual behavior, implicating differential roles for prefrontal and subcortical serotonin in decision-making across healthy and pathological populations.