1
|
Algermissen J, den Ouden HEM. Pupil dilation reflects effortful action invigoration in overcoming aversive Pavlovian biases. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2024:10.3758/s13415-024-01191-y. [PMID: 38773022 DOI: 10.3758/s13415-024-01191-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 04/22/2024] [Indexed: 05/23/2024]
Abstract
"Pavlovian" or "motivational" biases describe the phenomenon that the valence of prospective outcomes modulates action invigoration: Reward prospect invigorates action, whereas punishment prospect suppresses it. The adaptive role of these biases in decision-making is still unclear. One idea is that they constitute a fast-and-frugal decision strategy in situations characterized by high arousal, e.g., in presence of a predator, which demand a quick response. In this pre-registered study (N = 35), we tested whether such a situation-induced via subliminally presented angry versus neutral faces-leads to increased reliance on Pavlovian biases. We measured trial-by-trial arousal by tracking pupil diameter while participants performed an orthogonalized Motivational Go/NoGo Task. Pavlovian biases were present in responses, reaction times, and even gaze, with lower gaze dispersion under aversive cues reflecting "freezing of gaze." The subliminally presented faces did not affect responses, reaction times, or pupil diameter, suggesting that the arousal manipulation was ineffective. However, pupil dilations reflected facets of bias suppression, specifically the physical (but not cognitive) effort needed to overcome aversive inhibition: Particularly strong and sustained dilations occurred when participants managed to perform Go responses to aversive cues. Conversely, no such dilations occurred when they managed to inhibit responses to Win cues. These results suggest that pupil diameter does not reflect response conflict per se nor the inhibition of prepotent responses, but specifically effortful action invigoration as needed to overcome aversive inhibition. We discuss our results in the context of the "value of work" theory of striatal dopamine.
Collapse
Affiliation(s)
- Johannes Algermissen
- Donders Institute for Brain, Radboud University, Cognition, and Behaviour, Thomas van Aquinostraat 4, 6526 GD, Nijmegen, The Netherlands.
- Department of Experimental Psychology, University of Oxford, Oxford, UK.
| | - Hanneke E M den Ouden
- Donders Institute for Brain, Radboud University, Cognition, and Behaviour, Thomas van Aquinostraat 4, 6526 GD, Nijmegen, The Netherlands.
| |
Collapse
|
2
|
Algermissen J, Swart JC, Scheeringa R, Cools R, den Ouden HEM. Prefrontal signals precede striatal signals for biased credit assignment in motivational learning biases. Nat Commun 2024; 15:19. [PMID: 38168089 PMCID: PMC10762147 DOI: 10.1038/s41467-023-44632-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2021] [Accepted: 12/22/2023] [Indexed: 01/05/2024] Open
Abstract
Actions are biased by the outcomes they can produce: Humans are more likely to show action under reward prospect, but hold back under punishment prospect. Such motivational biases derive not only from biased response selection, but also from biased learning: humans tend to attribute rewards to their own actions, but are reluctant to attribute punishments to having held back. The neural origin of these biases is unclear. Specifically, it remains open whether motivational biases arise primarily from the architecture of subcortical regions or also reflect cortical influences, the latter being typically associated with increased behavioral flexibility and control beyond stereotyped behaviors. Simultaneous EEG-fMRI allowed us to track which regions encoded biased prediction errors in which order. Biased prediction errors occurred in cortical regions (dorsal anterior and posterior cingulate cortices) before subcortical regions (striatum). These results highlight that biased learning is not a mere feature of the basal ganglia, but arises through prefrontal cortical contributions, revealing motivational biases to be a potentially flexible, sophisticated mechanism.
Collapse
Affiliation(s)
- Johannes Algermissen
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| | - Jennifer C Swart
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - René Scheeringa
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
- Erwin L. Hahn Institute for Magnetic Resonance Imaging, University of Duisburg-Essen, Essen, Germany
| | - Roshan Cools
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, The Netherlands
| | - Hanneke E M den Ouden
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| |
Collapse
|
3
|
Queirazza F, Steele JD, Krishnadas R, Cavanagh J, Philiastides MG. Functional Magnetic Resonance Imaging Signatures of Pavlovian and Instrumental Valuation Systems during a Modified Orthogonalized Go/No-go Task. J Cogn Neurosci 2023; 35:2089-2109. [PMID: 37788326 DOI: 10.1162/jocn_a_02062] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/05/2023]
Abstract
Motivational (i.e., Pavlovian) values interfere with instrumental responding and can lead to suboptimal decision-making. In humans, task-based neuroimaging studies have only recently started illuminating the functional neuroanatomy of Pavlovian biasing of instrumental control. To provide a mechanistic understanding of the neural dynamics underlying the Pavlovian and instrumental valuation systems, analysis of neuroimaging data has been informed by computational modeling of conditioned behavior. Nonetheless, because of collinearities in Pavlovian and instrumental predictions, previous research failed to tease out hemodynamic activity that is parametrically and dynamically modulated by coexistent Pavlovian and instrumental value expectations. Moreover, neural correlates of Pavlovian to instrumental transfer effects have so far only been identified in extinction (i.e., in the absence of learning). In this study, we devised a modified version of the orthogonalized go/no-go paradigm, which introduced Pavlovian-only catch trials to better disambiguate trial-by-trial Pavlovian and instrumental predictions in both sexes. We found that hemodynamic activity in the ventromedial pFC covaried uniquely with the model-derived Pavlovian value expectations. Notably, modulation of neural activity encoding for instrumental predictions in the supplementary motor cortex was linked to successful action selection in conflict conditions. Furthermore, hemodynamic activity in regions pertaining to the limbic system and medial pFC was correlated with synergistic Pavlovian and instrumental predictions and improved conditioned behavior during congruent trials. Altogether, our results provide new insights into the functional neuroanatomy of decision-making and corroborate the validity of our variant of the orthogonalized go/no-go task as a behavioral assay of the Pavlovian and instrumental valuation systems.
Collapse
|
4
|
Chen Y, Chen W, Zhang L, Wei Y, Hu P. The Reward-Related Shift of Emotional Contagion from the Observer's Perspective Correlates to Their Intimacy with the Expresser. Behav Sci (Basel) 2023; 13:934. [PMID: 37998681 PMCID: PMC10669892 DOI: 10.3390/bs13110934] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 11/09/2023] [Accepted: 11/13/2023] [Indexed: 11/25/2023] Open
Abstract
Although previous studies have found a bidirectional relationship between emotional contagion and reward, there is insufficient research to prove the effect of reward on the social function of emotional contagion. To explore this issue, the current study used electroencephalography (EEG) and the interactive way in which the expresser played games to help participants obtain reward outcomes. The results demonstrated a significant correlation between changes in emotional contagion and closeness, indicating that emotional contagion has a social regulatory function. Regarding the impact of reward outcomes, the results showed that compared to the context of a loss, in the context of a win, participants' closeness toward the expresser shifted to a more intimate level, their emotional contagion changed in a more positive direction, and the activity of the late positive component (LPC) of the event-related potentials (ERPs) changed to a greater extent. Significantly, the mediation results demonstrated the effect of reward and indicated that changes in the LPC elicited while experiencing the expressers' emotion predicted the subsequent shifts in closeness through alterations in emotional contagion of the anger emotion in the winning context and the happy emotion in the loss context. This study provides empirical evidence regarding the social function of emotional contagion and proves for the first time that the reward context plays a role in it.
Collapse
Affiliation(s)
| | | | | | | | - Ping Hu
- Department of Psychology, Renmin University of China, No. 59 of Zhongguancun Street, Haidian District, Beijing 100872, China; (Y.C.); (W.C.); (L.Z.); (Y.W.)
| |
Collapse
|
5
|
Saeedpour S, Hossein MM, Deroy O, Bahrami B. Interindividual differences in Pavlovian influence on learning are consistent. ROYAL SOCIETY OPEN SCIENCE 2023; 10:230447. [PMID: 37736528 PMCID: PMC10509574 DOI: 10.1098/rsos.230447] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/07/2023] [Accepted: 08/29/2023] [Indexed: 09/23/2023]
Abstract
Pavlovian influences impair instrumental learning. It is easier to learn to approach reward-predictive signals and avoid punishment-predictive cues than their contrary. Whether the interindividual variability in this Pavlovian influence is consistent across time has been examined by a number of recent studies and met with mixed results. Here we introduce an open-source, web-based instance of a well-established Go-NoGo paradigm for measuring Pavlovian influence. We closely replicated the previous laboratory-based results. Moreover, the interindividual differences in Pavlovian influence were consistent across a two-week time window at the level of (i) raw measures of learning (i.e. performance accuracy), (ii) linear, descriptive estimates of Pavlovian bias (test-retest reliability: 0.40), and (iii) parameters obtained from reinforcement learning model fitting and model selection (test-retest reliability: 0.25). Nonetheless, the correlations reported here are still lower than the standards (i.e. 0.7) employed in psychometrics and self-reported measures. Our results provide support for trusting Pavlovian bias as a relatively stable individual characteristic and for using its measure in the computational understanding of human mental health.
Collapse
Affiliation(s)
- Sepehr Saeedpour
- Department of Electrical and Computer Engineering, University of Tehran, Tehran, Iran
| | | | - Ophelia Deroy
- Faculty of Philosophy, Ludwig Maximilian University, Munich, Germany
- Munich Center for Neuroscience, Ludwig Maximilian University, Munich, Germany
- School of Advanced Study, University of London, London, UK
| | - Bahador Bahrami
- Faculty of General Psychology and Education, Ludwig Maximilian University, Munich, Germany
| |
Collapse
|
6
|
Neuser MP, Kühnel A, Kräutlein F, Teckentrup V, Svaldi J, Kroemer NB. Reliability of gamified reinforcement learning in densely sampled longitudinal assessments. PLOS DIGITAL HEALTH 2023; 2:e0000330. [PMID: 37672521 PMCID: PMC10482292 DOI: 10.1371/journal.pdig.0000330] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/06/2022] [Accepted: 07/17/2023] [Indexed: 09/08/2023]
Abstract
Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. In this paper, we present an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) of current mental and physiological states for repeated assessment over weeks. In this task, players have to identify the most effective medication by integrating reward values with changing probabilities to win (according to random Gaussian walks). Participants can complete up to 31 runs with 150 trials each. To encourage replay, in-game screens provide feedback on the progress. Using an initial validation sample of 384 players (9729 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show poor to fair intra-class correlations (ICC: 0.22-0.53), indicating substantial within- and between-subject variance. Notably, items assessing the psychological state showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.
Collapse
Affiliation(s)
- Monja P. Neuser
- Department of Psychiatry and Psychotherapy, Tübingen Center for Mental Health, University of Tübingen, Tübingen, Germany
| | - Anne Kühnel
- Department of Psychiatry and Psychotherapy, Tübingen Center for Mental Health, University of Tübingen, Tübingen, Germany
- Department of Translational Psychiatry, Max Planck Institute of Psychiatry and International Max Planck Research School for Translational Psychiatry (IMPRS-TP), Munich, Germany
- Section of Medical Psychology, Department of Psychiatry & Psychotherapy, Faculty of Medicine, University of Bonn, Bonn, Germany
| | - Franziska Kräutlein
- Department of Psychiatry and Psychotherapy, Tübingen Center for Mental Health, University of Tübingen, Tübingen, Germany
| | - Vanessa Teckentrup
- Department of Psychiatry and Psychotherapy, Tübingen Center for Mental Health, University of Tübingen, Tübingen, Germany
- School of Psychology & Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland
| | - Jennifer Svaldi
- Department of Psychology, Clinical Psychology and Psychotherapy, University of Tübingen, Tübingen, Germany
| | - Nils B. Kroemer
- Department of Psychiatry and Psychotherapy, Tübingen Center for Mental Health, University of Tübingen, Tübingen, Germany
- School of Psychology & Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland
- German Center for Mental Health, Tübingen, Germany
| |
Collapse
|
7
|
Chen H, Belanger MJ, Garbusow M, Kuitunen-Paul S, Huys QJM, Heinz A, Rapp MA, Smolka MN. Susceptibility to interference between Pavlovian and instrumental control predisposes risky alcohol use developmental trajectory from ages 18 to 24. Addict Biol 2023; 28:e13263. [PMID: 36692874 DOI: 10.1111/adb.13263] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2022] [Revised: 10/14/2022] [Accepted: 11/23/2022] [Indexed: 01/06/2023]
Abstract
Pavlovian cues can influence ongoing instrumental behaviour via Pavlovian-to-instrumental transfer (PIT) processes. While appetitive Pavlovian cues tend to promote instrumental approach, they are detrimental when avoidance behaviour is required, and vice versa for aversive cues. We recently reported that susceptibility to interference between Pavlovian and instrumental control assessed via a PIT task was associated with risky alcohol use at age 18. We now investigated whether such susceptibility also predicts drinking trajectories until age 24, based on AUDIT (Alcohol Use Disorders Identification Test) consumption and binge drinking (gramme alcohol/drinking occasion) scores. The interference PIT effect, assessed at ages 18 and 21 during fMRI, was characterized by increased error rates (ER) and enhanced neural responses in the ventral striatum (VS), the lateral and dorsomedial prefrontal cortices (dmPFC) during conflict, that is, when an instrumental approach was required in the presence of an aversive Pavlovian cue or vice versa. We found that a stronger VS response during conflict at age 18 was associated with a higher starting point of both drinking trajectories but predicted a decrease in binge drinking. At age 21, high ER and enhanced neural responses in the dmPFC were associated with increasing AUDIT-C scores over the next 3 years until age 24. Overall, susceptibility to interference between Pavlovian and instrumental control might be viewed as a predisposing mechanism towards hazardous alcohol use during young adulthood, and the identified high-risk group may profit from targeted interventions.
Collapse
Affiliation(s)
- Hao Chen
- Department of Psychiatry and Psychotherapy, Technische Universität Dresden, Dresden, Germany
| | - Matthew J Belanger
- Department of Psychiatry and Psychotherapy, Technische Universität Dresden, Dresden, Germany
| | - Maria Garbusow
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Sören Kuitunen-Paul
- Institute of Clinical Psychology and Psychotherapy, Technische Universität Dresden, Dresden, Germany
| | - Quentin J M Huys
- Division of Psychiatry and Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, UK
| | - Andreas Heinz
- Department of Psychiatry and Psychotherapy, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Michael A Rapp
- Social and Preventive Medicine, Area of Excellence Cognitive Sciences, University of Potsdam, Potsdam, Germany
| | - Michael N Smolka
- Department of Psychiatry and Psychotherapy, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
8
|
Knolle F, Sterner E, Moutoussis M, Adams RA, Griffin JD, Haarsma J, Taverne H, Goodyer IM, Fletcher PC, Murray GK. Action selection in early stages of psychosis: an active inference approach. J Psychiatry Neurosci 2023; 48:E78-E89. [PMID: 36810306 PMCID: PMC9949875 DOI: 10.1503/jpn.220141] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/09/2022] [Revised: 11/11/2022] [Accepted: 11/28/2022] [Indexed: 02/23/2023] Open
Abstract
BACKGROUND To interact successfully with their environment, humans need to build a model to make sense of noisy and ambiguous inputs. An inaccurate model, as suggested to be the case for people with psychosis, disturbs optimal action selection. Recent computational models, such as active inference, have emphasized the importance of action selection, treating it as a key part of the inferential process. Based on an active inference framework, we sought to evaluate previous knowledge and belief precision in an action-based task, given that alterations in these parameters have been linked to the development of psychotic symptoms. We further sought to determine whether task performance and modelling parameters would be suitable for classification of patients and controls. METHODS Twenty-three individuals with an at-risk mental state, 26 patients with first-episode psychosis and 31 controls completed a probabilistic task in which action choice (go/no-go) was dissociated from outcome valence (gain or loss). We evaluated group differences in performance and active inference model parameters and performed receiver operating characteristic (ROC) analyses to assess group classification. RESULTS We found reduced overall performance in patients with psychosis. Active inference modelling revealed that patients showed increased forgetting, reduced confidence in policy selection and less optimal general choice behaviour, with poorer action-state associations. Importantly, ROC analysis showed fair-to-good classification performance for all groups, when combining modelling parameters and performance measures. LIMITATIONS The sample size is moderate. CONCLUSION Active inference modelling of this task provides further explanation for dysfunctional mechanisms underlying decision-making in psychosis and may be relevant for future research on the development of biomarkers for early identification of psychosis.
Collapse
Affiliation(s)
- Franziska Knolle
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Elisabeth Sterner
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Michael Moutoussis
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Rick A Adams
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Juliet D Griffin
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Joost Haarsma
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Hilde Taverne
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Ian M Goodyer
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Paul C Fletcher
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | - Graham K Murray
- From the Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany (Knolle, Sterner); the Department of Psychiatry, University of Cambridge, Cambridge, UK (Knolle, Griffin, Taverne, Goodyer, Fletcher, Murray); the Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London, UK (Moutoussis, Adams); the Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK (Adams); the Wellcome Centre for Human Neuroimaging, University College London, London, UK (Haarsma); the University of Amsterdam, Amsterdam, NL (Taverne); Wellcome Trust MRC Institute of Metabolic Science, University of Cambridge, Cambridge Biomedical Campus, Cambridge, UK (Goodyer, Fletcher); Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK (Murray)
| | | |
Collapse
|
9
|
Kanen JW, Luo Q, Rostami Kandroodi M, Cardinal RN, Robbins TW, Nutt DJ, Carhart-Harris RL, den Ouden HEM. Effect of lysergic acid diethylamide (LSD) on reinforcement learning in humans. Psychol Med 2022; 53:1-12. [PMID: 36411719 PMCID: PMC10600934 DOI: 10.1017/s0033291722002963] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/23/2021] [Revised: 08/28/2022] [Accepted: 08/31/2022] [Indexed: 11/23/2022]
Abstract
BACKGROUND The non-selective serotonin 2A (5-HT2A) receptor agonist lysergic acid diethylamide (LSD) holds promise as a treatment for some psychiatric disorders. Psychedelic drugs such as LSD have been suggested to have therapeutic actions through their effects on learning. The behavioural effects of LSD in humans, however, remain incompletely understood. Here we examined how LSD affects probabilistic reversal learning (PRL) in healthy humans. METHODS Healthy volunteers received intravenous LSD (75 μg in 10 mL saline) or placebo (10 mL saline) in a within-subjects design and completed a PRL task. Participants had to learn through trial and error which of three stimuli was rewarded most of the time, and these contingencies switched in a reversal phase. Computational models of reinforcement learning (RL) were fitted to the behavioural data to assess how LSD affected the updating ('learning rates') and deployment of value representations ('reinforcement sensitivity') during choice, as well as 'stimulus stickiness' (choice repetition irrespective of reinforcement history). RESULTS Raw data measures assessing sensitivity to immediate feedback ('win-stay' and 'lose-shift' probabilities) were unaffected, whereas LSD increased the impact of the strength of initial learning on perseveration. Computational modelling revealed that the most pronounced effect of LSD was the enhancement of the reward learning rate. The punishment learning rate was also elevated. Stimulus stickiness was decreased by LSD, reflecting heightened exploration. Reinforcement sensitivity differed by phase. CONCLUSIONS Increased RL rates suggest LSD induced a state of heightened plasticity. These results indicate a potential mechanism through which revision of maladaptive associations could occur in the clinical application of LSD.
Collapse
Affiliation(s)
- Jonathan W. Kanen
- Department of Psychology, University of Cambridge, Cambridge, UK
- Behavioural and Clinical Neuroscience Institute, University of Cambridge, Cambridge, UK
| | - Qiang Luo
- National Clinical Research Center for Aging and Medicine at Huashan Hospital, State Key Laboratory of Medical Neurobiology and Ministry of Education Frontiers Center for Brain Science, Institutes of Brain Science and Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, 200433, China
- Center for Computational Psychiatry, Ministry of Education-Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Human Phenome Institute, Fudan University, Shanghai, 200032, China
- Shanghai Key Laboratory of Mental Health and Psychological Crisis Intervention, School of Psychology and Cognitive Science, East China Normal University, Shanghai, 200241, China
| | - Mojtaba Rostami Kandroodi
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Rudolf N. Cardinal
- Behavioural and Clinical Neuroscience Institute, University of Cambridge, Cambridge, UK
- Department of Psychiatry, University of Cambridge, Cambridge, UK
- Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK
| | - Trevor W. Robbins
- Department of Psychology, University of Cambridge, Cambridge, UK
- Behavioural and Clinical Neuroscience Institute, University of Cambridge, Cambridge, UK
| | - David J. Nutt
- Department of Brain Sciences, Centre for Psychedelic Research, Imperial College London, London, UK
| | - Robin L. Carhart-Harris
- Neuroscape Psychedelics Division, University of California San Francisco, San Francisco, California, USA
| | - Hanneke E. M. den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| |
Collapse
|
10
|
Weber I, Zorowitz S, Niv Y, Bennett D. The effects of induced positive and negative affect on Pavlovian-instrumental interactions. Cogn Emot 2022; 36:1343-1360. [PMID: 35929878 PMCID: PMC9852069 DOI: 10.1080/02699931.2022.2109600] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2022] [Revised: 07/19/2022] [Accepted: 07/26/2022] [Indexed: 01/22/2023]
Abstract
Across species, animals have an intrinsic drive to approach appetitive stimuli and to withdraw from aversive stimuli. In affective science, influential theories of emotion link positive affect with strengthened behavioural approach and negative affect with avoidance. Based on these theories, we predicted that individuals' positive and negative affect levels should particularly influence their behaviour when innate Pavlovian approach/avoidance tendencies conflict with learned instrumental behaviours. Here, across two experiments - exploratory Experiment 1 (N = 91) and a preregistered confirmatory Experiment 2 (N = 335) - we assessed how induced positive and negative affect influenced Pavlovian-instrumental interactions in a reward/punishment Go/No-Go task. Contrary to our hypotheses, we found no evidence for a main effect of positive/negative affect on either approach/avoidance behaviour or Pavlovian-instrumental interactions. However, we did find evidence that the effects of induced affect on behaviour were moderated by individual differences in self-reported behavioural inhibition and gender. Exploratory computational modelling analyses explained these demographic moderating effects as arising from positive correlations between demographic factors and individual differences in the strength of Pavlovian-instrumental interactions. These findings serve to sharpen our understanding of the effects of positive and negative affect on instrumental behaviour.
Collapse
Affiliation(s)
- Isla Weber
- Princeton Neuroscience Institute, Princeton University, Princeton, USA
| | - Sam Zorowitz
- Princeton Neuroscience Institute, Princeton University, Princeton, USA
| | - Yael Niv
- Princeton Neuroscience Institute, Princeton University, Princeton, USA
- Department of Psychology, Princeton University, Princeton, USA
| | - Daniel Bennett
- School of Psychological Sciences, Monash University, Clayton, Australia
| |
Collapse
|
11
|
Bikute K, Di Bernardi Luft C, Beyer F. The value of an action: Impact of motor behaviour on outcome processing and stimulus preference. Eur J Neurosci 2022; 56:5823-5835. [PMID: 36114689 PMCID: PMC9828266 DOI: 10.1111/ejn.15826] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2022] [Revised: 08/11/2022] [Accepted: 09/13/2022] [Indexed: 01/12/2023]
Abstract
While influences of Pavlovian associations on instrumental behaviour are well established, we still do not know how motor actions affect the formation of Pavlovian associations. To address this question, we designed a task in which participants were presented with neutral stimuli, half of which were paired with an active response, half with a passive waiting period. Stimuli had an 80% chance of predicting either a monetary gain or loss. We compared the feedback-related negativity (FRN) in response to predictive stimuli and outcomes, as well as directed phase synchronization before and after outcome presentation between trials with versus without a motor response. We found a larger FRN amplitude in response to outcomes presented after a motor response (active trials). This effect was driven by a positive deflection in active reward trials, which was absent in passive reward trials. Connectivity analysis revealed that the motor action reversed the direction of the phase synchronization at the time of the feedback presentation: Top-down information flow during the outcome anticipation phase in active trials, but bottom-up information flow in passive trials. This main effect of action was mirrored in behavioural data showing that participants preferred stimuli associated with an active response. Our findings suggest an influence of neural systems that initiate motor actions on neural systems involved in reward processing. We suggest that motor actions might modulate the brain responses to feedback by affecting the dynamics of brain activity towards optimizing the processing of the resulting action outcome.
Collapse
Affiliation(s)
- Kotryna Bikute
- Department of Biological and Experimental PsychologyQueen Mary University of LondonLondonUK
| | | | - Frederike Beyer
- Department of Biological and Experimental PsychologyQueen Mary University of LondonLondonUK
| |
Collapse
|
12
|
Geurts DEM, Van den Heuvel TJ, Huys QJM, Verkes RJ, Cools R. Amygdala response predicts clinical symptom reduction in patients with borderline personality disorder: A pilot fMRI study. Front Behav Neurosci 2022; 16:938403. [PMID: 36110290 PMCID: PMC9468714 DOI: 10.3389/fnbeh.2022.938403] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2022] [Accepted: 07/27/2022] [Indexed: 11/13/2022] Open
Abstract
Borderline personality disorder (BPD) is a prevalent, devastating, and heterogeneous psychiatric disorder. Treatment success is highly variable within this patient group. A cognitive neuroscientific approach to BPD might contribute to precision psychiatry by identifying neurocognitive factors that predict who will benefit from a specific treatment. Here, we build on observations that BPD is accompanied by the enhanced impact of the aversive effect on behavior and abnormal neural signaling in the amygdala. We assessed whether BPD is accompanied by abnormal aversive regulation of instrumental behavior and associated neural signaling, in a manner that is predictive of symptom reduction after therapy. We tested a clinical sample of 15 female patients with BPD, awaiting dialectical behavior therapy (DBT), and 16 matched healthy controls using fMRI and an aversive Pavlovian-to-instrumental transfer (PIT) task that assesses how instrumental behaviors are influenced by aversive Pavlovian stimuli. Patients were assessed 1 year after the start of DBT to quantify changes in BPD symptom severity. At baseline, behavioral aversive PIT and associated neural signaling did not differ between groups. However, the BOLD signal in the amygdala measured during aversive PIT was associated with symptom reduction at 1-year follow-up: higher PIT-related aversive amygdala signaling before treatment was associated with reduced clinical improvement at follow-up. Thus, within the evaluated group of BPD patients, the BOLD signal in the amygdala before treatment was related to clinical symptom reduction 1 year after the start of treatment. The results suggest that less PIT-related responsiveness of the amygdala increases the chances of treatment success. We note that the relatively small sample size is a limitation of this study and that replication is warranted.
Collapse
Affiliation(s)
- Dirk E. M. Geurts
- Centre for Cognitive Neuroimaging, Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, Netherlands
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
| | - Thom J. Van den Heuvel
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
- Department of Scelta, Expert Centre for Personality Disorders, GGNet, Nijmegen, Netherlands
| | - Quentin J. M. Huys
- Mental Health Neuroscience Department, Division of Psychiatry and Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Institute of Neurology, University College London, London, United Kingdom
| | - Robbert J. Verkes
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
- Kairos Center for Forensic Psychiatry, Pro Persona Mental Health, Nijmegen, Netherlands
| | - Roshan Cools
- Centre for Cognitive Neuroimaging, Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, Netherlands
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
| |
Collapse
|
13
|
van den Bosch R, Lambregts B, Määttä J, Hofmans L, Papadopetraki D, Westbrook A, Verkes RJ, Booij J, Cools R. Striatal dopamine dissociates methylphenidate effects on value-based versus surprise-based reversal learning. Nat Commun 2022; 13:4962. [PMID: 36002446 PMCID: PMC9402573 DOI: 10.1038/s41467-022-32679-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2021] [Accepted: 08/10/2022] [Indexed: 11/13/2022] Open
Abstract
Psychostimulants such as methylphenidate are widely used for their cognitive enhancing effects, but there is large variability in the direction and extent of these effects. We tested the hypothesis that methylphenidate enhances or impairs reward/punishment-based reversal learning depending on baseline striatal dopamine levels and corticostriatal gating of reward/punishment-related representations in stimulus-specific sensory cortex. Young healthy adults (N = 100) were scanned with functional magnetic resonance imaging during a reward/punishment reversal learning task, after intake of methylphenidate or the selective D2/3-receptor antagonist sulpiride. Striatal dopamine synthesis capacity was indexed with [18F]DOPA positron emission tomography. Methylphenidate improved and sulpiride decreased overall accuracy and response speed. Both drugs boosted reward versus punishment learning signals to a greater degree in participants with higher dopamine synthesis capacity. By contrast, striatal and stimulus-specific sensory surprise signals were boosted in participants with lower dopamine synthesis. These results unravel the mechanisms by which methylphenidate gates both attention and reward learning. The mechanisms underpinning the variability in methylphenidate’s effects on cognition remain unclear. Here, the authors show that such effects reflect changes in striatal dopamine-related output gating of task-relevant cortical signals, and that these changes depend on baseline dopamine synthesis capacity.
Collapse
Affiliation(s)
- Ruben van den Bosch
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| | - Britt Lambregts
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Jessica Määttä
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
| | - Lieke Hofmans
- Department of Developmental Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - Danae Papadopetraki
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Andrew Westbrook
- Cognitive, Linguistic & Psychological Sciences Department, Brown University, Providence, RI, USA
| | - Robbert-Jan Verkes
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| | - Jan Booij
- Department of Radiology and Nuclear Medicine, Amsterdam University Medical Centers, location Academic Medical Center, Amsterdam, The Netherlands.,Radboud University Medical Center, Department of Medical Imaging, Nijmegen, The Netherlands
| | - Roshan Cools
- Radboud University Medical Center, Department of Psychiatry, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
| |
Collapse
|
14
|
Geurts DEM, den Ouden HEM, Janssen L, Swart JC, Froböse MI, Cools R, Speckens AEM. Aversive Pavlovian inhibition in adult attention-deficit/hyperactivity disorder and its restoration by mindfulness-based cognitive therapy. Front Behav Neurosci 2022; 16:938082. [PMID: 35957921 PMCID: PMC9359138 DOI: 10.3389/fnbeh.2022.938082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2022] [Accepted: 06/28/2022] [Indexed: 11/13/2022] Open
Abstract
Background Control over the tendency to make or withhold responses guided by contextual Pavlovian information plays a key role in understanding impulsivity and hyperactivity. Here we set out to assess (1) the understudied relation between contextual Pavlovian inhibitory control and hyperactivity/impulsivity in adults with ADHD and (2) whether this inhibition can be enhanced by mindfulness based cognitive therapy (MBCT). Methods Within the framework of a randomized controlled trial 50 Adult ADHD patients were assessed before and after 8 weeks of treatment as usual (TAU) with (n = 24) or without (n = 26) MBCT. We employed a well-established behavioral Pavlovian-to-instrumental transfer task that quantifies Pavlovian inhibitory control over instrumental behavior. Results Task results revealed (1) less aversive Pavlovian inhibition in ADHD patients with clinically relevant hyperactivity/impulsivity than in those without; and (2) enhanced Pavlovian inhibition across all ADHD patients after TAU+MBCT compared with TAU. Conclusion These findings offer new insights in the neurocognitive mechanisms of hyperactivity/impulsivity in ADHD and its treatment: We reveal a role for Pavlovian inhibitory mechanisms in understanding hyperactive/impulsive behaviors in ADHD and point toward MBCT as an intervention that might influence these mechanisms.
Collapse
Affiliation(s)
- Dirk E. M. Geurts
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, Netherlands
- *Correspondence: Dirk E. M. Geurts,
| | - Hanneke E. M. den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Lotte Janssen
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, Netherlands
| | - Jennifer C. Swart
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Monja I. Froböse
- Institute of Experimental Psychology, Heinrich Heine University of Düsseldorf, Düsseldorf, Germany
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, Netherlands
| | - Anne E. M. Speckens
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, Netherlands
| |
Collapse
|
15
|
Scholz V, Hook RW, Kandroodi MR, Algermissen J, Ioannidis K, Christmas D, Valle S, Robbins TW, Grant JE, Chamberlain SR, den Ouden HEM. Cortical dopamine reduces the impact of motivational biases governing automated behaviour. Neuropsychopharmacology 2022; 47:1503-1512. [PMID: 35260787 PMCID: PMC9206002 DOI: 10.1038/s41386-022-01291-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/30/2021] [Accepted: 02/02/2022] [Indexed: 11/09/2022]
Abstract
Motivations shape our behaviour: the promise of reward invigorates, while in the face of punishment, we hold back. Abnormalities of motivational processing are implicated in clinical disorders characterised by excessive habits and loss of top-down control, notably substance and behavioural addictions. Striatal and frontal dopamine have been hypothesised to play complementary roles in the respective generation and control of these motivational biases. However, while dopaminergic interventions have indeed been found to modulate motivational biases, these previous pharmacological studies used regionally non-selective pharmacological agents. Here, we tested the hypothesis that frontal dopamine controls the balance between Pavlovian, bias-driven automated responding and instrumentally learned action values. Specifically, we examined whether selective enhancement of cortical dopamine either (i) enables adaptive suppression of Pavlovian control when biases are maladaptive; or (ii) non-specifically modulates the degree of bias-driven automated responding. Healthy individuals (n = 35) received the catechol-o-methyltransferase (COMT) inhibitor tolcapone in a randomised, double-blind, placebo-controlled cross-over design, and completed a motivational Go NoGo task known to elicit motivational biases. In support of hypothesis (ii), tolcapone globally decreased motivational bias. Specifically, tolcapone improved performance on trials where the bias was unhelpful, but impaired performance in bias-congruent conditions. These results indicate a non-selective role for cortical dopamine in the regulation of motivational processes underpinning top-down control over automated behaviour. The findings have direct relevance to understanding neurobiological mechanisms underpinning addiction and obsessive-compulsive disorders, as well as highlighting a potential trans-diagnostic novel mechanism to address such symptoms.
Collapse
Affiliation(s)
- Vanessa Scholz
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands. .,Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Centre of Mental Health, University of Würzburg, Margarete-Höppel-Platz1, 97080, Würzburg, Germany.
| | - Roxanne W. Hook
- grid.5335.00000000121885934Department of Psychiatry, University of Cambridge, Cambridge, UK
| | - Mojtaba Rostami Kandroodi
- grid.5590.90000000122931605Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands ,grid.46072.370000 0004 0612 7950School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran
| | - Johannes Algermissen
- grid.5590.90000000122931605Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Konstantinos Ioannidis
- grid.5335.00000000121885934Department of Psychiatry, University of Cambridge, Cambridge, UK ,grid.450563.10000 0004 0412 9303Cambridgeshire & Peterborough NHS Foundation Trust, Cambridge, UK ,grid.5012.60000 0001 0481 6099Department of International Health, Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands
| | - David Christmas
- grid.5335.00000000121885934Department of Psychiatry, University of Cambridge, Cambridge, UK ,grid.450563.10000 0004 0412 9303Cambridgeshire & Peterborough NHS Foundation Trust, Cambridge, UK
| | - Stephanie Valle
- grid.170205.10000 0004 1936 7822Department of Psychiatry and Behavioral Neuroscience, University of Chicago, Chicago, IL USA
| | - Trevor W. Robbins
- grid.5335.00000000121885934Department of Psychology, and Behavioural and Clinical Neuroscience Institute, University of Cambridge, Cambridge, UK
| | - Jon E. Grant
- grid.170205.10000 0004 1936 7822Department of Psychiatry and Behavioral Neuroscience, University of Chicago, Chicago, IL USA
| | - Samuel R. Chamberlain
- grid.5335.00000000121885934Department of Psychiatry, University of Cambridge, Cambridge, UK ,grid.5491.90000 0004 1936 9297Department of Psychiatry, Faculty of Medicine, University of Southampton, Southampton, UK. Southern Health NHS Foundation Trust, Southampton, UK
| | - Hanneke E. M. den Ouden
- grid.5590.90000000122931605Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| |
Collapse
|
16
|
Cools R, Arnsten AFT. Neuromodulation of prefrontal cortex cognitive function in primates: the powerful roles of monoamines and acetylcholine. Neuropsychopharmacology 2022; 47:309-328. [PMID: 34312496 PMCID: PMC8617291 DOI: 10.1038/s41386-021-01100-8] [Citation(s) in RCA: 49] [Impact Index Per Article: 24.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/14/2021] [Revised: 07/06/2021] [Accepted: 07/06/2021] [Indexed: 02/07/2023]
Abstract
The primate prefrontal cortex (PFC) subserves our highest order cognitive operations, and yet is tremendously dependent on a precise neurochemical environment for proper functioning. Depletion of noradrenaline and dopamine, or of acetylcholine from the dorsolateral PFC (dlPFC), is as devastating as removing the cortex itself, and serotonergic influences are also critical to proper functioning of the orbital and medial PFC. Most neuromodulators have a narrow inverted U dose response, which coordinates arousal state with cognitive state, and contributes to cognitive deficits with fatigue or uncontrollable stress. Studies in monkeys have revealed the molecular signaling mechanisms that govern the generation and modulation of mental representations by the dlPFC, allowing dynamic regulation of network strength, a process that requires tight regulation to prevent toxic actions, e.g., as occurs with advanced age. Brain imaging studies in humans have observed drug and genotype influences on a range of cognitive tasks and on PFC circuit functional connectivity, e.g., showing that catecholamines stabilize representations in a baseline-dependent manner. Research in monkeys has already led to new treatments for cognitive disorders in humans, encouraging future research in this important field.
Collapse
Affiliation(s)
- Roshan Cools
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Amy F T Arnsten
- Department of Neuroscience, Yale University School of Medicine, New Haven, CT, USA.
| |
Collapse
|
17
|
Algermissen J, Swart JC, Scheeringa R, Cools R, den Ouden HEM. Striatal BOLD and Midfrontal Theta Power Express Motivation for Action. Cereb Cortex 2021; 32:2924-2942. [PMID: 34849626 PMCID: PMC9290551 DOI: 10.1093/cercor/bhab391] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2020] [Revised: 05/04/2021] [Accepted: 06/06/2021] [Indexed: 11/14/2022] Open
Abstract
Action selection is biased by the valence of anticipated outcomes. To assess mechanisms by which these motivational biases are expressed and controlled, we measured simultaneous EEG-fMRI during a motivational Go/NoGo learning task (N = 36), leveraging the temporal resolution of EEG and subcortical access of fMRI. VmPFC BOLD encoded cue valence, importantly predicting trial-by-trial valence-driven response speed differences and EEG theta power around cue onset. In contrast, striatal BOLD encoded selection of active Go responses and correlated with theta power around response time. Within trials, theta power ramped in the fashion of an evidence accumulation signal for the value of making a "Go" response, capturing the faster responding to reward cues. Our findings reveal a dual nature of midfrontal theta power, with early components reflecting the vmPFC contribution to motivational biases, and late components reflecting their striatal translation into behavior, in line with influential recent "value of work" theories of striatal processing.
Collapse
Affiliation(s)
- Johannes Algermissen
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Thomas van Aquinostraat 4, 6525 GD Nijmegen, The Netherlands
| | - Jennifer C Swart
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Thomas van Aquinostraat 4, 6525 GD Nijmegen, The Netherlands
| | - René Scheeringa
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Thomas van Aquinostraat 4, 6525 GD Nijmegen, The Netherlands.,Erwin L. Hahn Institute for Magnetic Resonance Imaging, University of Duisburg-Essen, Kokereiallee 7, 45141 Essen, Germany
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Thomas van Aquinostraat 4, 6525 GD Nijmegen, The Netherlands.,Department of Psychiatry, Radboud University Medical Centre, Reinier Postlaan 10, 6525 GC Nijmegen, The Netherlands
| | - Hanneke E M den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Thomas van Aquinostraat 4, 6525 GD Nijmegen, The Netherlands
| |
Collapse
|
18
|
Veselic S, Jocham G, Gausterer C, Wagner B, Ernhoefer-Reßler M, Lanzenberger R, Eisenegger C, Lamm C, Losecaat Vermeer A. A causal role of estradiol in human reinforcement learning. Horm Behav 2021; 134:105022. [PMID: 34273676 DOI: 10.1016/j.yhbeh.2021.105022] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/02/2020] [Revised: 06/12/2021] [Accepted: 06/22/2021] [Indexed: 10/20/2022]
Abstract
The sex hormone estradiol is hypothesized to play a key role in human cognition, and reward processing specifically, via increased dopamine D1-receptor signalling. However, the effect of estradiol on reward processing in men has never been established. To fill this gap, we performed a double-blind placebo-controlled study in which men (N = 100) received either a single dose of estradiol (2 mg) or a placebo. Subjects performed a probabilistic reinforcement learning task where they had to choose between two options with varying reward probabilities to maximize monetary reward. Results showed that estradiol administration increased reward sensitivity compared to placebo. This effect was observed in subjects' choices, how much weight they assigned to their previous choices, and subjective reports about the reward probabilities. Furthermore, effects of estradiol were moderated by reward sensitivity, as measured through the BIS/BAS questionnaire. Using reinforcement learning models, we found that behavioral effects of estradiol were reflected in increased learning rates. These results demonstrate a causal role of estradiol within the framework of reinforcement learning, by enhancing reward sensitivity and learning. Furthermore, they provide preliminary evidence for dopamine-related genetic variants moderating the effect of estradiol on reward processing.
Collapse
Affiliation(s)
- Sebastijan Veselic
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Austria; Department of Clinical and Movement Neurosciences, University College London, London, UK; Wellcome Centre for Human Neuroimaging, University College London, London, UK.
| | - Gerhard Jocham
- Biological Psychology of Decision Making, Institute of Experimental Psychology, Heinrich Heine University Düsseldorf, Germany
| | - Christian Gausterer
- FDZ-Forensisches DNA Zentrallabor GmbH, Medical University of Vienna, Austria
| | - Bernhard Wagner
- Laboratory for Chromatographic & Spectrometric Analysis, FH JOANNEUM, Graz, Austria
| | | | - Rupert Lanzenberger
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
| | - Christoph Eisenegger
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Austria
| | - Claus Lamm
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Austria; Vienna Cognitive Science Hub, University of Vienna, Austria
| | - Annabel Losecaat Vermeer
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Austria; Department of Decision Neuroscience and Nutrition, German Institute of Human Nutrition Potsdam-Rehbrücke, Germany; Charité-Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin, Germany.
| |
Collapse
|
19
|
Sebold M, Garbusow M, Cerci D, Chen K, Sommer C, Huys QJM, Nebe S, Rapp M, Veer IM, Zimmermann US, Smolka MN, Walter H, Heinz A, Friedel E. Association of the OPRM1 A118G polymorphism and Pavlovian-to-instrumental transfer: Clinical relevance for alcohol dependence. J Psychopharmacol 2021; 35:566-578. [PMID: 33726538 PMCID: PMC8155738 DOI: 10.1177/0269881121991992] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
Abstract
BACKGROUND Pavlovian-to-instrumental transfer (PIT) quantifies the extent to which a stimulus that has been associated with reward or punishment alters operant behaviour. In alcohol dependence (AD), the PIT effect serves as a paradigmatic model of cue-induced relapse. Preclinical studies have suggested a critical role of the opioid system in modulating Pavlovian-instrumental interactions. The A118G polymorphism of the OPRM1 gene affects opioid receptor availability and function. Furthermore, this polymorphism interacts with cue-induced approach behaviour and is a potential biomarker for pharmacological treatment response in AD. In this study, we tested whether the OPRM1 polymorphism is associated with the PIT effect and relapse in AD. METHODS Using a PIT task, we examined three independent samples: young healthy subjects (N = 161), detoxified alcohol-dependent patients (N = 186) and age-matched healthy controls (N = 105). We used data from a larger study designed to assess the role of learning mechanisms in the development and maintenance of AD. Subjects were genotyped for the A118G (rs1799971) polymorphism of the OPRM1 gene. Relapse was assessed after three months. RESULTS In all three samples, participants with the minor OPRM1 G-Allele (G+ carriers) showed increased expression of the PIT effect in the absence of learning differences. Relapse was not associated with the OPRM1 polymorphism. Instead, G+ carriers displaying increased PIT effects were particularly prone to relapse. CONCLUSION These results support a role for the opioid system in incentive salience motivation. Furthermore, they inform a mechanistic model of aberrant salience processing and are in line with the pharmacological potential of opioid receptor targets in the treatment of AD.
Collapse
Affiliation(s)
- Miriam Sebold
- Department of Psychiatry and
Psychotherapy, Charité – Universitätsmedizin Berlin, corporate member of Freie
Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health,
Berlin, Germany
- Department for Social and Preventive
Medicine, University of Potsdam, Potsdam, Germany
| | - Maria Garbusow
- Department of Psychiatry and
Psychotherapy, Charité – Universitätsmedizin Berlin, corporate member of Freie
Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health,
Berlin, Germany
| | - Deniz Cerci
- Klinik für Forensische Psychiatrie,
Universitätsmedizin Rostock, Rostock, Germany
| | - Ke Chen
- Department of Psychiatry and
Psychotherapy, Charité – Universitätsmedizin Berlin, corporate member of Freie
Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health,
Berlin, Germany
| | | | - Quentin JM Huys
- Division of Psychiatry, University
College London, London, UK
- Max Planck UCL Centre for Computational
Psychiatry and Ageing Research, University College London, London, UK
| | - Stephan Nebe
- Department of Economics, University of
Zurich, Zurich, Switzerland
| | - Michael Rapp
- Department for Social and Preventive
Medicine, University of Potsdam, Potsdam, Germany
| | - Ilya M Veer
- Department of Psychiatry and
Psychotherapy, Charité – Universitätsmedizin Berlin, corporate member of Freie
Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health,
Berlin, Germany
| | - Ulrich S Zimmermann
- Technical University of Dresden,
Dresden, Germany
- Department of Addiction Medicine and
Psychotherapy, kbo Isar-Amper-Klinikum, Munich, Germany
| | | | - Henrik Walter
- Department of Psychiatry and
Psychotherapy, Charité – Universitätsmedizin Berlin, corporate member of Freie
Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health,
Berlin, Germany
| | - Andreas Heinz
- Department of Psychiatry and
Psychotherapy, Charité – Universitätsmedizin Berlin, corporate member of Freie
Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health,
Berlin, Germany
| | - Eva Friedel
- Department of Psychiatry and
Psychotherapy, Charité – Universitätsmedizin Berlin, corporate member of Freie
Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health,
Berlin, Germany
| |
Collapse
|
20
|
Dose-response effects of d-amphetamine on effort-based decision-making and reinforcement learning. Neuropsychopharmacology 2021; 46:1078-1085. [PMID: 32722661 PMCID: PMC8115674 DOI: 10.1038/s41386-020-0779-8] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/21/2020] [Revised: 06/30/2020] [Accepted: 07/20/2020] [Indexed: 12/16/2022]
Abstract
Effort-related decision-making and reward learning are both dopamine-dependent, but preclinical research suggests they depend on different dopamine signaling dynamics. Therefore, the same dose of a dopaminergic medication could have differential effects on effort for reward vs. reward learning. However, no study has tested how effort and reward learning respond to the same dopaminergic medication within subjects. The current study aimed to test the effect of therapeutic doses of d-amphetamine on effort for reward and reward learning in the same healthy volunteers. Participants (n = 30) completed the Effort Expenditure for Reward Task (EEfRT) measure of effort-related decision-making, and the Probabilistic Reward Task (PRT) measure of reward learning, under placebo and two doses of d-amphetamine (10 mg, and 20 mg). Secondarily, we examined whether the individual characteristics of baseline working memory and willingness to exert effort for reward moderated the effects of d-amphetamine. d-Amphetamine increased willingness to exert effort, particularly at low to intermediate expected values of reward. Computational modeling analyses suggested this was due to decreased effort discounting rather than probability discounting or decision consistency. Both baseline effort and working memory emerged as moderators of this effect, such that d-amphetamine increased effort more in individuals with lower working memory and lower baseline effort, also primarily at low to intermediate expected values of reward. In contrast, d-amphetamine had no significant effect on reward learning. These results have implications for treatment of neuropsychiatric disorders, which may be characterized by multiple underlying reward dysfunctions.
Collapse
|
21
|
Ereira S, Pujol M, Guitart-Masip M, Dolan RJ, Kurth-Nelson Z. Overcoming Pavlovian bias in semantic space. Sci Rep 2021; 11:3416. [PMID: 33564034 PMCID: PMC7873193 DOI: 10.1038/s41598-021-82889-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2020] [Accepted: 01/25/2021] [Indexed: 01/25/2023] Open
Abstract
Action is invigorated in the presence of reward-predicting stimuli and inhibited in the presence of punishment-predicting stimuli. Although valuable as a heuristic, this Pavlovian bias can also lead to maladaptive behaviour and is implicated in addiction. Here we explore whether Pavlovian bias can be overcome through training. Across five experiments, we find that Pavlovian bias is resistant to unlearning under most task configurations. However, we demonstrate that when subjects engage in instrumental learning in a verbal semantic space, as opposed to a motoric space, not only do they exhibit the typical Pavlovian bias, but this Pavlovian bias diminishes with training. Our results suggest that learning within the semantic space is necessary, but not sufficient, for subjects to unlearn their Pavlovian bias, and that other task features, such as gamification and spaced stimulus presentation may also be necessary. In summary, we show that Pavlovian bias, whilst robust, is susceptible to change with experience, but only under specific environmental conditions.
Collapse
Affiliation(s)
- Sam Ereira
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, UCL, London, WC1B 5EH, UK.
- Wellcome Centre for Human Neuroimaging, UCL, London, WC1N 3BG, UK.
| | - Marine Pujol
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, UCL, London, WC1B 5EH, UK
- Sorbonne Université, Paris, France
| | - Marc Guitart-Masip
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, UCL, London, WC1B 5EH, UK
- Aging Research Centre, Karolinska Institute, 171 65, Stockholm, Sweden
| | - Raymond J Dolan
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, UCL, London, WC1B 5EH, UK
- Wellcome Centre for Human Neuroimaging, UCL, London, WC1N 3BG, UK
| | - Zeb Kurth-Nelson
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, UCL, London, WC1B 5EH, UK
- DeepMind, London, N1C 4AG, UK
| |
Collapse
|
22
|
Effects of methylphenidate on reinforcement learning depend on working memory capacity. Psychopharmacology (Berl) 2021; 238:3569-3584. [PMID: 34676440 PMCID: PMC8629893 DOI: 10.1007/s00213-021-05974-w] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Accepted: 08/25/2021] [Indexed: 11/25/2022]
Abstract
RATIONALE Brain catecholamines have long been implicated in reinforcement learning, exemplified by catecholamine drug and genetic effects on probabilistic reversal learning. However, the mechanisms underlying such effects are unclear. OBJECTIVES AND METHODS Here we investigated effects of an acute catecholamine challenge with methylphenidate (20 mg, oral) on a novel probabilistic reversal learning paradigm in a within-subject, double-blind randomised design. The paradigm was designed to disentangle effects on punishment avoidance from effects on reward perseveration. Given the known large individual variability in methylphenidate's effects, we stratified our effects by working memory capacity and trait impulsivity, putatively modulating the effects of methylphenidate, in a large sample (n = 102) of healthy volunteers. RESULTS Contrary to our prediction, methylphenidate did not alter performance in the reversal phase of the task. Our key finding is that methylphenidate altered learning of choice-outcome contingencies in a manner that depended on individual variability in working memory span. Specifically, methylphenidate improved performance by adaptively reducing the effective learning rate in participants with higher working memory capacity. CONCLUSIONS This finding emphasises the important role of working memory in reinforcement learning, as reported in influential recent computational modelling and behavioural work, and highlights the dependence of this interplay on catecholaminergic function.
Collapse
|
23
|
Huys QJM, Browning M, Paulus MP, Frank MJ. Advances in the computational understanding of mental illness. Neuropsychopharmacology 2021; 46:3-19. [PMID: 32620005 PMCID: PMC7688938 DOI: 10.1038/s41386-020-0746-4] [Citation(s) in RCA: 47] [Impact Index Per Article: 15.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/04/2020] [Revised: 06/11/2020] [Accepted: 06/15/2020] [Indexed: 12/11/2022]
Abstract
Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.
Collapse
Affiliation(s)
- Quentin J M Huys
- Division of Psychiatry and Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, UK.
- Camden and Islington NHS Trust, London, UK.
| | - Michael Browning
- Computational Psychiatry Lab, Department of Psychiatry, University of Oxford, Oxford, UK
- Oxford Health NHS Trust, Oxford, UK
| | - Martin P Paulus
- Laureate Institute For Brain Research (LIBR), Tulsa, OK, USA
| | - Michael J Frank
- Cognitive, Linguistic & Psychological Sciences, Neuroscience Graduate Program, Brown University, Providence, RI, USA
- Carney Center for Computational Brain Science, Carney Institute for Brain Science Psychiatry and Human Behavior, Brown University, Providence, RI, USA
| |
Collapse
|
24
|
Manohar SG. Tremor in Parkinson's disease inverts the effect of dopamine on reinforcement. Brain 2020; 143:3178-3180. [PMID: 33278817 DOI: 10.1093/brain/awaa363] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
This scientific commentary refers to ‘Effects of dopamine on reinforcement learning in Parkinson’s disease depend on motor phenotype’ by van Nuland et al. (doi:10.1093/brain/awaa335).
Collapse
Affiliation(s)
- Sanjay G Manohar
- Nuffield Department of Clinical Neurosciences, University of Oxford, UK
| |
Collapse
|
25
|
van Nuland AJ, Helmich RC, Dirkx MF, Zach H, Toni I, Cools R, den Ouden HEM. Effects of dopamine on reinforcement learning in Parkinson's disease depend on motor phenotype. Brain 2020; 143:3422-3434. [PMID: 33147621 PMCID: PMC7719026 DOI: 10.1093/brain/awaa335] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2019] [Revised: 07/10/2020] [Accepted: 08/06/2020] [Indexed: 01/16/2023] Open
Abstract
Parkinson's disease is clinically defined by bradykinesia, along with rigidity and tremor. However, the severity of these motor signs is greatly variable between individuals, particularly the presence or absence of tremor. This variability in tremor relates to variation in cognitive/motivational impairment, as well as the spatial distribution of neurodegeneration in the midbrain and dopamine depletion in the striatum. Here we ask whether interindividual heterogeneity in tremor symptoms could account for the puzzlingly large variability in the effects of dopaminergic medication on reinforcement learning, a fundamental cognitive function known to rely on dopamine. Given that tremor-dominant and non-tremor Parkinson's disease patients have different dopaminergic phenotypes, we hypothesized that effects of dopaminergic medication on reinforcement learning differ between tremor-dominant and non-tremor patients. Forty-three tremor-dominant and 20 non-tremor patients with Parkinson's disease were recruited to be tested both OFF and ON dopaminergic medication (200/50 mg levodopa-benserazide), while 22 age-matched control subjects were recruited to be tested twice OFF medication. Participants performed a reinforcement learning task designed to dissociate effects on learning rate from effects on motivational choice (i.e. the tendency to 'Go/NoGo' in the face of reward/threat of punishment). In non-tremor patients, dopaminergic medication improved reward-based choice, replicating previous studies. In contrast, in tremor-dominant patients, dopaminergic medication improved learning from punishment. Formal modelling showed divergent computational effects of dopaminergic medication as a function of Parkinson's disease motor phenotype, with a modulation of motivational choice bias and learning rate in non-tremor and tremor patients, respectively. This finding establishes a novel cognitive/motivational difference between tremor and non-tremor Parkinson's disease patients, and highlights the importance of considering motor phenotype in future work.
Collapse
Affiliation(s)
- Annelies J van Nuland
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, 6500 HB Nijmegen, The Netherlands
| | - Rick C Helmich
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, 6500 HB Nijmegen, The Netherlands
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Department of Neurology, 6500 HB Nijmegen, The Netherlands
| | - Michiel F Dirkx
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Department of Neurology, 6500 HB Nijmegen, The Netherlands
| | - Heidemarie Zach
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, 6500 HB Nijmegen, The Netherlands
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Department of Neurology, 6500 HB Nijmegen, The Netherlands
- Department of Neurology, Medical University Vienna, Vienna, Austria
| | - Ivan Toni
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, 6500 HB Nijmegen, The Netherlands
| | - Roshan Cools
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, 6500 HB Nijmegen, The Netherlands
- Radboud University Medical Center, Donders Institute for Brain, Cognition and Behaviour, Department of Psychiatry, Nijmegen, The Netherlands
| | - Hanneke E M den Ouden
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, 6500 HB Nijmegen, The Netherlands
| |
Collapse
|
26
|
Betts MJ, Richter A, de Boer L, Tegelbeckers J, Perosa V, Baumann V, Chowdhury R, Dolan RJ, Seidenbecher C, Schott BH, Düzel E, Guitart-Masip M, Krauel K. Learning in anticipation of reward and punishment: perspectives across the human lifespan. Neurobiol Aging 2020; 96:49-57. [DOI: 10.1016/j.neurobiolaging.2020.08.011] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2020] [Revised: 07/13/2020] [Accepted: 08/19/2020] [Indexed: 12/13/2022]
|
27
|
Using pharmacological manipulations to study the role of dopamine in human reward functioning: A review of studies in healthy adults. Neurosci Biobehav Rev 2020; 120:123-158. [PMID: 33202256 DOI: 10.1016/j.neubiorev.2020.11.004] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2020] [Revised: 10/30/2020] [Accepted: 11/01/2020] [Indexed: 01/08/2023]
Abstract
Dopamine (DA) plays a key role in reward processing and is implicated in psychological disorders such as depression, substance use, and schizophrenia. The role of DA in reward processing is an area of highly active research. One approach to this question is drug challenge studies with drugs known to alter DA function. These studies provide good experimental control and can be performed in parallel in laboratory animals and humans. This review aimed to summarize results of studies using pharmacological manipulations of DA in healthy adults. 'Reward' is a complex process, so we separated 'phases' of reward, including anticipation, evaluation of cost and benefits of upcoming reward, execution of actions to obtain reward, pleasure in response to receiving a reward, and reward learning. Results indicated that i) DAergic drugs have different effects on different phases of reward; ii) the relationship between DA and reward functioning appears unlikely to be linear; iii) our ability to detect the effects of DAergic drugs varies depending on whether subjective, behavioral, imaging measures are used.
Collapse
|
28
|
Crawley D, Zhang L, Jones EJH, Ahmad J, Oakley B, San José Cáceres A, Charman T, Buitelaar JK, Murphy DGM, Chatham C, den Ouden H, Loth E. Modeling flexible behavior in childhood to adulthood shows age-dependent learning mechanisms and less optimal learning in autism in each age group. PLoS Biol 2020; 18:e3000908. [PMID: 33108370 PMCID: PMC7591042 DOI: 10.1371/journal.pbio.3000908] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2019] [Accepted: 09/22/2020] [Indexed: 12/28/2022] Open
Abstract
Flexible behavior is critical for everyday decision-making and has been implicated in restricted, repetitive behaviors (RRB) in autism spectrum disorder (ASD). However, how flexible behavior changes developmentally in ASD remains largely unknown. Here, we used a developmental approach and examined flexible behavior on a probabilistic reversal learning task in 572 children, adolescents, and adults (ASD N = 321; typical development [TD] N = 251). Using computational modeling, we quantified latent variables that index mechanisms underlying perseveration and feedback sensitivity. We then assessed these variables in relation to diagnosis, developmental stage, core autism symptomatology, and associated psychiatric symptoms. Autistic individuals showed on average more perseveration and less feedback sensitivity than TD individuals, and, across cases and controls, older age groups showed more feedback sensitivity than younger age groups. Computational modeling revealed that dominant learning mechanisms underpinning flexible behavior differed across developmental stages and reduced flexible behavior in ASD was driven by less optimal learning on average within each age group. In autistic children, perseverative errors were positively related to anxiety symptoms, and in autistic adults, perseveration (indexed by both task errors and model parameter estimates) was positively related to RRB. These findings provide novel insights into reduced flexible behavior in relation to clinical symptoms in ASD.
Collapse
Affiliation(s)
- Daisy Crawley
- Department of Forensic and Neurodevelopmental Sciences, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
| | - Lei Zhang
- Institute of Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
- F. Hoffmann La Roche, Innovation Center Basel, Basel, Switzerland
| | - Emily J. H. Jones
- Centre for Brain and Cognitive Development, Birkbeck, University of London, London, United Kingdom
| | - Jumana Ahmad
- Department of Forensic and Neurodevelopmental Sciences, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
- Department of Psychology, Social Work and Counselling, University of Greenwich, London, United Kingdom
| | - Bethany Oakley
- Department of Forensic and Neurodevelopmental Sciences, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
| | - Antonia San José Cáceres
- Department of Forensic and Neurodevelopmental Sciences, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
- Instituto de Investigación Sanitaria Gregorio Marañón, Departamento de Psiquiatría del Niño y del Adolescente, Hospital General Universitario Gregorio Marañón, Madrid, Spain
| | - Tony Charman
- Department of Psychology, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
- South London and Maudsley NHS Foundation Trust (SLaM), London, United Kingdom
| | - Jan K. Buitelaar
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, Nijmegen, the Netherlands
- Department of Cognitive Neuroscience, Radboud University Nijmegen Medical Center, Nijmegen, the Netherlands
- Karakter Child and Adolescent Psychiatry University Centre, Nijmegen, the Netherlands
| | - Declan G. M. Murphy
- Department of Forensic and Neurodevelopmental Sciences, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
- South London and Maudsley NHS Foundation Trust (SLaM), London, United Kingdom
- Sackler Institute for Translational Neurodevelopment, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
| | | | - Hanneke den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, Nijmegen, the Netherlands
| | - Eva Loth
- Department of Forensic and Neurodevelopmental Sciences, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
- Sackler Institute for Translational Neurodevelopment, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom
| | | |
Collapse
|
29
|
Raab HA, Hartley CA. Adolescents exhibit reduced Pavlovian biases on instrumental learning. Sci Rep 2020; 10:15770. [PMID: 32978451 PMCID: PMC7519144 DOI: 10.1038/s41598-020-72628-w] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2020] [Accepted: 09/03/2020] [Indexed: 02/07/2023] Open
Abstract
Multiple learning systems allow individuals to flexibly respond to opportunities and challenges present in the environment. An evolutionarily conserved "Pavlovian" learning mechanism couples valence and action, promoting a tendency to approach cues associated with reward and to inhibit action in the face of anticipated punishment. Although this default response system may be adaptive, these hard-wired reactions can hinder the ability to learn flexible "instrumental" actions in pursuit of a goal. Such constraints on behavioral flexibility have been studied extensively in adults. However, the extent to which these valence-specific response tendencies bias instrumental learning across development remains poorly characterized. Here, we show that while Pavlovian response biases constrain flexible action learning in children and adults, these biases are attenuated in adolescents. This adolescent-specific reduction in Pavlovian bias may promote unbiased exploration of approach and avoidance responses, facilitating the discovery of rewarding behavior in the many novel contexts that adolescents encounter.
Collapse
Affiliation(s)
- Hillary A Raab
- Department of Psychology, New York University, New York, NY, USA
| | - Catherine A Hartley
- Department of Psychology, New York University, New York, NY, USA.
- Center for Neural Science, New York University, New York, NY, USA.
| |
Collapse
|
30
|
Tanimoto S, Kondo M, Morita K, Yoshida E, Matsuzaki M. Non-action Learning: Saving Action-Associated Cost Serves as a Covert Reward. Front Behav Neurosci 2020; 14:141. [PMID: 33100979 PMCID: PMC7498735 DOI: 10.3389/fnbeh.2020.00141] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2020] [Accepted: 07/22/2020] [Indexed: 01/20/2023] Open
Abstract
“To do or not to do” is a fundamental decision that has to be made in daily life. Behaviors related to multiple “to do” choice tasks have long been explained by reinforcement learning, and “to do or not to do” tasks such as the go/no-go task have also been recently discussed within the framework of reinforcement learning. In this learning framework, alternative actions and/or the non-action to take are determined by evaluating explicitly given (overt) reward and punishment. However, we assume that there are real life cases in which an action/non-action is repeated, even though there is no obvious reward or punishment, because implicitly given outcomes such as saving physical energy and regret (we refer to this as “covert reward”) can affect the decision-making. In the current task, mice chose to pull a lever or not according to two tone cues assigned with different water reward probabilities (70% and 30% in condition 1, and 30% and 10% in condition 2). As the mice learned, the probability that they would choose to pull the lever decreased (<0.25) in trials with a 30% reward probability cue (30% cue) in condition 1, and in trials with a 10% cue in condition 2, but increased (>0.8) in trials with a 70% cue in condition 1 and a 30% cue in condition 2, even though a non-pull was followed by neither an overt reward nor avoidance of overt punishment in any trial. This behavioral tendency was not well explained by a combination of commonly used Q-learning models, which take only the action choice with an overt reward outcome into account. Instead, we found that the non-action preference of the mice was best explained by Q-learning models, which regarded the non-action as the other choice, and updated non-action values with a covert reward. We propose that “doing nothing” can be actively chosen as an alternative to “doing something,” and that a covert reward could serve as a reinforcer of “doing nothing.”
Collapse
Affiliation(s)
- Sai Tanimoto
- Department of Physiology, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Masashi Kondo
- Department of Physiology, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Kenji Morita
- Physical and Health Education, Graduate School of Education, The University of Tokyo, Tokyo, Japan.,International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo Institutes for Advanced Study, Tokyo, Japan
| | - Eriko Yoshida
- Department of Physiology, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Masanori Matsuzaki
- Department of Physiology, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.,International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo Institutes for Advanced Study, Tokyo, Japan.,Brain Functional Dynamics Collaboration Laboratory, RIKEN Center for Brain Science, Saitama, Japan
| |
Collapse
|
31
|
Zhang L, Lengersdorff L, Mikus N, Gläscher J, Lamm C. Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices. Soc Cogn Affect Neurosci 2020; 15:695-707. [PMID: 32608484 PMCID: PMC7393303 DOI: 10.1093/scan/nsaa089] [Citation(s) in RCA: 55] [Impact Index Per Article: 13.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2019] [Revised: 06/03/2020] [Accepted: 06/15/2020] [Indexed: 12/29/2022] Open
Abstract
The recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla-Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.
Collapse
Affiliation(s)
- Lei Zhang
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna 1010, Austria
- Social, Cognitive and Affective Neuroscience Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna 1010, Austria
| | - Lukas Lengersdorff
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna 1010, Austria
- Social, Cognitive and Affective Neuroscience Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna 1010, Austria
| | - Nace Mikus
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna 1010, Austria
| | - Jan Gläscher
- Institute of Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg 20246, Germany
| | - Claus Lamm
- Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna 1010, Austria
- Social, Cognitive and Affective Neuroscience Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna 1010, Austria
- Vienna Cognitive Science Hub, University of Vienna, Vienna 1010, Austria
| |
Collapse
|
32
|
Luettgau L, Tempelmann C, Kaiser LF, Jocham G. Decisions bias future choices by modifying hippocampal associative memories. Nat Commun 2020; 11:3318. [PMID: 32620879 PMCID: PMC7335207 DOI: 10.1038/s41467-020-17192-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2019] [Accepted: 06/15/2020] [Indexed: 12/13/2022] Open
Abstract
Decision-making is guided by memories of option values. However, retrieving items from memory renders them malleable. Here, we show that merely retrieving values from memory and making a choice between options is sufficient both to induce changes to stimulus-reward associations in the hippocampus and to bias future decision-making. After allowing participants to make repeated choices between reward-conditioned stimuli, in the absence of any outcome, we observe that participants prefer stimuli they have previously chosen, and neglect previously unchosen stimuli, over otherwise identical-valued options. Using functional brain imaging, we show that decisions induce changes to hippocampal representations of stimulus-outcome associations. These changes are correlated with future decision biases. Our results indicate that choice-induced preference changes are partially driven by choice-induced modification of memory representations and suggest that merely making a choice - even without experiencing any outcomes - induces associative plasticity.
Collapse
Affiliation(s)
- Lennart Luettgau
- Biological Psychology of Decision Making, Institute of Experimental Psychology, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.
- Center for Behavioral Brain Sciences, Otto-von-Guericke University, Magdeburg, Germany.
| | - Claus Tempelmann
- Department of Neurology, Otto-von-Guericke University, Magdeburg, Germany
| | - Luca Franziska Kaiser
- Biological Psychology of Decision Making, Institute of Experimental Psychology, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
- Center for Behavioral Brain Sciences, Otto-von-Guericke University, Magdeburg, Germany
| | - Gerhard Jocham
- Biological Psychology of Decision Making, Institute of Experimental Psychology, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
- Center for Behavioral Brain Sciences, Otto-von-Guericke University, Magdeburg, Germany
| |
Collapse
|
33
|
Stimulation of the vagus nerve reduces learning in a go/no-go reinforcement learning task. Eur Neuropsychopharmacol 2020; 35:17-29. [PMID: 32404279 DOI: 10.1016/j.euroneuro.2020.03.023] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/22/2019] [Revised: 02/06/2020] [Accepted: 03/27/2020] [Indexed: 02/06/2023]
Abstract
When facing decisions to approach rewards or to avoid punishments, we often figuratively go with our gut, and the impact of metabolic states such as hunger on motivation are well documented. However, whether and how vagal feedback signals from the gut influence instrumental actions is unknown. Here, we investigated the effect of non-invasive transcutaneous auricular vagus nerve stimulation (taVNS) vs. sham (randomized cross-over design) on approach and avoidance behavior using an established go/no-go reinforcement learning paradigm in 39 healthy human participants (23 female) after an overnight fast. First, mixed-effects logistic regression analysis of choice accuracy showed that taVNS acutely impaired decision-making, p = .041. Computational reinforcement learning models identified the cause of this as a reduction in the learning rate through taVNS (∆α = -0.092, pboot = .002), particularly after punishment (∆αPun = -0.081, pboot = .012 vs. ∆αRew =-0.031, pboot = .22). However, taVNS had no effect on go biases, Pavlovian response biases or response time. Hence, taVNS appeared to influence learning rather than action execution. These results highlight a novel role of vagal afferent input in modulating reinforcement learning by tuning the learning rate according to homeostatic needs.
Collapse
|
34
|
Soutschek A, Kozak R, de Martinis N, Howe W, Burke CJ, Fehr E, Jetter A, Tobler PN. Activation of D1 receptors affects human reactivity and flexibility to valued cues. Neuropsychopharmacology 2020; 45:780-785. [PMID: 31962344 PMCID: PMC7075935 DOI: 10.1038/s41386-020-0617-z] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/13/2019] [Revised: 12/02/2019] [Accepted: 01/09/2020] [Indexed: 12/13/2022]
Abstract
Reward-predicting cues motivate goal-directed behavior, but in unstable environments humans must also be able to flexibly update cue-reward associations. While the capacity of reward cues to trigger motivation ('reactivity') as well as flexibility in cue-reward associations have been linked to the neurotransmitter dopamine in humans, the specific contribution of the dopamine D1 receptor family to these behaviors remained elusive. To fill this gap, we conducted a randomized, placebo-controlled, double-blind pharmacological study testing the impact of three different doses of a novel D1 agonist (relative to placebo) on reactivity to reward-predicting cues (Pavlovian-to-instrumental transfer) and flexibility of cue-outcome associations (reversal learning). We observed that the impact of the D1 agonist crucially depended on baseline working memory functioning, which has been identified as a proxy for baseline dopamine synthesis capacity. Specifically, increasing D1 receptor stimulation strengthened Pavlovian-to-instrumental transfer in individuals with high baseline working memory capacity. In contrast, higher doses of the D1 agonist improved reversal learning only in individuals with low baseline working memory functioning. Our findings suggest a crucial and baseline-dependent role of D1 receptor activation in controlling both cue reactivity and the flexibility of cue-reward associations.
Collapse
Affiliation(s)
- Alexander Soutschek
- Department of Psychology, Ludwig Maximilian University Munich, Munich, Germany. .,Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland.
| | - Rouba Kozak
- 0000 0004 0447 7762grid.419849.9Takeda Pharmaceuticals International, Cambridge, MA USA
| | | | - William Howe
- 0000 0001 0694 4940grid.438526.eSchool of Neuroscience, Virginia Tech, Blacksburg, VA USA
| | - Christopher J. Burke
- 0000 0004 1937 0650grid.7400.3Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
| | - Ernst Fehr
- 0000 0004 1937 0650grid.7400.3Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland ,Neuroscience Center Zurich, University of Zurich, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| | - Alexander Jetter
- Department of Clinical Pharmacology and Toxicology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Philippe N. Tobler
- 0000 0004 1937 0650grid.7400.3Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland ,Neuroscience Center Zurich, University of Zurich, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| |
Collapse
|
35
|
Kube J, Wiencke K, Hahn S, Villringer A, Neumann J. Enhanced Go and NoGo Learning in Individuals With Obesity. Front Behav Neurosci 2020; 14:15. [PMID: 32116595 PMCID: PMC7033453 DOI: 10.3389/fnbeh.2020.00015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2019] [Accepted: 01/22/2020] [Indexed: 11/13/2022] Open
Abstract
Overeating in individuals with obesity is hypothesized to be partly caused by automatic action tendencies to food cues that have the potential to override goal-directed dietary restriction. Individuals with obesity are often characterized by alterations in the processing of such rewarding food, but also of non-food stimuli, and previous research has suggested a stronger impact on the execution of goal-directed actions in obesity. Here, we investigated whether Pavlovian cues can also corrupt the learning of new approach or withdrawal behavior in individuals with obesity. We employed a probabilistic Pavlovian-instrumental learning paradigm in which participants (29 normal-weight and 29 obese) learned to actively respond (Go learning) or withhold a response (NoGo learning) in order to gain monetary rewards or avoid losses. Participants were better at learning active approach responses (Go) in the light of anticipated rewards and at learning to withhold a response (NoGo) in the light of imminent punishments. Importantly, there was no evidence for a stronger corruption of instrumental learning in individuals with obesity. Instead, they showed better learning across conditions than normal-weight participants. Using a computational reinforcement learning model, we additionally found an increased learning rate in individuals with obesity. Previous studies have mostly reported a lower reinforcement learning performance in individuals with obesity. Our results contradict this and suggest that their performance is not universally impaired: Instead, while previous studies found reduced stimulus-value learning, individuals with obesity may show better action-value learning. Our findings highlight the need for a broader investigation of behavioral adaptation in obesity across different task designs and types of reinforcement learning.
Collapse
Affiliation(s)
- Jana Kube
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Leipzig University Medical Center, IFB Adiposity Diseases, Leipzig, Germany.,Faculty 5-Business, Law and Social Sciences, Brandenburg University of Technology Cottbus-Senftenberg, Cottbus, Germany
| | - Kathleen Wiencke
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Leipzig University Medical Center, IFB Adiposity Diseases, Leipzig, Germany
| | - Sandra Hahn
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Arno Villringer
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Leipzig University Medical Center, IFB Adiposity Diseases, Leipzig, Germany.,Clinic of Cognitive Neurology, University Hospital Leipzig, Leipzig, Germany.,Berlin School of Mind and Brain, Mind and Brain Institute, Humboldt-University, Berlin, Germany
| | - Jane Neumann
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Leipzig University Medical Center, IFB Adiposity Diseases, Leipzig, Germany.,Department of Medical Engineering and Biotechnology, University of Applied Sciences, Jena, Germany
| |
Collapse
|
36
|
Froböse MI, Westbrook A, Bloemendaal M, Aarts E, Cools R. Catecholaminergic modulation of the cost of cognitive control in healthy older adults. PLoS One 2020; 15:e0229294. [PMID: 32084218 PMCID: PMC7034873 DOI: 10.1371/journal.pone.0229294] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2019] [Accepted: 02/03/2020] [Indexed: 11/19/2022] Open
Abstract
Catecholamines have long been associated with cognitive control and value-based decision-making. More recently, we have shown that catecholamines also modulate value-based decision-making about whether or not to engage in cognitive control. Yet it is unclear whether catecholamines influence these decisions by altering the subjective value of control. Thus, we tested whether tyrosine, a catecholamine precursor altered the subjective value of performing a demanding working memory task among healthy older adults (60-75 years). Contrary to our prediction, tyrosine administration did not significantly increase the subjective value of conducting an N-back task for reward, as a main effect. Instead, in line with our previous study, exploratory analyses indicated that drug effects varied as a function of participants' trait impulsivity scores. Specifically, tyrosine increased the subjective value of conducting an N-back task in low impulsive participants, while reducing its value in more impulsive participants. One implication of these findings is that the over-the-counter tyrosine supplements may be accompanied by an undermining effect on the motivation to perform demanding cognitive tasks, at least in certain older adults. Taken together, these findings indicate that catecholamines can alter cognitive control by modulating motivation (rather than just the ability) to exert cognitive control.
Collapse
Affiliation(s)
- Monja I. Froböse
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Institute of Experimental Psychology, Heinrich-Heine University, Düsseldorf, Germany
| | - Andrew Westbrook
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI, United States of America
| | - Mirjam Bloemendaal
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Esther Aarts
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Dept Psychiatry, Radboud University Medical Centre, Nijmegen, The Netherlands
| |
Collapse
|
37
|
Adams RA, Moutoussis M, Nour MM, Dahoun T, Lewis D, Illingworth B, Veronese M, Mathys C, de Boer L, Guitart-Masip M, Friston KJ, Howes OD, Roiser JP. Variability in Action Selection Relates to Striatal Dopamine 2/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models. Cereb Cortex 2020; 30:3573-3589. [PMID: 32083297 PMCID: PMC7233027 DOI: 10.1093/cercor/bhz327] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2019] [Revised: 11/18/2019] [Accepted: 12/05/2019] [Indexed: 12/17/2022] Open
Abstract
Choosing actions that result in advantageous outcomes is a fundamental function of nervous systems. All computational decision-making models contain a mechanism that controls the variability of (or confidence in) action selection, but its neural implementation is unclear-especially in humans. We investigated this mechanism using two influential decision-making frameworks: active inference (AI) and reinforcement learning (RL). In AI, the precision (inverse variance) of beliefs about policies controls action selection variability-similar to decision 'noise' parameters in RL-and is thought to be encoded by striatal dopamine signaling. We tested this hypothesis by administering a 'go/no-go' task to 75 healthy participants, and measuring striatal dopamine 2/3 receptor (D2/3R) availability in a subset (n = 25) using [11C]-(+)-PHNO positron emission tomography. In behavioral model comparison, RL performed best across the whole group but AI performed best in participants performing above chance levels. Limbic striatal D2/3R availability had linear relationships with AI policy precision (P = 0.029) as well as with RL irreducible decision 'noise' (P = 0.020), and this relationship with D2/3R availability was confirmed with a 'decision stochasticity' factor that aggregated across both models (P = 0.0006). These findings are consistent with occupancy of inhibitory striatal D2/3Rs decreasing the variability of action selection in humans.
Collapse
Affiliation(s)
- Rick A Adams
- Institute of Cognitive Neuroscience, University College London, London WC1N 3AZ, UK.,Division of Psychiatry, University College London, London W1T 7NF, UK.,Psychiatric Imaging Group, Robert Steiner MRI Unit, MRC London Institute of Medical Sciences, Hammersmith Hospital, London W12 0NN, UK.,Institute of Clinical Sciences, Faculty of Medicine, Imperial College London, Hammersmith Hospital, London W12 0NN, UK
| | - Michael Moutoussis
- Wellcome Centre for Human Neuroimaging, University College London, London WC1N 3BG, UK.,Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London WC1B 5EH, UK
| | - Matthew M Nour
- Psychiatric Imaging Group, Robert Steiner MRI Unit, MRC London Institute of Medical Sciences, Hammersmith Hospital, London W12 0NN, UK.,Institute of Clinical Sciences, Faculty of Medicine, Imperial College London, Hammersmith Hospital, London W12 0NN, UK.,Department of Psychosis Studies, Institute of Psychiatry, Psychology & Neuroscience (IoPPN), King's College London, London SE5 8AF, UK
| | - Tarik Dahoun
- Psychiatric Imaging Group, Robert Steiner MRI Unit, MRC London Institute of Medical Sciences, Hammersmith Hospital, London W12 0NN, UK.,Institute of Clinical Sciences, Faculty of Medicine, Imperial College London, Hammersmith Hospital, London W12 0NN, UK.,Department of Psychiatry, University of Oxford, Warneford Hospital, Oxford OX3 7JX, UK
| | - Declan Lewis
- Institute of Cognitive Neuroscience, University College London, London WC1N 3AZ, UK
| | - Benjamin Illingworth
- Institute of Cognitive Neuroscience, University College London, London WC1N 3AZ, UK
| | - Mattia Veronese
- Centre for Neuroimaging Sciences, Institute of Psychiatry, Psychology & Neuroscience (IoPPN), King's College London, London SE5 8AF, UK
| | - Christoph Mathys
- Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London WC1B 5EH, UK.,Scuola Internazionale Superiore di Studi Avanzati (SISSA), 34136 Trieste, Italy.,Translational Neuromodeling Unit (TNU), Institute for Biomedical Engineering, University of Zurich and ETH Zurich, 8032 Zurich, Switzerland
| | - Lieke de Boer
- Aging Research Center, Karolinska Institute, 171 65 Stockholm, Sweden
| | - Marc Guitart-Masip
- Max Planck-UCL Centre for Computational Psychiatry and Ageing Research, London WC1B 5EH, UK.,Aging Research Center, Karolinska Institute, 171 65 Stockholm, Sweden
| | - Karl J Friston
- Wellcome Centre for Human Neuroimaging, University College London, London WC1N 3BG, UK
| | - Oliver D Howes
- Psychiatric Imaging Group, Robert Steiner MRI Unit, MRC London Institute of Medical Sciences, Hammersmith Hospital, London W12 0NN, UK.,Institute of Clinical Sciences, Faculty of Medicine, Imperial College London, Hammersmith Hospital, London W12 0NN, UK.,Department of Psychosis Studies, Institute of Psychiatry, Psychology & Neuroscience (IoPPN), King's College London, London SE5 8AF, UK
| | - Jonathan P Roiser
- Institute of Cognitive Neuroscience, University College London, London WC1N 3AZ, UK
| |
Collapse
|
38
|
Perosa V, de Boer L, Ziegler G, Apostolova I, Buchert R, Metzger C, Amthauer H, Guitart-Masip M, Düzel E, Betts MJ. The Role of the Striatum in Learning to Orthogonalize Action and Valence: A Combined PET and 7 T MRI Aging Study. Cereb Cortex 2020; 30:3340-3351. [PMID: 31897476 DOI: 10.1093/cercor/bhz313] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022] Open
Abstract
Pavlovian biases influence instrumental learning by coupling reward seeking with action invigoration and punishment avoidance with action suppression. Using a probabilistic go/no-go task designed to orthogonalize action (go/no-go) and valence (reward/punishment), recent studies have shown that the interaction between the two is dependent on the striatum and its key neuromodulator dopamine. Using this task, we sought to identify how structural and neuromodulatory age-related differences in the striatum may influence Pavlovian biases and instrumental learning in 25 young and 31 older adults. Computational modeling revealed a significant age-related reduction in reward and punishment sensitivity and marked (albeit not significant) reduction in learning rate and lapse rate (irreducible noise). Voxel-based morphometry analysis using 7 Tesla MRI images showed that individual differences in learning rate in older adults were related to the volume of the caudate nucleus. In contrast, dopamine synthesis capacity in the dorsal striatum, assessed using [18F]-DOPA positron emission tomography in 22 of these older adults, was not associated with learning performance and did not moderate the relationship between caudate volume and learning rate. This multiparametric approach suggests that age-related differences in striatal volume may influence learning proficiency in old age.
Collapse
Affiliation(s)
- Valentina Perosa
- Institute of Cognitive Neurology and Dementia Research, Otto-von-Guericke-University Magdeburg, Leipzigerstr. 44, 39120, Magdeburg, Germany.,German Center for Neurodegenerative Diseases (DZNE), Leipzigerstr. 44 39120, Magdeburg, Germany.,Department of Neurology, Otto-von-Guericke University, Leipzigerstr. 44, 39120, Magdeburg, Germany
| | - Lieke de Boer
- Ageing Research Centre, Karolinska Institute, SE-11330 Stockholm, Sweden
| | - Gabriel Ziegler
- Institute of Cognitive Neurology and Dementia Research, Otto-von-Guericke-University Magdeburg, Leipzigerstr. 44, 39120, Magdeburg, Germany.,German Center for Neurodegenerative Diseases (DZNE), Leipzigerstr. 44 39120, Magdeburg, Germany
| | - Ivayla Apostolova
- Department of Radiology and Nuclear Medicine, University Hospital Hamburg-Eppendorf, Germany
| | - Ralph Buchert
- Department of Radiology and Nuclear Medicine, University Hospital Hamburg-Eppendorf, Germany.,Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London WC1B 5EH, UK
| | - Coraline Metzger
- Institute of Cognitive Neurology and Dementia Research, Otto-von-Guericke-University Magdeburg, Leipzigerstr. 44, 39120, Magdeburg, Germany.,German Center for Neurodegenerative Diseases (DZNE), Leipzigerstr. 44 39120, Magdeburg, Germany
| | - Holger Amthauer
- Department of Nuclear Medicine, Charité-Universitätsmedizin Berlin, Augustenburger Platz 1, 13353 Berlin, Germany
| | - Marc Guitart-Masip
- Ageing Research Centre, Karolinska Institute, SE-11330 Stockholm, Sweden.,Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London WC1B 5EH, UK
| | - Emrah Düzel
- Institute of Cognitive Neurology and Dementia Research, Otto-von-Guericke-University Magdeburg, Leipzigerstr. 44, 39120, Magdeburg, Germany.,German Center for Neurodegenerative Diseases (DZNE), Leipzigerstr. 44 39120, Magdeburg, Germany.,Institute of Cognitive Neuroscience, University College London, 17 Queen Square, London, UK
| | - Matthew J Betts
- Institute of Cognitive Neurology and Dementia Research, Otto-von-Guericke-University Magdeburg, Leipzigerstr. 44, 39120, Magdeburg, Germany.,German Center for Neurodegenerative Diseases (DZNE), Leipzigerstr. 44 39120, Magdeburg, Germany
| |
Collapse
|
39
|
Baas M, Boot N, van Gaal S, de Dreu CK, Cools R. Methylphenidate does not affect convergent and divergent creative processes in healthy adults. Neuroimage 2020; 205:116279. [DOI: 10.1016/j.neuroimage.2019.116279] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2019] [Revised: 10/09/2019] [Accepted: 10/13/2019] [Indexed: 01/24/2023] Open
|
40
|
Controllability governs the balance between Pavlovian and instrumental action selection. Nat Commun 2019; 10:5826. [PMID: 31862876 PMCID: PMC6925275 DOI: 10.1038/s41467-019-13737-7] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2019] [Accepted: 11/25/2019] [Indexed: 11/17/2022] Open
Abstract
A Pavlovian bias to approach reward-predictive cues and avoid punishment-predictive cues can conflict with instrumentally-optimal actions. Here, we propose that the brain arbitrates between Pavlovian and instrumental control by inferring which is a better predictor of reward. The instrumental predictor is more flexible; it can learn values that depend on both stimuli and actions, whereas the Pavlovian predictor learns values that depend only on stimuli. The arbitration theory predicts that the Pavlovian predictor will be favored when rewards are relatively uncontrollable, because the additional flexibility of the instrumental predictor is not useful. Consistent with this hypothesis, we find that the Pavlovian approach bias is stronger under low control compared to high control contexts. Pavlovian and instrumentally driven actions often conflict when determining the best outcome. Here, the authors present an arbitration theory supported by human behavioral data where Pavlovian predictors drive action selection in an uncontrollable environment, while more flexible instrumental prediction dominates under conditions of high controllability.
Collapse
|
41
|
Cook JL, Swart JC, Froböse MI, Diaconescu AO, Geurts DEM, den Ouden HEM, Cools R. Catecholaminergic modulation of meta-learning. eLife 2019; 8:e51439. [PMID: 31850844 PMCID: PMC6974360 DOI: 10.7554/elife.51439] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2019] [Accepted: 12/18/2019] [Indexed: 01/03/2023] Open
Abstract
The remarkable expedience of human learning is thought to be underpinned by meta-learning, whereby slow accumulative learning processes are rapidly adjusted to the current learning environment. To date, the neurobiological implementation of meta-learning remains unclear. A burgeoning literature argues for an important role for the catecholamines dopamine and noradrenaline in meta-learning. Here, we tested the hypothesis that enhancing catecholamine function modulates the ability to optimise a meta-learning parameter (learning rate) as a function of environmental volatility. 102 participants completed a task which required learning in stable phases, where the probability of reinforcement was constant, and volatile phases, where probabilities changed every 10-30 trials. The catecholamine transporter blocker methylphenidate enhanced participants' ability to adapt learning rate: Under methylphenidate, compared with placebo, participants exhibited higher learning rates in volatile relative to stable phases. Furthermore, this effect was significant only with respect to direct learning based on the participants' own experience, there was no significant effect on inferred-value learning where stimulus values had to be inferred. These data demonstrate a causal link between catecholaminergic modulation and the adjustment of the meta-learning parameter learning rate.
Collapse
Affiliation(s)
- Jennifer L Cook
- School of PsychologyUniversity of BirminghamBirminghamUnited Kingdom
| | - Jennifer C Swart
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
| | - Monja I Froböse
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
| | - Andreea O Diaconescu
- Translational Neuromodeling Unit, Institute for Biomedical EngineeringUniversity of Zurich and ETH ZurichZurichSwitzerland
- Department of PsychiatryUniversity of BaselBaselSwitzerland
- Krembil Centre for Neuroinformatics,CAMHUniversity of TorontoTorontoCanada
| | - Dirk EM Geurts
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
- Department of PsychiatryRadboud University Medical CentreNijmegenNetherlands
| | - Hanneke EM den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
- Department of PsychiatryRadboud University Medical CentreNijmegenNetherlands
| |
Collapse
|
42
|
Cools R. Chemistry of the Adaptive Mind: Lessons from Dopamine. Neuron 2019; 104:113-131. [DOI: 10.1016/j.neuron.2019.09.035] [Citation(s) in RCA: 54] [Impact Index Per Article: 10.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2019] [Revised: 09/19/2019] [Accepted: 09/20/2019] [Indexed: 12/21/2022]
|
43
|
Sidarus N, Palminteri S, Chambon V. Cost-benefit trade-offs in decision-making and learning. PLoS Comput Biol 2019; 15:e1007326. [PMID: 31490934 PMCID: PMC6750595 DOI: 10.1371/journal.pcbi.1007326] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2018] [Revised: 09/18/2019] [Accepted: 08/08/2019] [Indexed: 11/18/2022] Open
Abstract
Value-based decision-making involves trading off the cost associated with an action against its expected reward. Research has shown that both physical and mental effort constitute such subjective costs, biasing choices away from effortful actions, and discounting the value of obtained rewards. Facing conflicts between competing action alternatives is considered aversive, as recruiting cognitive control to overcome conflict is effortful. Moreover, engaging control to proactively suppress irrelevant information that could conflict with task-relevant information would presumably also be cognitively costly. Yet, it remains unclear whether the cognitive control demands involved in preventing and resolving conflict also constitute costs in value-based decisions. The present study investigated this question by embedding irrelevant distractors (flanker arrows) within a reversal-learning task, with intermixed free and instructed trials. Results showed that participants learned to adapt their free choices to maximize rewards, but were nevertheless biased to follow the suggestions of irrelevant distractors. Thus, the perceived cost of investing cognitive control to suppress an external suggestion could sometimes trump internal value representations. By adapting computational models of reinforcement learning, we assessed the influence of conflict at both the decision and learning stages. Modelling the decision showed that free choices were more biased when participants were less sure about which action was more rewarding. This supports the hypothesis that the costs linked to conflict management were traded off against expected rewards. During the learning phase, we found that learning rates were reduced in instructed, relative to free, choices. Learning rates were further reduced by conflict between an instruction and subjective action values, whereas learning was not robustly influenced by conflict between one’s actions and external distractors. Our results show that the subjective cognitive control costs linked to conflict factor into value-based decision-making, and highlight that different types of conflict may have different effects on learning about action outcomes. Value-based decision-making involves trading off the cost associated with an action–such as physical or mental effort–against its expected reward. Although facing conflicts between competing action alternatives is considered aversive and effortful, it remains unclear whether conflict also constitutes a cost in value-based decisions. We tested this hypothesis by combining a classic conflict (flanker) task with a reinforcement-learning task. Results showed that participants learned to maximise their earnings, but were nevertheless biased to follow irrelevant suggestions. Computational model-based analyses showed a greater choice bias with more uncertainty about the best action to make, supporting the hypothesis that the costs linked to conflict management were traded off against expected rewards. We additionally found that learning rates were reduced when following instructions, relative to when choosing freely what to do. Learning was further reduced by conflict between instructions and subjective action values. In short, we found that the subjective cognitive control costs linked to conflict factor into value-based decision-making, and that different types of conflict may have different effects on learning about action outcomes.
Collapse
Affiliation(s)
- Nura Sidarus
- Institut Jean Nicod, Département d’Études Cognitives, École Normale Supérieure, EHESS, CNRS, PSL University, Paris, France
- Laboratoire de Neurosciences Cognitives Computationnelles, Département d’Études Cognitives, École Normale Supérieure, INSERM, PSL University, Paris, France
- Department of Psychology, Royal Holloway University of London, Surrey, United Kingdom
- * E-mail:
| | - Stefano Palminteri
- Laboratoire de Neurosciences Cognitives Computationnelles, Département d’Études Cognitives, École Normale Supérieure, INSERM, PSL University, Paris, France
| | - Valérian Chambon
- Institut Jean Nicod, Département d’Études Cognitives, École Normale Supérieure, EHESS, CNRS, PSL University, Paris, France
| |
Collapse
|
44
|
Xia Y, Gurkina A, Bach DR. Pavlovian-to-instrumental transfer after human threat conditioning. ACTA ACUST UNITED AC 2019; 26:167-175. [PMID: 31004041 PMCID: PMC6478249 DOI: 10.1101/lm.049338.119] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2019] [Accepted: 04/02/2019] [Indexed: 11/25/2022]
Abstract
Threat conditioning is a common associative learning model with translational relevance. How threat-conditioned cues impact on formally unrelated instrumental behavior in humans is not well known. Such an effect is known as Pavlovian-to-instrumental transfer (PIT). While PIT with aversive primary Pavlovian reinforcers is established in nonhuman animals, this is less clear in humans, where secondary reinforcers or instructed instrumental responses are most often investigated. We modified an existing human PIT procedure to include primary reinforcers. Participants first learned to obtain (or avoid losing) appetitive instrumental reinforcement (chocolate) by appropriate approach or avoidance actions. They either had to act (Go) or to withhold an action (NoGo), and in the Go condition either to approach a reward target to collect it or to withdraw from the reward target to avoid losing it. Then they learned to associate screen color (CS) with aversive Pavlovian reinforcement (electric shock US). In the transfer phase, we conducted the instrumental task during the presence of Pavlovian CS. In a first experiment, we show that the aversive Pavlovian CS+, compared to CS−, increased response rate in Go-Withdraw trials, i.e., induce conditioned facilitation of avoidance responses. This finding was confirmed in a second and independent experiment with an increased number of Go-Withdraw trials. Notably, we observed no appreciable conditioned suppression of approach responses. Effect size to distinguish CS+/CS− in Go-Withdraw trials was d = 0.42 in the confirmation sample. This would require n = 37 participants to demonstrate threat learning with 80% power. Thus, the effect size is on a practically useful scale although smaller than for model-based analysis of autonomic measures. In summary, our results indicate conditioned facilitation of formally unrelated instrumental avoidance behavior in humans and provide a novel behavioral threat learning measure that requires only key presses.
Collapse
Affiliation(s)
- Yanfang Xia
- Computational Psychiatry Research, Department of Psychiatry, Psychotherapy, and Psychosomatics, Psychiatric Hospital, University of Zurich, 8032 Zurich, Switzerland.,Neuroscience Center Zurich; University of Zurich, 8057 Zurich, Switzerland
| | - Angelina Gurkina
- Computational Psychiatry Research, Department of Psychiatry, Psychotherapy, and Psychosomatics, Psychiatric Hospital, University of Zurich, 8032 Zurich, Switzerland.,Neuroscience Center Zurich; University of Zurich, 8057 Zurich, Switzerland
| | - Dominik R Bach
- Computational Psychiatry Research, Department of Psychiatry, Psychotherapy, and Psychosomatics, Psychiatric Hospital, University of Zurich, 8032 Zurich, Switzerland.,Wellcome Trust Centre for Human Neuroimaging and Max Planck/UCL Centre for Computational Psychiatry and Ageing Research, University College London, London WC1 3BG, United Kingdom
| |
Collapse
|
45
|
Ly V, Wang KS, Bhanji J, Delgado MR. A Reward-Based Framework of Perceived Control. Front Neurosci 2019; 13:65. [PMID: 30809112 PMCID: PMC6379460 DOI: 10.3389/fnins.2019.00065] [Citation(s) in RCA: 36] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2018] [Accepted: 01/22/2019] [Indexed: 12/20/2022] Open
Abstract
Perceived control can be broadly defined as the belief in one's ability to exert control over situations or events. It has long been known that perceived control is a major contributor toward mental and physical health as well as a strong predictor of achievements in life. However, one issue that limits a mechanistic understanding of perceived control is the heterogeneity of how the term is defined in models in psychology and neuroscience, and used in experimental settings across a wide spectrum of studies. Here, we propose a framework for studying perceived control by integrating the ideas from traditionally separate work on perceived control. Specifically, we discuss key properties of perceived control from a reward-based framework, including choice opportunity, instrumental contingency, and success/reward rate. We argue that these separate reward-related processes are integral to fostering an enhanced perception of control and influencing an individual's behavior and well-being. We draw on select studies to elucidate how these reward-related elements are implicated separately and collectively in the investigation of perceived control. We highlight the role of dopamine within corticostriatal pathways shared by reward-related processes and perceived control. Finally, through the lens of this reward-based framework of perceived control, we consider the implications of perceived control in clinical deficits and how these insights could help us better understand psychopathology and treatment options.
Collapse
Affiliation(s)
- Verena Ly
- Institute of Psychology, Leiden University, Leiden, Netherlands
- Leiden Institute for Brain and Cognition, Leiden, Netherlands
- Department of Psychology, Rutgers University – Newark, Newark, NJ, United States
| | - Kainan S. Wang
- Behavioral and Neural Sciences Graduate Program, Rutgers University – Newark, Newark, NJ, United States
| | - Jamil Bhanji
- Department of Psychology, Rutgers University – Newark, Newark, NJ, United States
| | - Mauricio R. Delgado
- Department of Psychology, Rutgers University – Newark, Newark, NJ, United States
| |
Collapse
|
46
|
Cools R, Froböse M, Aarts E, Hofmans L. Dopamine and the motivation of cognitive control. HANDBOOK OF CLINICAL NEUROLOGY 2019; 163:123-143. [DOI: 10.1016/b978-0-12-804281-6.00007-0] [Citation(s) in RCA: 32] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
|
47
|
Dorsal striatal dopamine D1 receptor availability predicts an instrumental bias in action learning. Proc Natl Acad Sci U S A 2018; 116:261-270. [PMID: 30563856 PMCID: PMC6320523 DOI: 10.1073/pnas.1816704116] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/01/2023] Open
Abstract
The brain’s dopaminergic pathways are crucially important for adaptive behavior. They are thought to enable us to approach rewards and stay away from punishments. During learning, dopaminergic reward prediction errors are thought to reinforce previously rewarded actions, so they become easier to repeat. This dopaminergic activity could lead to a systematic bias by which rewarded actions are more readily learned than rewarded inactions. We present two findings. First, dopamine receptors in cortex, dorsal striatum, and nucleus accumbens provide distinct sources of variance in the human brain. Second, the boost in an individual’s learning rate from previously rewarded actions is dependent on the dopamine receptor density in dorsal striatum, a central structure in the dopaminergic circuit. Learning to act to obtain reward and inhibit to avoid punishment is easier compared with learning the opposite contingencies. This coupling of action and valence is often thought of as a Pavlovian bias, although recent research has shown it may also emerge through instrumental mechanisms. We measured this learning bias with a rewarded go/no-go task in 60 adults of different ages. Using computational modeling, we characterized the bias as being instrumental. To assess the role of endogenous dopamine (DA) in the expression of this bias, we quantified DA D1 receptor availability using positron emission tomography (PET) with the radioligand [11C]SCH23390. Using principal-component analysis on the binding potentials in a number of cortical and striatal regions of interest, we demonstrated that cortical, dorsal striatal, and ventral striatal areas provide independent sources of variance in DA D1 receptor availability. Interindividual variation in the dorsal striatal component was related to the strength of the instrumental bias during learning. These data suggest at least three anatomical sources of variance in DA D1 receptor availability separable using PET in humans, and we provide evidence that human dorsal striatal DA D1 receptors are involved in the modulation of instrumental learning biases.
Collapse
|
48
|
Emotionally Aversive Cues Suppress Neural Systems Underlying Optimal Learning in Socially Anxious Individuals. J Neurosci 2018; 39:1445-1456. [PMID: 30559152 DOI: 10.1523/jneurosci.1394-18.2018] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2018] [Revised: 11/19/2018] [Accepted: 12/11/2018] [Indexed: 11/21/2022] Open
Abstract
Learning and decision-making are modulated by socio-emotional processing and such modulation is implicated in clinically relevant personality traits of social anxiety. The present study elucidates the computational and neural mechanisms by which emotionally aversive cues disrupt learning in socially anxious human individuals. Healthy volunteers with low or high trait social anxiety performed a reversal learning task requiring learning actions in response to angry or happy face cues. Choice data were best captured by a computational model in which learning rate was adjusted according to the history of surprises. High trait socially anxious individuals used a less-dynamic strategy for adjusting their learning rate in trials started with angry face cues and unlike the low social anxiety group, their dorsal anterior cingulate cortex (dACC) activity did not covary with the learning rate. Our results demonstrate that trait social anxiety is accompanied by disruption of optimal learning and dACC activity in threatening situations.SIGNIFICANCE STATEMENT Social anxiety is known to influence a broad range of cognitive functions. This study tests whether and how social anxiety affects human value-based learning as a function of uncertainty in the learning environment. The findings indicate that, in a threatening context evoked by an angry face, socially anxious individuals fail to benefit from a stable learning environment with highly predictable stimulus-response-outcome associations. Under those circumstances, socially anxious individuals failed to use their dorsal anterior cingulate cortex, a region known to adjust learning rate to environmental uncertainty. These findings open the way to modify neurobiological mechanisms of maladaptive learning in anxiety and depressive disorders.
Collapse
|
49
|
Moutoussis M, Bullmore ET, Goodyer IM, Fonagy P, Jones PB, Dolan RJ, Dayan P. Change, stability, and instability in the Pavlovian guidance of behaviour from adolescence to young adulthood. PLoS Comput Biol 2018; 14:e1006679. [PMID: 30596638 PMCID: PMC6329529 DOI: 10.1371/journal.pcbi.1006679] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2018] [Revised: 01/11/2019] [Accepted: 11/27/2018] [Indexed: 12/24/2022] Open
Abstract
Pavlovian influences are important in guiding decision-making across health and psychopathology. There is an increasing interest in using concise computational tasks to parametrise such influences in large populations, and especially to track their evolution during development and changes in mental health. However, the developmental course of Pavlovian influences is uncertain, a problem compounded by the unclear psychometric properties of the relevant measurements. We assessed Pavlovian influences in a longitudinal sample using a well characterised and widely used Go-NoGo task. We hypothesized that the strength of Pavlovian influences and other 'psychomarkers' guiding decision-making would behave like traits. As reliance on Pavlovian influence is not as profitable as precise instrumental decision-making in this Go-NoGo task, we expected this influence to decrease with higher IQ and age. Additionally, we hypothesized it would correlate with expressions of psychopathology. We found that Pavlovian effects had weak temporal stability, while model-fit was more stable. In terms of external validity, Pavlovian effects decreased with increasing IQ and experience within the task, in line with normative expectations. However, Pavlovian effects were poorly correlated with age or psychopathology. Thus, although this computational construct did correlate with important aspects of development, it does not meet conventional requirements for tracking individual development. We suggest measures that might improve psychometric properties of task-derived Pavlovian measures for future studies.
Collapse
Affiliation(s)
- Michael Moutoussis
- Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom
- Max Planck Centre for Computational Psychiatry and Ageing, University College London, United Kingdom
| | - Edward T. Bullmore
- Department of Psychiatry, University of Cambridge, Cambridge, United Kingdom
- Cambridgeshire and Peterborough National Health Service Foundation Trust, Cambridge, United Kingdom
- Medical Research Council/Wellcome Trust Behavioural and Clinical Neuroscience Institute, University of Cambridge, Cambridge, United Kingdom
- ImmunoPsychiatry, GlaxoSmithKline Research and Development, Stevenage, United Kingdom
| | - Ian M. Goodyer
- Department of Psychiatry, University of Cambridge, Cambridge, United Kingdom
- Cambridgeshire and Peterborough National Health Service Foundation Trust, Cambridge, United Kingdom
| | - Peter Fonagy
- Research Department of Clinical, Educational and Health Psychology, University College London, London, United Kingdom
| | - Peter B. Jones
- Department of Psychiatry, University of Cambridge, Cambridge, United Kingdom
- Cambridgeshire and Peterborough National Health Service Foundation Trust, Cambridge, United Kingdom
| | - Raymond J. Dolan
- Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom
- Max Planck Centre for Computational Psychiatry and Ageing, University College London, United Kingdom
| | - Peter Dayan
- Max Planck Institute of Biological Cybernetics, Tübingen, Germany
| | | |
Collapse
|
50
|
Swart JC, Frank MJ, Määttä JI, Jensen O, Cools R, den Ouden HEM. Frontal network dynamics reflect neurocomputational mechanisms for reducing maladaptive biases in motivated action. PLoS Biol 2018; 16:e2005979. [PMID: 30335745 PMCID: PMC6207318 DOI: 10.1371/journal.pbio.2005979] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2018] [Revised: 10/30/2018] [Accepted: 07/18/2018] [Indexed: 01/04/2023] Open
Abstract
Motivation exerts control over behavior by eliciting Pavlovian responses, which can either match or conflict with instrumental action. We can overcome maladaptive motivational influences putatively through frontal cognitive control. However, the neurocomputational mechanisms subserving this control are unclear; does control entail up-regulating instrumental systems, down-regulating Pavlovian systems, or both? We combined electroencephalography (EEG) recordings with a motivational Go/NoGo learning task (N = 34), in which multiple Go options enabled us to disentangle selective action learning from nonselective Pavlovian responses. Midfrontal theta-band (4 Hz–8 Hz) activity covaried with the level of Pavlovian conflict and was associated with reduced Pavlovian biases rather than reduced instrumental learning biases. Motor and lateral prefrontal regions synchronized to the midfrontal cortex, and these network dynamics predicted the reduction of Pavlovian biases over and above local, midfrontal theta activity. This work links midfrontal processing to detecting Pavlovian conflict and highlights the importance of network processing in reducing the impact of maladaptive, Pavlovian biases. The anticipation of reward and punishment are key drivers of behavior: we tend to take action for rewards while holding back in the face of punishment. This motivational bias might have an overall evolutionary advantage but can also work against us in specific situations. Here, we first asked whether this motivational bias relies on innate, automatic action tendencies or whether this bias might actually itself be learned. Secondly, we studied which brain processes reduce the impact of these motivational drives when they become dysfunctional. By comparing the actions of human participants to the predictions of several mathematical models, we showed that the motivational bias in action relies partly on automatic tendencies and partly on asymmetric learning from experience. We then observed that activity over the midfrontal cortex specifically increased as a function of how dysfunctional the automatic tendencies were. Additionally, this midfrontal cortex activity was functionally connected to the motor and lateral frontal cortices, which play a role in activating and inhibiting behavior. By incorporating this connectivity into the mathematical models, we showed that stronger midfrontal connectivity predicted reduced impact of dysfunctional automatic tendencies on behavior. We propose that the midfrontal cortex detects dysfunctional action tendencies and implements cognitive control by signaling across the network.
Collapse
Affiliation(s)
- Jennifer C. Swart
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- * E-mail: (JCS); (HEMdO)
| | - Michael J. Frank
- Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, Rhode Island, United States of America
- Brown Institute for Brain Sciences, Brown University, Providence, Rhode Island, United States of America
| | - Jessica I. Määttä
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, The Netherlands
| | - Ole Jensen
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- School of Psychology, University of Birmingham, Birmingham, United Kingdom
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Department of Psychiatry, Radboud University Medical Centre, Nijmegen, The Netherlands
| | - Hanneke E. M. den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- * E-mail: (JCS); (HEMdO)
| |
Collapse
|