1
|
Grabenhorst F, Báez-Mendoza R. Dynamic coding and sequential integration of multiple reward attributes by primate amygdala neurons. Nat Commun 2025; 16:3119. [PMID: 40169589 PMCID: PMC11962072 DOI: 10.1038/s41467-025-58270-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2024] [Accepted: 03/12/2025] [Indexed: 04/03/2025] Open
Abstract
The value of visual stimuli guides learning, decision-making, and motivation. Although stimulus values often depend on multiple attributes, how neurons extract and integrate distinct value components from separate cues remains unclear. Here we recorded the activity of amygdala neurons while two male monkeys viewed sequential cues indicating the probability and magnitude of expected rewards. Amygdala neurons frequently signaled reward probability in an abstract, stimulus-independent code that generalized across cue formats. While some probability-coding neurons were insensitive to magnitude information, signaling 'pure' probability rather than value, many neurons showed biphasic responses that signaled probability and magnitude in a dynamic (temporally-patterned) and flexible (reversible) value code. Specific amygdala neurons integrated these reward attributes into risk signals that quantified the variance of expected rewards, distinct from value. Population codes were accurate, mutually transferable between value components, and expressed differently across amygdala nuclei. Our findings identify amygdala neurons as a substrate for the sequential integration of multiple reward attributes into value and risk.
Collapse
Affiliation(s)
- Fabian Grabenhorst
- Department of Experimental Psychology, University of Oxford, Oxford, UK.
| | | |
Collapse
|
2
|
Calangiu I, Kollmorgen S, Reppas J, Mante V. Prospective and retrospective representations of saccadic movements in primate prefrontal cortex. Cell Rep 2025; 44:115289. [PMID: 39946232 DOI: 10.1016/j.celrep.2025.115289] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2022] [Revised: 11/24/2024] [Accepted: 01/17/2025] [Indexed: 02/28/2025] Open
Abstract
The dorso-lateral prefrontal cortex (dlPFC) contributes to flexible, goal-directed behaviors. However, a coherent picture of dlPFC function is lacking, as its activity is often studied only in relation to a few events within a fully learned behavioral task. Here we obtain a comprehensive description of dlPFC activity across different task epochs, saccade types, tasks, and learning stages. We consistently observe the strongest modulation of neural activity in relation to a retrospective representation of the most recent saccade. Prospective, planning-like activity is limited to task-related, delayed saccades directly eligible for a reward. The link between prospective and retrospective representations is highly structured, potentially reflecting a hard-wired feature of saccade responses. Only prospective representations are modulated by the recent behavioral history, but neither representation is modulated by day-to-day behavioral improvements. The dlPFC thus combines tightly linked flexible and rigid representations with a dominant contribution from retrospective signals maintaining the memory of past actions.
Collapse
Affiliation(s)
- Ioana Calangiu
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland.
| | - Sepp Kollmorgen
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland
| | - John Reppas
- Department of Neurobiology and Howard Hughes Medical Institute, Stanford University, Stanford, CA, USA
| | - Valerio Mante
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland.
| |
Collapse
|
3
|
Zhang Y, Van Dam NT, Ai H, Xu P. Frontostriatal connectivity dynamically modulates the adaptation to environmental volatility. Neuroimage 2025; 307:121027. [PMID: 39818279 DOI: 10.1016/j.neuroimage.2025.121027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2024] [Revised: 01/13/2025] [Accepted: 01/13/2025] [Indexed: 01/18/2025] Open
Abstract
Humans adjust their learning strategies in changing environments by estimating the volatility of the reinforcement conditions. Here, we examine how volatility affects learning and the underlying functional brain organizations using a probabilistic reward reversal learning task. We found that the order of states was critically important; participants adjusted learning rate going from volatile to stable, but not from stable to volatile environments. Subjective volatility of the environment was encoded in the striatal reward system and its dynamic connections with the prefrontal control system. Flexibility, which captures the dynamic changes of network modularity in the brain, was higher in the environmental transition from volatile to stable than from stable to volatile. These findings suggest that behavioral adaptations and dynamic brain organizations in transitions between stable and volatile environments are asymmetric, providing critical insights into the way that people adapt to changing environments.
Collapse
Affiliation(s)
- Yuxuan Zhang
- Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education (BNU), Faculty of Psychology, Beijing Normal University, Beijing, China
| | - Nicholas T Van Dam
- Melbourne School of Psychological Sciences, The University of Melbourne, Melbourne, Australia
| | - Hui Ai
- Institute of Applied Psychology, Tianjin University, Tianjin, China; Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China.
| | - Pengfei Xu
- Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education (BNU), Faculty of Psychology, Beijing Normal University, Beijing, China; Center for Neuroimaging, Shenzhen Institute of Neuroscience, Shenzhen, China.
| |
Collapse
|
4
|
Sarrazin V, Overman MJ, Mezossy-Dona L, Browning M, O'Shea J. Prefrontal cortex stimulation normalizes deficient adaptive learning from outcome contingencies in low mood. Transl Psychiatry 2024; 14:487. [PMID: 39695073 DOI: 10.1038/s41398-024-03204-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/19/2024] [Revised: 12/02/2024] [Accepted: 12/10/2024] [Indexed: 12/20/2024] Open
Abstract
Depression and anxiety are associated with deficits in adjusting learning behaviour to changing outcome contingencies. This is likely to drive and maintain symptoms, for instance, by perpetuating negative biases or a sense of uncontrollability. Normalising such deficits in adaptive learning might therefore be a novel treatment target for affective disorders. The aim of this experimental medicine study was to test whether prefrontal cortex transcranial direct current stimulation (tDCS) could normalise these aberrant learning processes in depressed mood. To test proof-of-concept, we combined tDCS with a decision-making paradigm that manipulates the volatility of reward and punishment associations. 85 participants with low mood received tDCS during (or before) the task. In two sessions participants received real or sham tDCS in counter-balanced order. Compared to healthy controls (n = 40), individuals with low mood showed significantly impaired adjustment of learning rates to the volatility of loss outcomes. Prefrontal tDCS applied during task performance normalised this deficit, by increasing the adjustment of loss learning rates. As predicted, prefrontal tDCS before task performance (control) had no effect. Thus, the effect was cognitive-state dependent. Our study shows, for the first time, that a candidate depression treatment, prefrontal tDCS, when paired with a task, can reverse deficits in adaptive learning from outcome contingencies in low mood. Thus, combining neurostimulation with a concurrent cognitive manipulation is a potential novel strategy to enhance the effect of tDCS in depression treatment.
Collapse
Affiliation(s)
- Verena Sarrazin
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, Oxford, United Kingdom.
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, Oxford, United Kingdom.
- Oxford Centre for Human Brain Activity (OHBA), University of Oxford, OX3 7JX, Oxford, United Kingdom.
| | - Margot Juliëtte Overman
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, Oxford, United Kingdom
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, Oxford, United Kingdom
- Oxford Centre for Human Brain Activity (OHBA), University of Oxford, OX3 7JX, Oxford, United Kingdom
| | - Luca Mezossy-Dona
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, Oxford, United Kingdom
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, Oxford, United Kingdom
- Oxford Centre for Human Brain Activity (OHBA), University of Oxford, OX3 7JX, Oxford, United Kingdom
| | - Michael Browning
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, Oxford, United Kingdom
| | - Jacinta O'Shea
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, Oxford, United Kingdom
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, Oxford, United Kingdom
- Oxford Centre for Human Brain Activity (OHBA), University of Oxford, OX3 7JX, Oxford, United Kingdom
| |
Collapse
|
5
|
Song M, Shin EJ, Seo H, Soltani A, Steinmetz NA, Lee D, Jung MW, Paik SB. Hierarchical gradients of multiple timescales in the mammalian forebrain. Proc Natl Acad Sci U S A 2024; 121:e2415695121. [PMID: 39671181 DOI: 10.1073/pnas.2415695121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2024] [Accepted: 11/14/2024] [Indexed: 12/14/2024] Open
Abstract
Many anatomical and physiological features of cortical circuits, ranging from the biophysical properties of synapses to the connectivity patterns among different neuron types, exhibit consistent variation along the hierarchical axis from sensory to association areas. Notably, the temporal correlation of neural activity at rest, known as the intrinsic timescale, increases systematically along this hierarchy in both primates and rodents, analogous to the increasing scale and complexity of spatial receptive fields. However, how the timescales for task-related activity vary across brain regions and whether their hierarchical organization appears consistently across different mammalian species remain unexplored. Here, we show that both the intrinsic timescale and those of task-related activity follow a similar hierarchical gradient in the cortices of monkeys, rats, and mice. We also found that these timescales covary similarly in both the cortex and basal ganglia, whereas the timescales of thalamic activity are shorter than cortical timescales and do not conform to the hierarchical order predicted by their cortical projections. These results suggest that the hierarchical gradient of cortical timescales might represent a universal feature of intracortical circuits in the mammalian brain.
Collapse
Affiliation(s)
- Min Song
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
| | - Eun Ju Shin
- Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon 34141, Republic of Korea
| | - Hyojung Seo
- Department of Psychiatry, Yale University, New Haven, CT 06520
| | - Alireza Soltani
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH 03755
| | - Nicholas A Steinmetz
- Department of Neurobiology and Biophysics, University of Washington, Seattle, WA 98195
| | - Daeyeol Lee
- Zanvyl Krieger Mind/Brain Institute, Johns Hopkins University, Baltimore, MD 21218
- Kavli Discovery Neuroscience Institute, Johns Hopkins University, Baltimore, MD 21218
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD 21218
- Department of Neuroscience, Johns Hopkins University, Baltimore, MD 21218
| | - Min Whan Jung
- Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon 34141, Republic of Korea
| | - Se-Bum Paik
- Department of Brain and Cognitive Sciences, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
| |
Collapse
|
6
|
Suthaharan P, Thompson SL, Rossi-Goldthorpe RA, Rudebeck PH, Walton ME, Chakraborty S, Noonan MP, Costa VD, Murray EA, Mathys CD, Groman SM, Mitchell AS, Taylor JR, Corlett PR, Chang SWC. Lesions to the mediodorsal thalamus, but not orbitofrontal cortex, enhance volatility beliefs linked to paranoia. Cell Rep 2024; 43:114355. [PMID: 38870010 PMCID: PMC11231991 DOI: 10.1016/j.celrep.2024.114355] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2023] [Revised: 04/13/2024] [Accepted: 05/29/2024] [Indexed: 06/15/2024] Open
Abstract
Beliefs-attitudes toward some state of the environment-guide action selection and should be robust to variability but sensitive to meaningful change. Beliefs about volatility (expectation of change) are associated with paranoia in humans, but the brain regions responsible for volatility beliefs remain unknown. The orbitofrontal cortex (OFC) is central to adaptive behavior, whereas the magnocellular mediodorsal thalamus (MDmc) is essential for arbitrating between perceptions and action policies. We assessed belief updating in a three-choice probabilistic reversal learning task following excitotoxic lesions of the MDmc (n = 3) or OFC (n = 3) and compared performance with that of unoperated monkeys (n = 14). Computational analyses indicated a double dissociation: MDmc, but not OFC, lesions were associated with erratic switching behavior and heightened volatility belief (as in paranoia in humans), whereas OFC, but not MDmc, lesions were associated with increased lose-stay behavior and reward learning rates. Given the consilience across species and models, these results have implications for understanding paranoia.
Collapse
Affiliation(s)
- Praveen Suthaharan
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT, USA; Kavli Institute for Neuroscience, Yale University, New Haven, CT, USA; Department of Psychiatry, Yale University, New Haven, CT, USA
| | | | - Rosa A Rossi-Goldthorpe
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT, USA; Department of Psychiatry, Yale University, New Haven, CT, USA
| | | | - Mark E Walton
- Department of Experimental Psychology, Oxford University, Oxford, UK
| | - Subhojit Chakraborty
- Department of Experimental Psychology, Oxford University, Oxford, UK; NIHR Biomedical Research Centre, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London EC1V 9EL, UK
| | - Maryann P Noonan
- Department of Experimental Psychology, Oxford University, Oxford, UK; Department of Psychology, University of York, York, UK
| | - Vincent D Costa
- Division of Neuroscience, Oregon National Primate Research Center, Oregon Health and Science University, Beaverton, OR, USA
| | | | - Christoph D Mathys
- Interacting Minds Centre, Aarhus University, Aarhus, Denmark; Translational Neuromodeling Unit (TNU), Institute for Biomedical Engineering, University of Zurich and ETH Zurich, Zurich, Switzerland
| | - Stephanie M Groman
- Department of Psychiatry, Yale University, New Haven, CT, USA; Department of Neuroscience, University of Minnesota Medical School, Minneapolis, MN, USA; Department of Anesthesia and Critical Care, University of Chicago, Chicago, IL, USA
| | - Anna S Mitchell
- Department of Experimental Psychology, Oxford University, Oxford, UK; School of Psychology, Speech, and Hearing, University of Canterbury, Christchurch, New Zealand
| | - Jane R Taylor
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT, USA; Department of Psychiatry, Yale University, New Haven, CT, USA; Department of Psychology, Yale University, New Haven, CT, USA; Wu Tsai Institute, Yale University, New Haven, CT, USA; Department of Neuroscience, Yale University, New Haven, CT, USA
| | - Philip R Corlett
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT, USA; Kavli Institute for Neuroscience, Yale University, New Haven, CT, USA; Department of Psychiatry, Yale University, New Haven, CT, USA; Department of Psychology, Yale University, New Haven, CT, USA; Wu Tsai Institute, Yale University, New Haven, CT, USA.
| | - Steve W C Chang
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT, USA; Kavli Institute for Neuroscience, Yale University, New Haven, CT, USA; Department of Psychology, Yale University, New Haven, CT, USA; Wu Tsai Institute, Yale University, New Haven, CT, USA; Department of Neuroscience, Yale University, New Haven, CT, USA.
| |
Collapse
|
7
|
Pardina‐Torner H, De Paepe AE, Garcia‐Gorro C, Rodriguez‐Dechicha N, Vaquer I, Calopa M, Ruiz‐Idiago J, Mareca C, de Diego‐Balaguer R, Camara E. Disentangling the neurobiological bases of temporal impulsivity in Huntington's disease. Brain Behav 2024; 14:e3335. [PMID: 38450912 PMCID: PMC10918610 DOI: 10.1002/brb3.3335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Revised: 10/10/2023] [Accepted: 11/08/2023] [Indexed: 03/08/2024] Open
Abstract
BACKGROUND Despite its impact on daily life, impulsivity in Huntington's disease (HD) is understudied as a neuropsychiatric symptom. Our aim is to characterize temporal impulsivity in HD and to disentangle the white matter correlate associated with impulsivity. METHODS Forty-seven HD individuals and 36 healthy controls were scanned and evaluated for temporal impulsivity using a delay-discounting (DD) task and complementary Sensitivity to Punishment and Sensitivity to Reward Questionnaire. Diffusion tensor imaging was employed to characterize the structural connectivity of three limbic tracts: the uncinate fasciculus (UF), the accumbofrontal tract (NAcc-OFC), and the dorsolateral prefrontal cortex connectig the caudate nucleus (DLPFC-cn). Multiple linear regression analyses were applied to analyze the relationship between impulsive behavior and white matter microstructural integrity. RESULTS Our results revealed altered structural connectivity in the DLPC-cn, the NAcc-OFC and the UF in HD individuals. At the same time, the variability in structural connectivity of these tracts was associated with the individual differences in temporal impulsivity. Specifically, increased structural connectivity in the right NAcc-OFC and reduced connectivity in the left UF were associated with higher temporal impulsivity scores. CONCLUSIONS The present findings highlight the importance of investigating the spectrum of temporal impulsivity in HD. As, while less prevalent than other psychiatric features, this symptom is still reported to significantly impact the quality of life of patients and caregivers. This study provides evidence that individual differences observed in temporal impulsivity may be explained by variability in limbic frontostriatal tracts, while shedding light on the role of sensitivity to reward in modulating impulsive behavior through the selection of immediate rewards.
Collapse
Affiliation(s)
- Helena Pardina‐Torner
- Cognition and Brain Plasticity UnitBellvitge Biomedical Research Institute (IDIBELL)BarcelonaSpain
| | - Audrey E. De Paepe
- Cognition and Brain Plasticity UnitBellvitge Biomedical Research Institute (IDIBELL)BarcelonaSpain
| | - Clara Garcia‐Gorro
- Cognition and Brain Plasticity UnitBellvitge Biomedical Research Institute (IDIBELL)BarcelonaSpain
| | - Nadia Rodriguez‐Dechicha
- Hestia Duran i ReynalsHospital Duran i Reynals, Hospitalet de LlobregatBarcelonaSpain
- Departament de Psicologia Clínica i de la SalutUniversitat Autònoma de BarcelonaBarcelonaSpain
| | - Irene Vaquer
- Hestia Duran i ReynalsHospital Duran i Reynals, Hospitalet de LlobregatBarcelonaSpain
- Departament de Psicologia Clínica i de la SalutUniversitat Autònoma de BarcelonaBarcelonaSpain
| | - Matilde Calopa
- Movement Disorders Unit, Neurology ServiceHospital Universitari de BellvitgeBarcelonaSpain
- ICREA (Catalan Institute for Research and Advanced Studies)BarcelonaSpain
| | - Jesus Ruiz‐Idiago
- Department of Psychiatry and Forensic MedicineUniversitat Autònoma de BarcelonaBarcelonaSpain
- Hospital Mare de Deu de la MercèBarcelonaSpain
| | - Celia Mareca
- Department of Psychiatry and Forensic MedicineUniversitat Autònoma de BarcelonaBarcelonaSpain
- Hospital Mare de Deu de la MercèBarcelonaSpain
| | - Ruth de Diego‐Balaguer
- Cognition and Brain Plasticity UnitBellvitge Biomedical Research Institute (IDIBELL)BarcelonaSpain
- Department of Cognition, Development and Education PsychologyUniversitat de BarcelonaBarcelonaSpain
- Institute of NeurosciencesUniversitat de BarcelonaBarcelonaSpain
- ICREA (Catalan Institute for Research and Advanced Studies)BarcelonaSpain
| | - Estela Camara
- Cognition and Brain Plasticity UnitBellvitge Biomedical Research Institute (IDIBELL)BarcelonaSpain
| |
Collapse
|
8
|
Danskin BP, Hattori R, Zhang YE, Babic Z, Aoi M, Komiyama T. Exponential history integration with diverse temporal scales in retrosplenial cortex supports hyperbolic behavior. SCIENCE ADVANCES 2023; 9:eadj4897. [PMID: 38019904 PMCID: PMC10686558 DOI: 10.1126/sciadv.adj4897] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 10/27/2023] [Indexed: 12/01/2023]
Abstract
Animals use past experience to guide future choices. The integration of experiences typically follows a hyperbolic, rather than exponential, decay pattern with a heavy tail for distant history. Hyperbolic integration affords sensitivity to both recent environmental dynamics and long-term trends. However, it is unknown how the brain implements hyperbolic integration. We found that mouse behavior in a foraging task showed hyperbolic decay of past experience, but the activity of cortical neurons showed exponential decay. We resolved this apparent mismatch by observing that cortical neurons encode history information with heterogeneous exponential time constants that vary across neurons. A model combining these diverse timescales recreated the heavy-tailed, hyperbolic history integration observed in behavior. In particular, the time constants of retrosplenial cortex (RSC) neurons best matched the behavior, and optogenetic inactivation of RSC uniquely reduced behavioral history dependence. These results indicate that behavior-relevant history information is maintained across multiple timescales in parallel and that RSC is a critical reservoir of information guiding decision-making.
Collapse
Affiliation(s)
- Bethanny P. Danskin
- Department of Neurobiology, University of California San Diego, La Jolla, CA, USA
- Center for Neural Circuits and Behavior, University of California San Diego, La Jolla, CA, USA
- Department of Neurosciences, University of California San Diego, La Jolla, CA, USA
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
| | - Ryoma Hattori
- Department of Neurobiology, University of California San Diego, La Jolla, CA, USA
- Center for Neural Circuits and Behavior, University of California San Diego, La Jolla, CA, USA
- Department of Neurosciences, University of California San Diego, La Jolla, CA, USA
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
| | - Yu E. Zhang
- Department of Neurobiology, University of California San Diego, La Jolla, CA, USA
- Center for Neural Circuits and Behavior, University of California San Diego, La Jolla, CA, USA
- Department of Neurosciences, University of California San Diego, La Jolla, CA, USA
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
| | - Zeljana Babic
- Department of Neurobiology, University of California San Diego, La Jolla, CA, USA
- Center for Neural Circuits and Behavior, University of California San Diego, La Jolla, CA, USA
- Department of Neurosciences, University of California San Diego, La Jolla, CA, USA
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
| | - Mikio Aoi
- Department of Neurobiology, University of California San Diego, La Jolla, CA, USA
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
| | - Takaki Komiyama
- Department of Neurobiology, University of California San Diego, La Jolla, CA, USA
- Center for Neural Circuits and Behavior, University of California San Diego, La Jolla, CA, USA
- Department of Neurosciences, University of California San Diego, La Jolla, CA, USA
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
| |
Collapse
|
9
|
Nejati V, Famininejad Z, Rad JA. Neural symphony of risky decision making in children with ADHD: Insights from transcranial alternating current stimulation and cognitive modeling. Neurophysiol Clin 2023; 53:102898. [PMID: 37659136 DOI: 10.1016/j.neucli.2023.102898] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Revised: 07/29/2023] [Accepted: 07/31/2023] [Indexed: 09/04/2023] Open
Abstract
BACKGROUND The ventromedial prefrontal cortex (vmPFC) and dorsolateral prefrontal cortex (dlPFC) are key brain regions involved in risky decision making, affected in individuals with attention deficit hyperactivity disorder (ADHD). This study aims to examine how entrainment of these areas impacts the process and outcome of risky decision making in children with ADHD. METHODS Eighteen children with ADHD performed the balloon analogue risk-taking task (BART) during five different sessions of tACS (1.5 mA, 6 Hz), separated by one-week intervals, via (1) two channels with synchronized stimulation over the left dlPFC and right vmPFC, (2) the same electrode placement with anti-phase stimulation, (3) stimulation over the left dlPFC only, (4) stimulation over right vmPFC only, and (5) sham stimulation. Four-parameter and constant-sensitivity models were used to model the data. RESULTS The study showed that synchronized stimulation was associated with a reduction in positive prior belief, risk propensity, and deterministic selection. Desynchronized stimulation was associated with accelerated learning from initial selections. Isolated stimulation of the dlPFC leads to riskier decision enhanced learning updates and risk propensity, whereas isolated stimulation of the vmPFC facilitated faster learning and increased probabilistic selection. CONCLUSION The results highlight the important roles of the dlPFC and vmPFC and their communication in decision making, showcasing their impact on various aspects of the decision-making process. The findings provide valuable insights into the complex interplay between cognitive and emotional factors in shaping our choices.
Collapse
Affiliation(s)
- Vahid Nejati
- Department of Psychology, Shahid Beheshti University, Tehran, Iran.
| | | | - Jamal Amani Rad
- Department of Cognitive Modeling, Institute of Cognitive and Brain Sciences, Shahid Beheshti University, Tehran, Iran
| |
Collapse
|
10
|
Woo JH, Aguirre CG, Bari BA, Tsutsui KI, Grabenhorst F, Cohen JY, Schultz W, Izquierdo A, Soltani A. Mechanisms of adjustments to different types of uncertainty in the reward environment across mice and monkeys. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2023; 23:600-619. [PMID: 36823249 PMCID: PMC10444905 DOI: 10.3758/s13415-022-01059-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 12/22/2022] [Indexed: 02/25/2023]
Abstract
Despite being unpredictable and uncertain, reward environments often exhibit certain regularities, and animals navigating these environments try to detect and utilize such regularities to adapt their behavior. However, successful learning requires that animals also adjust to uncertainty associated with those regularities. Here, we analyzed choice data from two comparable dynamic foraging tasks in mice and monkeys to investigate mechanisms underlying adjustments to different types of uncertainty. In these tasks, animals selected between two choice options that delivered reward probabilistically, while baseline reward probabilities changed after a variable number (block) of trials without any cues to the animals. To measure adjustments in behavior, we applied multiple metrics based on information theory that quantify consistency in behavior, and fit choice data using reinforcement learning models. We found that in both species, learning and choice were affected by uncertainty about reward outcomes (in terms of determining the better option) and by expectation about when the environment may change. However, these effects were mediated through different mechanisms. First, more uncertainty about the better option resulted in slower learning and forgetting in mice, whereas it had no significant effect in monkeys. Second, expectation of block switches accompanied slower learning, faster forgetting, and increased stochasticity in choice in mice, whereas it only reduced learning rates in monkeys. Overall, while demonstrating the usefulness of metrics based on information theory in examining adaptive behavior, our study provides evidence for multiple types of adjustments in learning and choice behavior according to uncertainty in the reward environment.
Collapse
Affiliation(s)
- Jae Hyung Woo
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Claudia G Aguirre
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Bilal A Bari
- Department of Psychiatry, Massachusetts General Hospital, Boston, MA, USA
| | - Ken-Ichiro Tsutsui
- Department of Physiology, Development & Neuroscience, University of Cambridge, Cambridge, UK
- Laboratory of Systems Neuroscience, Tohoku University Graduate School of Life Sciences, Sendai, Japan
| | - Fabian Grabenhorst
- Department of Physiology, Development & Neuroscience, University of Cambridge, Cambridge, UK
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
- Allen Institute for Neural Dynamics, Seattle, WA, USA
| | - Wolfram Schultz
- Department of Physiology, Development & Neuroscience, University of Cambridge, Cambridge, UK
| | - Alicia Izquierdo
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
- The Brain Research Institute, University of California, Los Angeles, Los Angeles, CA, USA
| | - Alireza Soltani
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| |
Collapse
|
11
|
Overman MJ, Sarrazin V, Browning M, O'Shea J. Stimulating human prefrontal cortex increases reward learning. Neuroimage 2023; 271:120029. [PMID: 36925089 PMCID: PMC11968408 DOI: 10.1016/j.neuroimage.2023.120029] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2022] [Revised: 03/07/2023] [Accepted: 03/13/2023] [Indexed: 03/17/2023] Open
Abstract
Work in computational psychiatry suggests that mood disorders may stem from aberrant reinforcement learning processes. Specifically, it has been proposed that depressed individuals believe that negative events are more informative than positive events, resulting in higher learning rates from negative outcomes (Pulcu and Browning, 2019). In this proof-of-concept study, we investigated whether transcranial direct current stimulation (tDCS) applied to dorsolateral prefrontal cortex, as commonly used in depression treatment trials, might change learning rates for affective outcomes. Healthy adults completed an established reinforcement learning task (Pulcu and Browning, 2017) in which the information content of reward and loss outcomes was manipulated by varying the volatility of stimulus-outcome associations. Learning rates on the tasks were quantified using computational models. Stimulation over dorsolateral prefrontal cortex (DLPFC) but not motor cortex (M1) increased learning rates specifically for reward outcomes. The effects of prefrontal tDCS were cognitive state-dependent: tDCS applied during task performance increased learning rates for wins; tDCS applied before task performance decreased both win and loss learning rates. A replication study confirmed the key finding that tDCS to DLPFC during task performance increased learning rates specifically for rewards. Taken together, these findings demonstrate the potential of tDCS for modulating computational parameters of reinforcement learning that are relevant to mood disorders.
Collapse
Affiliation(s)
- Margot Juliëtte Overman
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, United Kingdom
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, United Kingdom
- Oxford Centre for Human Brain Activity (OHBA), University of Oxford, OX3 7JX, United Kingdom
| | - Verena Sarrazin
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, United Kingdom
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, United Kingdom
- Oxford Centre for Human Brain Activity (OHBA), University of Oxford, OX3 7JX, United Kingdom
| | - Michael Browning
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, United Kingdom
| | - Jacinta O'Shea
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, OX3 9DU, United Kingdom
- Department of Psychiatry, Warneford Hospital, University of Oxford, OX3 7JX, United Kingdom
- Oxford Centre for Human Brain Activity (OHBA), University of Oxford, OX3 7JX, United Kingdom
| |
Collapse
|
12
|
Wen T, Geddert RM, Madlon-Kay S, Egner T. Transfer of Learned Cognitive Flexibility to Novel Stimuli and Task Sets. Psychol Sci 2023; 34:435-454. [PMID: 36693129 PMCID: PMC10236430 DOI: 10.1177/09567976221141854] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2022] [Accepted: 11/03/2022] [Indexed: 01/25/2023] Open
Abstract
Adaptive behavior requires learning about the structure of one's environment to derive optimal action policies, and previous studies have documented transfer of such structural knowledge to bias choices in new environments. Here, we asked whether people could also acquire and transfer more abstract knowledge across different task environments, specifically expectations about cognitive control demands. Over three experiments, participants (Amazon Mechanical Turk workers; N = ~80 adults per group) performed a probabilistic card-sorting task in environments of either a low or high volatility of task rule changes (requiring low or high cognitive flexibility, respectively) before transitioning to a medium-volatility environment. Using reinforcement-learning modeling, we consistently found that previous exposure to high task rule volatilities led to faster adaptation to rule changes in the subsequent transfer phase. These transfers of expectations about cognitive flexibility demands were both task independent (Experiment 2) and stimulus independent (Experiment 3), thus demonstrating the formation and generalization of environmental structure knowledge to guide cognitive control.
Collapse
Affiliation(s)
- Tanya Wen
- Center for Cognitive Neuroscience, Duke
University
| | | | - Seth Madlon-Kay
- Department of Biostatistics and
Bioinformatics, Duke University School of Medicine
| | - Tobias Egner
- Center for Cognitive Neuroscience, Duke
University
- Department of Psychology and
Neuroscience, Duke University
| |
Collapse
|
13
|
Bakst L, McGuire JT. Experience-driven recalibration of learning from surprising events. Cognition 2023; 232:105343. [PMID: 36481590 PMCID: PMC9851993 DOI: 10.1016/j.cognition.2022.105343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Revised: 10/13/2022] [Accepted: 11/21/2022] [Indexed: 12/12/2022]
Abstract
Different environments favor different patterns of adaptive learning. A surprising event that in one context would accelerate belief updating might, in another context, be downweighted as a meaningless outlier. Here, we investigated whether people would spontaneously regulate the influence of surprise on learning in response to event-by-event experiential feedback. Across two experiments, we examined whether participants performing a perceptual judgment task under spatial uncertainty (n = 29, n = 63) adapted their patterns of predictive gaze according to the informativeness or uninformativeness of surprising events in their current environment. Uninstructed predictive eye movements exhibited a form of metalearning in which surprise came to modulate event-by-event learning rates in opposite directions across contexts. Participants later appropriately readjusted their patterns of adaptive learning when the statistics of the environment underwent an unsignaled reversal. Although significant adjustments occurred in both directions, performance was consistently superior in environments in which surprising events reflected meaningful change, potentially reflecting a bias towards interpreting surprise as informative and/or difficulty ignoring salient outliers. Our results provide evidence for spontaneous, context-appropriate recalibration of the role of surprise in adaptive learning.
Collapse
Affiliation(s)
- Leah Bakst
- Department of Psychological & Brain Sciences, Boston University, 64 Cummington Mall, Boston, MA 02215, USA; Center for Systems Neuroscience, Boston University, 610 Commonwealth Avenue, Boston, MA 02215, USA.
| | - Joseph T McGuire
- Department of Psychological & Brain Sciences, Boston University, 64 Cummington Mall, Boston, MA 02215, USA; Center for Systems Neuroscience, Boston University, 610 Commonwealth Avenue, Boston, MA 02215, USA.
| |
Collapse
|
14
|
Liu M, Dong W, Wu Y, Verbeke P, Verguts T, Chen Q. Modulating hierarchical learning by high-definition transcranial alternating current stimulation at theta frequency. Cereb Cortex 2022; 33:4421-4431. [PMID: 36089836 DOI: 10.1093/cercor/bhac352] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 08/09/2022] [Accepted: 08/11/2022] [Indexed: 11/12/2022] Open
Abstract
Considerable evidence highlights the dorsolateral prefrontal cortex (DLPFC) as a key region for hierarchical (i.e. multilevel) learning. In a previous electroencephalography (EEG) study, we found that the low-level prediction errors were encoded by frontal theta oscillations (4-7 Hz), centered on right DLPFC (rDLPFC). However, the causal relationship between frontal theta oscillations and hierarchical learning remains poorly understood. To investigate this question, in the current study, participants received theta (6 Hz) and sham high-definition transcranial alternating current stimulation (HD-tACS) over the rDLPFC while performing the probabilistic reversal learning task. Behaviorally, theta tACS induced a significant reduction in accuracy for the stable environment, but not for the volatile environment, relative to the sham condition. Computationally, we implemented a combination of a hierarchical Bayesian learning and a decision model. Theta tACS induced a significant increase in low-level (i.e. probability-level) learning rate and uncertainty of low-level estimation relative to sham condition. Instead, the temperature parameter of the decision model, which represents (inverse) decision noise, was not significantly altered due to theta stimulation. These results indicate that theta frequency may modulate the (low-level) learning rate. Furthermore, environmental features (e.g. its stability) may determine whether learning is optimized as a result.
Collapse
Affiliation(s)
- Meng Liu
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China.,School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
| | - Wenshan Dong
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China.,School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
| | - Yiling Wu
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China.,School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
| | - Pieter Verbeke
- Department of Experimental Psychology, Ghent University, B-9000 Ghent, Belgium
| | - Tom Verguts
- Department of Experimental Psychology, Ghent University, B-9000 Ghent, Belgium
| | - Qi Chen
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China.,School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
| |
Collapse
|
15
|
Pereira CLW, Zhou R, Pitt MA, Myung JI, Rossi PJ, Caverzasi E, Rah E, Allen IE, Mandelli ML, Meyer M, Miller ZA, Gorno Tempini ML. Probabilistic Decision-Making in Children With Dyslexia. Front Neurosci 2022; 16:782306. [PMID: 35769704 PMCID: PMC9235406 DOI: 10.3389/fnins.2022.782306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2021] [Accepted: 04/19/2022] [Indexed: 11/24/2022] Open
Abstract
Background Neurocognitive mechanisms underlying developmental dyslexia (dD) remain poorly characterized apart from phonological and/or visual processing deficits. Assuming such deficits, the process of learning complex tasks like reading requires the learner to make decisions (i.e., word pronunciation) based on uncertain information (e.g., aberrant phonological percepts)-a cognitive process known as probabilistic decision making, which has been linked to the striatum. We investigate (1) the relationship between dD and probabilistic decision-making and (2) the association between the volume of striatal structures and probabilistic decision-making in dD and typical readers. Methods Twenty four children diagnosed with dD underwent a comprehensive evaluation and MRI scanning (3T). Children with dD were compared to age-matched typical readers (n = 11) on a probabilistic, risk/reward fishing task that utilized a Bayesian cognitive model with game parameters of risk propensity (γ+) and behavioral consistency (β), as well as an overall adjusted score (average number of casts, excluding forced-fail trials). Volumes of striatal structures (caudate, putamen, and nucleus accumbens) were analyzed between groups and associated with game parameters. Results dD was associated with greater risk propensity and decreased behavioral consistency estimates compared to typical readers. Cognitive model parameters associated with timed pseudoword reading across groups. Risk propensity related to caudate volumes, particularly in the dD group. Conclusion Decision-making processes differentiate dD, associate with the caudate, and may impact learning mechanisms. This study suggests the need for further research into domain-general probabilistic decision-making in dD, neurocognitive mechanisms, and targeted interventions in dD.
Collapse
Affiliation(s)
- Christa L. Watson Pereira
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
| | - Ran Zhou
- Department of Psychology, Ohio State University, Columbus, OH, United States
| | - Mark A. Pitt
- Department of Psychology, Ohio State University, Columbus, OH, United States
| | - Jay I. Myung
- Department of Psychology, Ohio State University, Columbus, OH, United States
| | - P. Justin Rossi
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
- Department of Psychiatry, Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, United States
| | - Eduardo Caverzasi
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
- Department of Neurology, University of California, San Francisco, San Francisco, CA, United States
| | - Esther Rah
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
| | - Isabel E. Allen
- Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, CA, United States
| | - Maria Luisa Mandelli
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
| | - Marita Meyer
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
| | - Zachary A. Miller
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
| | - Maria Luisa Gorno Tempini
- Department of Neurology, UCSF Dyslexia Center, UCSF Memory and Aging Center, University of California, San Francisco, San Francisco, CA, United States
| |
Collapse
|
16
|
Grossman CD, Cohen JY. Neuromodulation and Neurophysiology on the Timescale of Learning and Decision-Making. Annu Rev Neurosci 2022; 45:317-337. [PMID: 35363533 DOI: 10.1146/annurev-neuro-092021-125059] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Nervous systems evolved to effectively navigate the dynamics of the environment to achieve their goals. One framework used to study this fundamental problem arose in the study of learning and decision-making. In this framework, the demands of effective behavior require slow dynamics-on the scale of seconds to minutes-of networks of neurons. Here, we review the phenomena and mechanisms involved. Using vignettes from a few species and areas of the nervous system, we view neuromodulators as key substrates for temporal scaling of neuronal dynamics. Expected final online publication date for the Annual Review of Neuroscience, Volume 45 is July 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Collapse
Affiliation(s)
- Cooper D Grossman
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, and Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA;
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, and Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA;
| |
Collapse
|
17
|
Miyamoto K, Setsuie R, Miyashita Y. Conversion of concept-specific decision confidence into integrative introspection in primates. Cell Rep 2022; 38:110581. [PMID: 35354028 DOI: 10.1016/j.celrep.2022.110581] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2021] [Revised: 12/21/2021] [Accepted: 03/07/2022] [Indexed: 11/26/2022] Open
Abstract
Introspection based on the integration of uncertain evidence is critical for acting upon abstract thinking and imagining future scenarios. However, it is unknown how confidence read-outs from multiple sources of different concepts are integrated, especially considering the relationships among the concepts. In this study, monkeys performed wagering based on an estimation of their performance in a preceding mnemonic decision. We found that the longer the response times for post-decision wagering, the more relieved the impairments having been caused by frontal disruption. This suggests the existence of a time-consuming compensatory metacognitive process. We found posterior inferior parietal lobe (pIPL) as its candidate, which was not coding the wagering per se (i.e., just high bet or low bet), but became more active when monkeys successfully chose the optimal bet option based on mnemonic decision performance. Thereafter, the pIPL prompts dorsal anterior cingulate cortex to carry the chosen wagering option. Our findings suggest a role for the pIPL in metacognitive concept integration.
Collapse
Affiliation(s)
- Kentaro Miyamoto
- Department of Physiology, The University of Tokyo School of Medicine, Bunkyo-ku, Tokyo 113-0033, Japan; Juntendo University, Graduate School of Medicine, Bunkyo-ku, Tokyo 113-8421, Japan; Department of Experimental Psychology, University of Oxford, Oxford, OXON OX1 3TA, UK; Laboratory for Imagination and Executive Functions, RIKEN Center for Brain Science, 2-1 Hirosawa, Wako-shi, Saitama 351-0198, Japan.
| | - Rieko Setsuie
- Department of Physiology, The University of Tokyo School of Medicine, Bunkyo-ku, Tokyo 113-0033, Japan; Juntendo University, Graduate School of Medicine, Bunkyo-ku, Tokyo 113-8421, Japan; Laboratory for Cognition Circuit Dynamics, RIKEN Center for Brain Science, Wako-shi, Saitama 351-0198, Japan; Brain Functional Dynamics Collaboration Laboratory, RIKEN Center for Brain Science, Wako-shi, Saitama 351-0198, Japan
| | - Yasushi Miyashita
- Department of Physiology, The University of Tokyo School of Medicine, Bunkyo-ku, Tokyo 113-0033, Japan; Juntendo University, Graduate School of Medicine, Bunkyo-ku, Tokyo 113-8421, Japan; Laboratory for Cognition Circuit Dynamics, RIKEN Center for Brain Science, Wako-shi, Saitama 351-0198, Japan
| |
Collapse
|
18
|
Grossman CD, Bari BA, Cohen JY. Serotonin neurons modulate learning rate through uncertainty. Curr Biol 2022; 32:586-599.e7. [PMID: 34936883 PMCID: PMC8825708 DOI: 10.1016/j.cub.2021.12.006] [Citation(s) in RCA: 61] [Impact Index Per Article: 20.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Revised: 10/11/2021] [Accepted: 12/03/2021] [Indexed: 12/20/2022]
Abstract
Regulating how fast to learn is critical for flexible behavior. Learning about the consequences of actions should be slow in stable environments, but accelerate when that environment changes. Recognizing stability and detecting change are difficult in environments with noisy relationships between actions and outcomes. Under these conditions, theories propose that uncertainty can be used to modulate learning rates ("meta-learning"). We show that mice behaving in a dynamic foraging task exhibit choice behavior that varied as a function of two forms of uncertainty estimated from a meta-learning model. The activity of dorsal raphe serotonin neurons tracked both types of uncertainty in the foraging task as well as in a dynamic Pavlovian task. Reversible inhibition of serotonin neurons in the foraging task reproduced changes in learning predicted by a simulated lesion of meta-learning in the model. We thus provide a quantitative link between serotonin neuron activity, learning, and decision making.
Collapse
Affiliation(s)
- Cooper D Grossman
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Bilal A Bari
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA.
| |
Collapse
|
19
|
Groman SM, Thompson SL, Lee D, Taylor JR. Reinforcement learning detuned in addiction: integrative and translational approaches. Trends Neurosci 2022; 45:96-105. [PMID: 34920884 PMCID: PMC8770604 DOI: 10.1016/j.tins.2021.11.007] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2021] [Revised: 11/04/2021] [Accepted: 11/19/2021] [Indexed: 02/03/2023]
Abstract
Suboptimal decision-making strategies have been proposed to contribute to the pathophysiology of addiction. Decision-making, however, arises from a collection of computational components that can independently influence behavior. Disruptions in these different components can lead to decision-making deficits that appear similar behaviorally, but differ at the computational, and likely the neurobiological, level. Here, we discuss recent studies that have used computational approaches to investigate the decision-making processes underlying addiction. Studies in animal models have found that value updating following positive, but not negative, outcomes is predictive of drug use, whereas value updating following negative, but not positive, outcomes is disrupted following drug self-administration. We contextualize these findings with studies on the circuit and biological mechanisms of decision-making to develop a framework for revealing the biobehavioral mechanisms of addiction.
Collapse
Affiliation(s)
- Stephanie M. Groman
- Department of Neuroscience, University of Minnesota,Department of Psychiatry, Yale University,Correspondence to be directed to: Stephanie Groman, 321 Church Street SE, 4-125 Jackson Hall Minneapolis MN 55455,
| | | | - Daeyeol Lee
- The Zanvyl Krieger Mind/Brain Institute, The Solomon H Snyder Department of Neuroscience, Department of Psychological and Brain Sciences, Kavli Neuroscience Discovery Institute, Johns Hopkins University
| | - Jane R. Taylor
- Department of Psychiatry, Yale University,Department of Neuroscience, Yale University,Department of Psychology, Yale University
| |
Collapse
|
20
|
Rudebeck PH, Izquierdo A. Foraging with the frontal cortex: A cross-species evaluation of reward-guided behavior. Neuropsychopharmacology 2022; 47:134-146. [PMID: 34408279 PMCID: PMC8617092 DOI: 10.1038/s41386-021-01140-0] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/05/2021] [Revised: 07/30/2021] [Accepted: 07/30/2021] [Indexed: 02/07/2023]
Abstract
Efficient foraging is essential to survival and depends on frontal cortex in mammals. Because of its role in psychiatric disorders, frontal cortex and its contributions to reward procurement have been studied extensively in both rodents and non-human primates. How frontal cortex of these animal models compares is a source of intense debate. Here we argue that translating findings from rodents to non-human primates requires an appreciation of both the niche in which each animal forages as well as the similarities in frontal cortex anatomy and function. Consequently, we highlight similarities and differences in behavior and anatomy, before focusing on points of convergence in how parts of frontal cortex contribute to distinct aspects of foraging in rats and macaques, more specifically. In doing so, our aim is to emphasize where translation of frontal cortex function between species is clearer, where there is divergence, and where future work should focus. We finish by highlighting aspects of foraging for which have received less attention but we believe are critical to uncovering how frontal cortex promotes survival in each species.
Collapse
Affiliation(s)
| | - Alicia Izquierdo
- Department of Psychology, UCLA, Los Angeles, CA, USA.
- The Brain Research Institute, UCLA, Los Angeles, CA, USA.
- Integrative Center for Learning and Memory, UCLA, Los Angeles, CA, USA.
- Integrative Center for Addictions, UCLA, Los Angeles, CA, USA.
| |
Collapse
|
21
|
Soltani A, Koechlin E. Computational models of adaptive behavior and prefrontal cortex. Neuropsychopharmacology 2022; 47:58-71. [PMID: 34389808 PMCID: PMC8617006 DOI: 10.1038/s41386-021-01123-1] [Citation(s) in RCA: 36] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/10/2021] [Revised: 07/19/2021] [Accepted: 07/20/2021] [Indexed: 02/07/2023]
Abstract
The real world is uncertain, and while ever changing, it constantly presents itself in terms of new sets of behavioral options. To attain the flexibility required to tackle these challenges successfully, most mammalian brains are equipped with certain computational abilities that rely on the prefrontal cortex (PFC). By examining learning in terms of internal models associating stimuli, actions, and outcomes, we argue here that adaptive behavior relies on specific interactions between multiple systems including: (1) selective models learning stimulus-action associations through rewards; (2) predictive models learning stimulus- and/or action-outcome associations through statistical inferences anticipating behavioral outcomes; and (3) contextual models learning external cues associated with latent states of the environment. Critically, the PFC combines these internal models by forming task sets to drive behavior and, moreover, constantly evaluates the reliability of actor task sets in predicting external contingencies to switch between task sets or create new ones. We review different models of adaptive behavior to demonstrate how their components map onto this unifying framework and specific PFC regions. Finally, we discuss how our framework may help to better understand the neural computations and the cognitive architecture of PFC regions guiding adaptive behavior.
Collapse
Affiliation(s)
- Alireza Soltani
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| | - Etienne Koechlin
- Institut National de la Sante et de la Recherche Medicale, Universite Pierre et Marie Curie, Ecole Normale Superieure, Paris, France.
| |
Collapse
|
22
|
Overman MJ, Browning M, O'Shea J. Inducing Affective Learning Biases with Cognitive Training and Prefrontal tDCS: A Proof-of-Concept Study. COGNITIVE THERAPY AND RESEARCH 2021; 45:869-884. [PMID: 34720259 PMCID: PMC8550254 DOI: 10.1007/s10608-020-10146-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/28/2020] [Indexed: 11/27/2022]
Abstract
Background Cognitive models of mood disorders emphasize a causal role of negative affective biases in depression. Computational work suggests that these biases may stem from a belief that negative events have a higher information content than positive events, resulting in preferential processing of and learning from negative outcomes. Learning biases therefore represent a promising target for therapeutic interventions. In this proof-of-concept study in healthy volunteers, we assessed the malleability of biased reinforcement learning using a novel cognitive training paradigm and concurrent transcranial direct current stimulation (tDCS). Methods In two studies, young healthy adults completed two sessions of negative (n = 20) or positive (n = 20) training designed to selectively increase learning from loss or win outcomes, respectively. During training active or sham tDCS was applied bilaterally to dorsolateral prefrontal cortex. Analyses tested for changes both in learning rates and win- and loss-driven behaviour. Potential positive/negative emotional transfer of win/loss learning was assessed by a facial emotion recognition task and mood questionnaires. Results Negative and positive training increased learning rates for losses and wins, respectively. With negative training, there was also a trend for win (but not loss) learning rates to decrease over successive task blocks. After negative training, there was evidence for near transfer in the form of an increase in loss-driven choices when participants performed a similar (untrained) task. There was no change in far transfer measures of emotional face processing or mood. tDCS had no effect on any aspect of behaviour. Discussion and Conclusions Negative training induced a mild negative bias in healthy adults as reflected in loss-driven choice behaviour. Prefrontal tDCS had no effect. Further research is needed to assess if this training procedure can be adapted to enhance learning from positive outcomes and whether effects translate to affective disorders. Electronic supplementary material The online version of this article (10.1007/s10608-020-10146-9) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Margot Juliëtte Overman
- Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, OX3 9DU England
| | - Michael Browning
- Department of Psychiatry, University of Oxford, Warneford Hospital, Oxford, OX3 7JX England.,Oxford Health NHS Foundation Trust, Warneford Hospital, Warneford Lane, Oxford, OX3 7JX England
| | - Jacinta O'Shea
- Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, OX3 9DU England.,Department of Psychiatry, University of Oxford, Warneford Hospital, Oxford, OX3 7JX England.,Oxford Centre for Human Brain Activity (OHBA), Wellcome Centre for Integrative Neuroimaging, Department of Psychiatry, University of Oxford, Oxford, OX3 7JX England
| |
Collapse
|
23
|
Abstract
We live in a world that changes on many timescales. To learn and make decisions appropriately, the human brain has evolved to integrate various types of information, such as sensory evidence and reward feedback, on multiple timescales. This is reflected in cortical hierarchies of timescales consisting of heterogeneous neuronal activities and expression of genes related to neurotransmitters critical for learning. We review the recent findings on how timescales of sensory and reward integration are affected by the temporal properties of sensory and reward signals in the environment. Despite existing evidence linking behavioral and neuronal timescales, future studies must examine how neural computations at multiple timescales are adjusted and combined to influence behavior flexibly.
Collapse
Affiliation(s)
- Alireza Soltani
- Department of Psychological and Brain Sciences, Dartmouth College, Moore Hall, 3 Maynard St, Hanover, NH 03755
| | - John D. Murray
- Department of Psychiatry, Yale School of Medicine, 300 George Street, New Haven, CT 06511
| | - Hyojung Seo
- Department of Psychiatry, Yale School of Medicine, 300 George Street, New Haven, CT 06511
| | - Daeyeol Lee
- The Zanvyl Krieger Mind/Brain Institute, Department of Neuroscience, Department of Psychological Sciences, Kavli Neuroscience Discovery Institute, Johns Hopkins University, 3400 North Charles Street, Baltimore, MD 21218
| |
Collapse
|
24
|
Ghambaryan A, Gutkin B, Klucharev V, Koechlin E. Additively Combining Utilities and Beliefs: Research Gaps and Algorithmic Developments. Front Neurosci 2021; 15:704728. [PMID: 34658760 PMCID: PMC8517513 DOI: 10.3389/fnins.2021.704728] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2021] [Accepted: 09/13/2021] [Indexed: 11/20/2022] Open
Abstract
Value-based decision making in complex environments, such as those with uncertain and volatile mapping of reward probabilities onto options, may engender computational strategies that are not necessarily optimal in terms of normative frameworks but may ensure effective learning and behavioral flexibility in conditions of limited neural computational resources. In this article, we review a suboptimal strategy - additively combining reward magnitude and reward probability attributes of options for value-based decision making. In addition, we present computational intricacies of a recently developed model (named MIX model) representing an algorithmic implementation of the additive strategy in sequential decision-making with two options. We also discuss its opportunities; and conceptual, inferential, and generalization issues. Furthermore, we suggest future studies that will reveal the potential and serve the further development of the MIX model as a general model of value-based choice making.
Collapse
Affiliation(s)
- Anush Ghambaryan
- Centre for Cognition and Decision Making, HSE University, Moscow, Russia
- Ecole Normale Supérieure, PSL Research University, Paris, France
| | - Boris Gutkin
- Centre for Cognition and Decision Making, HSE University, Moscow, Russia
- Ecole Normale Supérieure, PSL Research University, Paris, France
| | - Vasily Klucharev
- Centre for Cognition and Decision Making, HSE University, Moscow, Russia
| | - Etienne Koechlin
- Ecole Normale Supérieure, PSL Research University, Paris, France
| |
Collapse
|
25
|
Groman SM, Lee D, Taylor JR. Unlocking the reinforcement-learning circuits of the orbitofrontal cortex. Behav Neurosci 2021; 135:120-128. [PMID: 34060870 DOI: 10.1037/bne0000414] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
Abstract
Neuroimaging studies have consistently identified the orbitofrontal cortex (OFC) as being affected in individuals with neuropsychiatric disorders. OFC dysfunction has been proposed to be a key mechanism by which decision-making impairments emerge in diverse clinical populations, and recent studies employing computational approaches have revealed that distinct reinforcement-learning mechanisms of decision-making differ among diagnoses. In this perspective, we propose that these computational differences may be linked to select OFC circuits and present our recent work that has used a neurocomputational approach to understand the biobehavioral mechanisms of addiction pathology in rodent models. We describe how combining translationally analogous behavioral paradigms with reinforcement-learning algorithms and sophisticated neuroscience techniques in animals can provide critical insights into OFC pathology in biobehavioral disorders. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
Collapse
|
26
|
Cavanagh SE, Hunt LT, Kennerley SW. A Diversity of Intrinsic Timescales Underlie Neural Computations. Front Neural Circuits 2020; 14:615626. [PMID: 33408616 PMCID: PMC7779632 DOI: 10.3389/fncir.2020.615626] [Citation(s) in RCA: 54] [Impact Index Per Article: 10.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2020] [Accepted: 11/18/2020] [Indexed: 12/05/2022] Open
Abstract
Neural processing occurs across a range of temporal scales. To facilitate this, the brain uses fast-changing representations reflecting momentary sensory input alongside more temporally extended representations, which integrate across both short and long temporal windows. The temporal flexibility of these representations allows animals to behave adaptively. Short temporal windows facilitate adaptive responding in dynamic environments, while longer temporal windows promote the gradual integration of information across time. In the cognitive and motor domains, the brain sets overarching goals to be achieved within a long temporal window, which must be broken down into sequences of actions and precise movement control processed across much shorter temporal windows. Previous human neuroimaging studies and large-scale artificial network models have ascribed different processing timescales to different cortical regions, linking this to each region's position in an anatomical hierarchy determined by patterns of inter-regional connectivity. However, even within cortical regions, there is variability in responses when studied with single-neuron electrophysiology. Here, we review a series of recent electrophysiology experiments that demonstrate the heterogeneity of temporal receptive fields at the level of single neurons within a cortical region. This heterogeneity appears functionally relevant for the computations that neurons perform during decision-making and working memory. We consider anatomical and biophysical mechanisms that may give rise to a heterogeneity of timescales, including recurrent connectivity, cortical layer distribution, and neurotransmitter receptor expression. Finally, we reflect on the computational relevance of each brain region possessing a heterogeneity of neuronal timescales. We argue that this architecture is of particular importance for sensory, motor, and cognitive computations.
Collapse
Affiliation(s)
- Sean E. Cavanagh
- Department of Clinical and Movement Neurosciences, University College London, London, United Kingdom
| | - Laurence T. Hunt
- Department of Clinical and Movement Neurosciences, University College London, London, United Kingdom
- Wellcome Trust Centre for Neuroimaging, University College London, London, United Kingdom
- Max Planck-UCL Centre for Computational Psychiatry and Aging, University College London, London, United Kingdom
- Department of Psychiatry, Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, United Kingdom
| | - Steven W. Kennerley
- Department of Clinical and Movement Neurosciences, University College London, London, United Kingdom
| |
Collapse
|
27
|
Wang J, Hosseini E, Meirhaeghe N, Akkad A, Jazayeri M. Reinforcement regulates timing variability in thalamus. eLife 2020; 9:55872. [PMID: 33258769 PMCID: PMC7707818 DOI: 10.7554/elife.55872] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2020] [Accepted: 11/06/2020] [Indexed: 01/19/2023] Open
Abstract
Learning reduces variability but variability can facilitate learning. This paradoxical relationship has made it challenging to tease apart sources of variability that degrade performance from those that improve it. We tackled this question in a context-dependent timing task requiring humans and monkeys to flexibly produce different time intervals with different effectors. We identified two opposing factors contributing to timing variability: slow memory fluctuation that degrades performance and reward-dependent exploratory behavior that improves performance. Signatures of these opposing factors were evident across populations of neurons in the dorsomedial frontal cortex (DMFC), DMFC-projecting neurons in the ventrolateral thalamus, and putative target of DMFC in the caudate. However, only in the thalamus were the performance-optimizing regulation of variability aligned to the slow performance-degrading memory fluctuations. These findings reveal how variability caused by exploratory behavior might help to mitigate other undesirable sources of variability and highlight a potential role for thalamocortical projections in this process.
Collapse
Affiliation(s)
- Jing Wang
- Department of Bioengineering, University of Missouri, Columbia, United States.,McGovern Institute for Brain Research, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States
| | - Eghbal Hosseini
- McGovern Institute for Brain Research, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States
| | - Nicolas Meirhaeghe
- Harvard-MIT Division of Health Sciences and Technology, Cambridge, United States
| | - Adam Akkad
- McGovern Institute for Brain Research, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States
| | - Mehrdad Jazayeri
- McGovern Institute for Brain Research, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States
| |
Collapse
|
28
|
Blain B, Rutledge RB. Momentary subjective well-being depends on learning and not reward. eLife 2020; 9:57977. [PMID: 33200989 PMCID: PMC7755387 DOI: 10.7554/elife.57977] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2020] [Accepted: 11/16/2020] [Indexed: 01/20/2023] Open
Abstract
Subjective well-being or happiness is often associated with wealth. Recent studies suggest that momentary happiness is associated with reward prediction error, the difference between experienced and predicted reward, a key component of adaptive behaviour. We tested subjects in a reinforcement learning task in which reward size and probability were uncorrelated, allowing us to dissociate between the contributions of reward and learning to happiness. Using computational modelling, we found convergent evidence across stable and volatile learning tasks that happiness, like behaviour, is sensitive to learning-relevant variables (i.e. probability prediction error). Unlike behaviour, happiness is not sensitive to learning-irrelevant variables (i.e. reward prediction error). Increasing volatility reduces how many past trials influence behaviour but not happiness. Finally, depressive symptoms reduce happiness more in volatile than stable environments. Our results suggest that how we learn about our world may be more important for how we feel than the rewards we actually receive. Many people believe they would be happier if only they had more money. And events such as winning the lottery or receiving a large pay rise do make people happy, at least temporarily. But recent studies suggest that the main factor driving happiness on such occasions is not the size of the reward received. Instead, it is how well that reward matches up with expectations. Receiving a 10% pay rise when you were expecting 1% will make you feel happier than receiving 10% when you had been expecting 20%. This difference between an expected and an actual reward is referred to as a reward prediction error. Reward prediction errors have a key role in learning. They motivate people to repeat behaviours that led to unexpectedly large rewards. But they also enable people to update their beliefs about the world, which is rewarding in itself. Could it be that reward prediction errors are associated with happiness mainly because they help us understand the world a little better than before? To test this idea, Blain and Rutledge designed a task in which the likelihood of receiving a reward was unrelated to the size of the reward. This study design makes it possible to separate out the contributions of learning versus reward to moment-by-moment happiness. In the task, volunteers had to decide which of two cars would win a race. In the ‘stable’ condition, one of the cars always had an 80% chance of winning. In the ‘volatile’ condition, one car had an 80% chance of winning for the first 20 trials. The other car then had an 80% chance of winning for the next 20 trials. The volunteers were not told these probabilities in advance, but had to work them out by playing the game. However, on every trial, the volunteers were shown the reward they would receive if they chose either of the cars and that car went on to win. The size of the rewards varied at random and was unrelated to the likelihood of a car winning. Every few trials, the volunteers were asked to indicate their current level of happiness on a scale. The results showed that volunteers were happier after winning than after losing. On average they were also happier in the stable condition than in the volatile condition. This was especially true for volunteers with pre-existing symptoms of depression. Moreover, happiness after wins did not depend on how large the reward they got was, but instead simply on how surprised they were to win. These results suggest that how we learn about the world around us can be more important for how we feel than rewards we receive directly. Measuring happiness in various types of environment could help us understand factors affecting mental health. The current results suggest, for example, that uncertain environments may be especially unpleasant for people with depression. Further research is needed to understand why this might be the case. In the real world, rewards are often uncertain and infrequent, but learning may nevertheless have the potential to boost happiness.
Collapse
Affiliation(s)
- Bastien Blain
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom
| | - Robb B Rutledge
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom.,Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom.,Department of Psychology, Yale University, New Haven, United States
| |
Collapse
|
29
|
Dalgleish HWP, Russell LE, Packer AM, Roth A, Gauld OM, Greenstreet F, Thompson EJ, Häusser M. How many neurons are sufficient for perception of cortical activity? eLife 2020; 9:e58889. [PMID: 33103656 PMCID: PMC7695456 DOI: 10.7554/elife.58889] [Citation(s) in RCA: 60] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Accepted: 10/17/2020] [Indexed: 01/12/2023] Open
Abstract
Many theories of brain function propose that activity in sparse subsets of neurons underlies perception and action. To place a lower bound on the amount of neural activity that can be perceived, we used an all-optical approach to drive behaviour with targeted two-photon optogenetic activation of small ensembles of L2/3 pyramidal neurons in mouse barrel cortex while simultaneously recording local network activity with two-photon calcium imaging. By precisely titrating the number of neurons stimulated, we demonstrate that the lower bound for perception of cortical activity is ~14 pyramidal neurons. We find a steep sigmoidal relationship between the number of activated neurons and behaviour, saturating at only ~37 neurons, and show this relationship can shift with learning. Furthermore, activation of ensembles is balanced by inhibition of neighbouring neurons. This surprising perceptual sensitivity in the face of potent network suppression supports the sparse coding hypothesis, and suggests that cortical perception balances a trade-off between minimizing the impact of noise while efficiently detecting relevant signals.
Collapse
Affiliation(s)
- Henry WP Dalgleish
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| | - Lloyd E Russell
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| | - Adam M Packer
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| | - Arnd Roth
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| | - Oliver M Gauld
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| | - Francesca Greenstreet
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| | - Emmett J Thompson
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| | - Michael Häusser
- Wolfson Institute for Biomedical Research, University College LondonLondonUnited Kingdom
| |
Collapse
|
30
|
Ounjai K, Suppaso L, Hohwy J, Lauwereyns J. Tracking the Influence of Predictive Cues on the Evaluation of Food Images: Volatility Enables Nudging. Front Psychol 2020; 11:569078. [PMID: 33041935 PMCID: PMC7522349 DOI: 10.3389/fpsyg.2020.569078] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2020] [Accepted: 08/13/2020] [Indexed: 11/25/2022] Open
Abstract
In previous research on the evaluation of food images, we found that appetitive food images were rated higher following a positive prediction than following a negative prediction, and vice versa for aversive food images. The findings suggested an active confirmation bias. Here, we examine whether this influence from prediction depends on the evaluative polarization of the food images. Specifically, we divided the set of food images into “strong” and “mild” images by how polarized (i.e., extreme) their average ratings were across all conditions. With respect to the influence from prediction, we raise two alternative hypotheses. According to a predictive dissonance hypothesis, the larger the discrepancy between prediction and outcome, the stronger the active inference toward accommodating the outcome with the prediction; thus, the confirmation bias should obtain particularly with strong images. Conversely, according to a nudging-in-volatility hypothesis, the active confirmation bias operates only on images within a dynamic range, where the values of images are volatile, and not on the evaluation of images that are too obviously appetitive or aversive; accordingly, the effects from prediction should occur predominately with mild images. Across the data from two experiments, we found that the evaluation of mild images tended to exhibit the confirmation bias, with ratings that followed the direction given by the prediction. For strong images, there was no confirmation bias. Our findings corroborate the nudging-in-volatility hypothesis, suggesting that predictive cues may be able to tip the balance of evaluation particularly for food images that do not have a strongly polarized value.
Collapse
Affiliation(s)
- Kajornvut Ounjai
- Biological Engineering Program, Faculty of Engineering, King Mongkut's University of Technology Thonburi, Bangkok, Thailand
| | - Lalida Suppaso
- Biological Engineering Program, Faculty of Engineering, King Mongkut's University of Technology Thonburi, Bangkok, Thailand
| | - Jakob Hohwy
- School of Philosophical, Historical, and International Studies, Monash University, Melbourne, VIC, Australia
| | - Johan Lauwereyns
- Graduate School of Systems Life Sciences, Kyushu University, Fukuoka, Japan.,School of Interdisciplinary Science and Innovation, Kyushu University, Fukuoka, Japan.,Faculty of Arts and Science, Kyushu University, Fukuoka, Japan
| |
Collapse
|
31
|
|
32
|
Moin Afshar N, Keip AJ, Taylor JR, Lee D, Groman SM. Reinforcement Learning during Adolescence in Rats. J Neurosci 2020; 40:5857-5870. [PMID: 32601244 PMCID: PMC7380962 DOI: 10.1523/jneurosci.0910-20.2020] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2020] [Revised: 06/07/2020] [Accepted: 06/17/2020] [Indexed: 12/14/2022] Open
Abstract
The most dynamic period of postnatal brain development occurs during adolescence, the period between childhood and adulthood. Neuroimaging studies have observed morphologic and functional changes during adolescence, and it is believed that these changes serve to improve the functions of circuits that underlie decision-making. Direct evidence in support of this hypothesis, however, has been limited because most preclinical decision-making paradigms are not readily translated to humans. Here, we developed a reversal-learning protocol for the rapid assessment of adaptive choice behavior in dynamic environments in rats as young as postnatal day 30. A computational framework was used to elucidate the reinforcement-learning mechanisms that change in adolescence and into adulthood. Using a cross-sectional and longitudinal design, we provide the first evidence that value-based choice behavior in a reversal-learning task improves during adolescence in male and female Long-Evans rats and demonstrate that the increase in reversal performance is due to alterations in value updating for positive outcomes. Furthermore, we report that reversal-learning trajectories in adolescence reliably predicted reversal performance in adulthood. This novel behavioral protocol provides a unique platform for conducting biological and systems-level analyses of the neurodevelopmental mechanisms of decision-making.SIGNIFICANCE STATEMENT The neurodevelopmental adaptations that occur during adolescence are hypothesized to underlie age-related improvements in decision-making, but evidence to support this hypothesis has been limited. Here, we describe a novel behavioral protocol for rapidly assessing adaptive choice behavior in adolescent rats with a reversal-learning paradigm. Using a computational approach, we demonstrate that age-related changes in reversal-learning performance in male and female Long-Evans rats are linked to specific reinforcement-learning mechanisms and are predictive of reversal-learning performance in adulthood. Our behavioral protocol provides a unique platform for elucidating key components of adolescent brain function.
Collapse
Affiliation(s)
- Neema Moin Afshar
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| | - Alex J Keip
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| | - Jane R Taylor
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
- Department of Neuroscience, Yale School of Medicine, New Haven, Connecticut 06520-8001
| | - Daeyeol Lee
- The Zanvyl Krieger Mind/Brain Institute, Johns Hopkins University, Baltimore, Maryland 21218
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins University School of Medicine, Baltimore, Maryland 21205
- Department of Psychological & Brain Sciences, Johns Hopkins University, Baltimore, Maryland 21218
- Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, Maryland 21205
| | - Stephanie M Groman
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| |
Collapse
|
33
|
Soltani A, Izquierdo A. Adaptive learning under expected and unexpected uncertainty. Nat Rev Neurosci 2020; 20:635-644. [PMID: 31147631 DOI: 10.1038/s41583-019-0180-y] [Citation(s) in RCA: 143] [Impact Index Per Article: 28.6] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
The outcome of a decision is often uncertain, and outcomes can vary over repeated decisions. Whether decision outcomes should substantially affect behaviour and learning depends on whether they are representative of a typically experienced range of outcomes or signal a change in the reward environment. Successful learning and decision-making therefore require the ability to estimate expected uncertainty (related to the variability of outcomes) and unexpected uncertainty (related to the variability of the environment). Understanding the bases and effects of these two types of uncertainty and the interactions between them - at the computational and the neural level - is crucial for understanding adaptive learning. Here, we examine computational models and experimental findings to distil computational principles and neural mechanisms for adaptive learning under uncertainty.
Collapse
Affiliation(s)
- Alireza Soltani
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| | - Alicia Izquierdo
- Department of Psychology, The Brain Research Institute, University of California, Los Angeles, Los Angeles, CA, USA.
| |
Collapse
|
34
|
Cook JL, Swart JC, Froböse MI, Diaconescu AO, Geurts DEM, den Ouden HEM, Cools R. Catecholaminergic modulation of meta-learning. eLife 2019; 8:e51439. [PMID: 31850844 PMCID: PMC6974360 DOI: 10.7554/elife.51439] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2019] [Accepted: 12/18/2019] [Indexed: 01/03/2023] Open
Abstract
The remarkable expedience of human learning is thought to be underpinned by meta-learning, whereby slow accumulative learning processes are rapidly adjusted to the current learning environment. To date, the neurobiological implementation of meta-learning remains unclear. A burgeoning literature argues for an important role for the catecholamines dopamine and noradrenaline in meta-learning. Here, we tested the hypothesis that enhancing catecholamine function modulates the ability to optimise a meta-learning parameter (learning rate) as a function of environmental volatility. 102 participants completed a task which required learning in stable phases, where the probability of reinforcement was constant, and volatile phases, where probabilities changed every 10-30 trials. The catecholamine transporter blocker methylphenidate enhanced participants' ability to adapt learning rate: Under methylphenidate, compared with placebo, participants exhibited higher learning rates in volatile relative to stable phases. Furthermore, this effect was significant only with respect to direct learning based on the participants' own experience, there was no significant effect on inferred-value learning where stimulus values had to be inferred. These data demonstrate a causal link between catecholaminergic modulation and the adjustment of the meta-learning parameter learning rate.
Collapse
Affiliation(s)
- Jennifer L Cook
- School of PsychologyUniversity of BirminghamBirminghamUnited Kingdom
| | - Jennifer C Swart
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
| | - Monja I Froböse
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
| | - Andreea O Diaconescu
- Translational Neuromodeling Unit, Institute for Biomedical EngineeringUniversity of Zurich and ETH ZurichZurichSwitzerland
- Department of PsychiatryUniversity of BaselBaselSwitzerland
- Krembil Centre for Neuroinformatics,CAMHUniversity of TorontoTorontoCanada
| | - Dirk EM Geurts
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
- Department of PsychiatryRadboud University Medical CentreNijmegenNetherlands
| | - Hanneke EM den Ouden
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
| | - Roshan Cools
- Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive NeuroimagingRadboud UniversityNijmegenNetherlands
- Department of PsychiatryRadboud University Medical CentreNijmegenNetherlands
| |
Collapse
|
35
|
Farashahi S, Donahue CH, Hayden BY, Lee D, Soltani A. Flexible combination of reward information across primates. Nat Hum Behav 2019; 3:1215-1224. [PMID: 31501543 PMCID: PMC6856432 DOI: 10.1038/s41562-019-0714-3] [Citation(s) in RCA: 56] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2018] [Accepted: 07/29/2019] [Indexed: 01/09/2023]
Abstract
A fundamental but rarely contested assumption in economics and neuroeconomics is that decision-makers compute subjective values of risky options by multiplying functions of reward probability and magnitude. By contrast, an additive strategy for valuation allows flexible combination of reward information required in uncertain or changing environments. We hypothesized that the level of uncertainty in the reward environment should determine the strategy used for valuation and choice. To test this hypothesis, we examined choice between risky options in humans and rhesus macaques across three tasks with different levels of uncertainty. We found that whereas humans and monkeys adopted a multiplicative strategy under risk when probabilities are known, both species spontaneously adopted an additive strategy under uncertainty when probabilities must be learned. Additionally, the level of volatility influenced relative weighting of certain and uncertain reward information, and this was reflected in the encoding of reward magnitude by neurons in the dorsolateral prefrontal cortex.
Collapse
Affiliation(s)
- Shiva Farashahi
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Christopher H Donahue
- The Gladstone Institutes, San Francisco, CA, USA
- Department of Neuroscience, Yale School of Medicine, New Haven, CT, USA
| | - Benjamin Y Hayden
- Department of Neuroscience and Center for Magnetic Resonance Imaging, University of Minnesota, Minneapolis, MN, USA
| | - Daeyeol Lee
- Department of Neuroscience, Yale School of Medicine, New Haven, CT, USA
- The Zanvyl Krieger Mind/Brain Institute, Department of Neuroscience, Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Alireza Soltani
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| |
Collapse
|
36
|
Pulcu E, Browning M. The Misestimation of Uncertainty in Affective Disorders. Trends Cogn Sci 2019; 23:865-875. [PMID: 31431340 DOI: 10.1016/j.tics.2019.07.007] [Citation(s) in RCA: 78] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2019] [Revised: 07/22/2019] [Accepted: 07/22/2019] [Indexed: 01/28/2023]
Abstract
Our knowledge about the state of the world is often incomplete, making it difficult to select the best course of action. One strategy that can be used to improve our ability to make decisions is to identify the causes of our ignorance (i.e., why an unexpected event might have occurred) and use estimates of the uncertainty induced by these causes to guide our learning. Here, we explain the logic behind this process and describe the evidence that human learners use estimates of uncertainty to sculpt their learning. Finally, we describe recent work suggesting that misestimation of uncertainty is involved in the development of anxiety and depression and describe how these ideas may be advanced.
Collapse
Affiliation(s)
- Erdem Pulcu
- Department of Psychiatry, University of Oxford, Oxford, UK
| | - Michael Browning
- Department of Psychiatry, University of Oxford, Oxford, UK; Oxford Health Foundation Trust, Oxford, UK.
| |
Collapse
|
37
|
Grabenhorst F, Tsutsui KI, Kobayashi S, Schultz W. Primate prefrontal neurons signal economic risk derived from the statistics of recent reward experience. eLife 2019; 8:e44838. [PMID: 31343407 PMCID: PMC6658165 DOI: 10.7554/elife.44838] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2019] [Accepted: 07/12/2019] [Indexed: 01/28/2023] Open
Abstract
Risk derives from the variation of rewards and governs economic decisions, yet how the brain calculates risk from the frequency of experienced events, rather than from explicit risk-descriptive cues, remains unclear. Here, we investigated whether neurons in dorsolateral prefrontal cortex process risk derived from reward experience. Monkeys performed in a probabilistic choice task in which the statistical variance of experienced rewards evolved continually. During these choices, prefrontal neurons signaled the reward-variance associated with specific objects ('object risk') or actions ('action risk'). Crucially, risk was not derived from explicit, risk-descriptive cues but calculated internally from the variance of recently experienced rewards. Support-vector-machine decoding demonstrated accurate neuronal risk discrimination. Within trials, neuronal signals transitioned from experienced reward to risk (risk updating) and from risk to upcoming choice (choice computation). Thus, prefrontal neurons encode the statistical variance of recently experienced rewards, complying with formal decision variables of object risk and action risk.
Collapse
Affiliation(s)
- Fabian Grabenhorst
- Department of Physiology, Development and NeuroscienceUniversity of CambridgeCambridgeUnited Kingdom
| | - Ken-Ichiro Tsutsui
- Department of Physiology, Development and NeuroscienceUniversity of CambridgeCambridgeUnited Kingdom
| | - Shunsuke Kobayashi
- Department of Physiology, Development and NeuroscienceUniversity of CambridgeCambridgeUnited Kingdom
| | - Wolfram Schultz
- Department of Physiology, Development and NeuroscienceUniversity of CambridgeCambridgeUnited Kingdom
| |
Collapse
|
38
|
Groman SM, Keistler C, Keip AJ, Hammarlund E, DiLeone RJ, Pittenger C, Lee D, Taylor JR. Orbitofrontal Circuits Control Multiple Reinforcement-Learning Processes. Neuron 2019; 103:734-746.e3. [PMID: 31253468 DOI: 10.1016/j.neuron.2019.05.042] [Citation(s) in RCA: 98] [Impact Index Per Article: 16.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2019] [Revised: 04/18/2019] [Accepted: 05/24/2019] [Indexed: 12/18/2022]
Abstract
Adaptive decision making in dynamic environments requires multiple reinforcement-learning steps that may be implemented by dissociable neural circuits. Here, we used a novel directionally specific viral ablation approach to investigate the function of several anatomically defined orbitofrontal cortex (OFC) circuits during adaptive, flexible decision making in rats trained on a probabilistic reversal learning task. Ablation of OFC neurons projecting to the nucleus accumbens selectively disrupted performance following a reversal, by disrupting the use of negative outcomes to guide subsequent choices. Ablation of amygdala neurons projecting to the OFC also impaired reversal performance, but due to disruptions in the use of positive outcomes to guide subsequent choices. Ablation of OFC neurons projecting to the amygdala, by contrast, enhanced reversal performance by destabilizing action values. Our data are inconsistent with a unitary function of the OFC in decision making. Rather, distinct OFC-amygdala-striatal circuits mediate distinct components of the action-value updating and maintenance necessary for decision making.
Collapse
Affiliation(s)
| | - Colby Keistler
- Department of Psychiatry, Yale University, New Haven, CT 06515, USA
| | - Alex J Keip
- Department of Psychiatry, Yale University, New Haven, CT 06515, USA
| | - Emma Hammarlund
- Department of Psychiatry, Yale University, New Haven, CT 06515, USA
| | - Ralph J DiLeone
- Department of Psychiatry, Yale University, New Haven, CT 06515, USA; Department of Neuroscience, Yale University, New Haven, CT 06515, USA
| | - Christopher Pittenger
- Department of Psychiatry, Yale University, New Haven, CT 06515, USA; Child Study Center, Yale University, New Haven, CT 06515, USA
| | - Daeyeol Lee
- Department of Psychiatry, Yale University, New Haven, CT 06515, USA; Department of Neuroscience, Yale University, New Haven, CT 06515, USA; Department of Psychology, Yale University, New Haven, CT 06515, USA
| | - Jane R Taylor
- Department of Psychiatry, Yale University, New Haven, CT 06515, USA; Department of Neuroscience, Yale University, New Haven, CT 06515, USA; Department of Psychology, Yale University, New Haven, CT 06515, USA.
| |
Collapse
|
39
|
Sarafyazd M, Jazayeri M. Hierarchical reasoning by neural circuits in the frontal cortex. Science 2019; 364:364/6441/eaav8911. [DOI: 10.1126/science.aav8911] [Citation(s) in RCA: 68] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2018] [Accepted: 04/01/2019] [Indexed: 12/20/2022]
Abstract
Humans process information hierarchically. In the presence of hierarchies, sources of failures are ambiguous. Humans resolve this ambiguity by assessing their confidence after one or more attempts. To understand the neural basis of this reasoning strategy, we recorded from dorsomedial frontal cortex (DMFC) and anterior cingulate cortex (ACC) of monkeys in a task in which negative outcomes were caused either by misjudging the stimulus or by a covert switch between two stimulus-response contingency rules. We found that both areas harbored a representation of evidence supporting a rule switch. Additional perturbation experiments revealed that ACC functioned downstream of DMFC and was directly and specifically involved in inferring covert rule switches. These results reveal the computational principles of hierarchical reasoning, as implemented by cortical circuits.
Collapse
|