Hypothalamic Interactions with Large-Scale Neural Circuits Underlying Reinforcement Learning and Motivated Behavior

Single-cell and spatial omics: exploring hypothalamic heterogeneity

Elucidating the complex dynamic cellular organization in the hypothalamus is critical for understanding its role in coordinating fundamental body functions. Over the past decade, single-cell and spatial omics technologies have significantly evolved, overcoming initial technical challenges in capturing and analyzing individual cells. These high-throughput omics technologies now offer a remarkable opportunity to comprehend the complex spatiotemporal patterns of transcriptional diversity and cell-type characteristics across the entire hypothalamus. Current single-cell and single-nucleus RNA sequencing methods comprehensively quantify gene expression by exploring distinct phenotypes across various subregions of the hypothalamus. However, single-cell/single-nucleus RNA sequencing requires isolating the cell/nuclei from the tissue, potentially resulting in the loss of spatial information concerning neuronal networks. Spatial transcriptomics methods, by bypassing the cell dissociation, can elucidate the intricate spatial organization of neural networks through their imaging and sequencing technologies. In this review, we highlight the applicative value of single-cell and spatial transcriptomics in exploring the complex molecular-genetic diversity of hypothalamic cell types, driven by recent high-throughput achievements.

Dorsal-Ventral Reinforcement Learning Network Connectivity and Incentive-Driven Changes in Exploration

Probabilistic reinforcement learning (RL) tasks assay how individuals make decisions under uncertainty. The use of internal models (model-based) or direct learning from experiences (model-free), and the degree of choice stochasticity across alternatives (i.e., exploration), can all be influenced by the state space of the decision-making task. There is considerable individual variation in the balance between model-based and model-free control during decision-making, and this balance is affected by incentive motivation. The effect of variable reward incentives on the arbitration between model-based and model-free learning remains understudied, and individual differences in neural signatures and cognitive traits that moderate the effect of reward on model-free/model-based control are unknown. Here we combined a two-stage decision-making task utilizing differing reward incentives with computational modeling, neuropsychological tests, and neuroimaging to address these questions. Results showed the prospect of greater reward decreased exploration of alternative options and increased the balance toward model-based learning. These behavioral effects were replicated across two independent datasets including both sexes. Individual differences in processing speed and analytical thinking style affected how reward altered the dependence on both systems. Using a systems neuroscience-inspired approach to resting-state functional connectivity, we found reduced exploration of the options during the first stage of our task under high relative to low incentives was predicted by increased cross-network coupling between ventral and dorsal RL circuitry. These findings suggest that integrity of functional connections between stimulus valuation (ventral) and action valuation (dorsal) RL networks is associated with changes in the balance between explore-exploit decisions under changing reward incentives.

Developmental changes in brain-wide fear memory networks

Memory retrieval involves coordinated activity across multiple brain regions. Yet how the organization of memory networks evolves throughout development remains poorly understood. In this study, we compared whole-brain functional networks that are active during contextual fear memory recall in infant, juvenile, and adult mice. Our analyses revealed that long-term memory networks change significantly across postnatal development. Infant fear memory networks are dense and heterogeneous, whereas adult networks are sparse and have a small-world topology. While hippocampal subregions were highly connected nodes at all ages, the cortex gained many functional connections across development. Different functional connections matured at different rates, but their developmental timing fell into three major categories: stepwise change between two ages, linear change across all ages, or inverted-U, with elevated functional connectivity in juveniles. Our work highlights how a subset of brain regions likely maintain important roles in fear memory encoding, but the functional connectivity of fear memory networks undergoes significant reorganization across development. Together, these results provide a blueprint for studying how correlated cellular activity in key areas distinctly regulates memory storage and retrieval across development.

Hypothalamic resting-state functional connectivity and self-evaluated aggression in young adults

INTRODUCTION

The hypothalamus plays a pivotal role in supporting motivated behaviors, including aggression. Previous work suggested differential roles of the medial (MH) and lateral hypothalamus (LH) in aggressive behaviors, but little is known about how their resting-state functional connectivity (rsFC) may relate to aggression in humans.

METHODS

We employed the data from the Human Connectome Project (HCP) and examined the rsFC's of LH and MH in 745 young adults (393 women). We also explored sex differences in the rsFC's. We processed the imaging data with published routines and evaluated the results of voxel-wise regression on the aggression score, as obtained from Achenbach Adult Self Report, with a corrected threshold.

RESULTS

The analyses revealed significant rsFC's between the LH and clusters in the middle temporal and occipital gyri across all subjects and in the thalamus for men, both in negative correlation with the aggression score. The slope test confirmed sex difference in the correlation between the LH-thalamus rsFC and aggression score. No significant rsFC was observed for the MH.

CONCLUSIONS

These findings suggest a role of the LH rsFC's and sex differences in the LH-thalamus rsFC in the manifestation of aggression in humans. The findings highlight the need for further research into sex-specific neural pathways in aggression and other related behavioral traits of importance to mental health.

Applications of Behavioral Economics and Neuroeconomics in Mental Health

Objective: The integration of behavioral economics and neuroeconomics into mental health offers innovative perspectives on understanding and addressing psychological disorders. This overview aims to synthesize current knowledge and explore the implications of these interdisciplinary approaches in the context of mental health. Method : In this narrative review, we summarized the current evidence regarding the applications of behavioral economics and neuroeconomics approaches in the field of mental health. Results: Behavioral economics and neuroeconomics provide valuable insights into the cognitive and emotional processes underlying mental health disorders, such as irrational decision-making, impulsivity, and self-control issues. Concepts such as loss aversion, temporal discounting, and framing effects inform the development of innovative interventions and policy initiatives. Behavioral economic interventions, including nudges, incentives, and commitment devices, show promise in promoting treatment adherence, reducing risky behaviors, and enhancing mental well-being. Neuroeconomics contributes by identifying neural markers predictive of treatment response and relapse risk, paving the way for personalized treatment approaches. Conclusion: The integration of behavioral economics and neuroeconomics into mental health research and practice holds significant potential for improving the understanding of psychological disorders and developing more effective, personalized interventions. Further research is needed to elucidate the mechanisms of action, optimize intervention strategies, and address ethical considerations associated with these approaches in mental health settings.

PM2.5 Exposure Triggers Hypothalamic Oxidative and ER Stress Leading to Depressive-like Behaviors in Rats

Epidemiological studies have linked fine dust pollution to depression, yet the underlying mechanisms remain unclear. Oxidative stress and endoplasmic reticulum (ER) stress are known contributors to depression, but their induction by particulate matter (PM), particularly PM2.5, in animal models has been limited. This study aimed to establish a rat model of PM2.5-induced depression-like behaviors and elucidate the underlying molecular mechanisms. Adult male Sprague-Dawley rats received daily intranasal PM2.5 for four weeks. Behavioral assessments, including the open field test (OFT), forced swim test (FST), and light-dark box (LDB) test, were conducted weekly. PM2.5-exposed rats displayed depressive-like behaviors, particularly in the FST, reflecting decreased motivation and learned helplessness. Molecular analyses indicated a specific increase in ER stress markers (CHOP, eIF2α, GRP78, and P16) and NOX4 in the hypothalamus, while other brain regions (striatum, cortex, and hippocampus) were not as pronounced. Additionally, PM2.5 exposure reduced tyrosine hydroxylase (TH) levels in the hypothalamus, suggesting impaired dopamine synthesis. These findings indicate that PM2.5 induces depressive-like behaviors via hypothalamic ER stress and oxidative stress pathways, leading to dopaminergic dysfunction. Targeting oxidative and ER stress within the hypothalamus may offer new therapeutic strategies for treating depression associated with environmental pollutants.

Modeling long-term nutritional behaviors using deep homeostatic reinforcement learning

The continual generation of behaviors that satisfy all conflicting demands that cannot be satisfied simultaneously, is a situation that is seen naturally in autonomous agents such as long-term operating household robots, and in animals in the natural world. Homeostatic reinforcement learning (homeostatic RL) is known as a bio-inspired framework that achieves such multiobjective control through behavioral optimization. Homeostatic RL achieves autonomous behavior optimization using only internal body information in complex environmental systems, including continuous motor control. However, it is still unknown whether the resulting behaviors actually have the similar long-term properties as real animals. To clarify this issue, this study focuses on the balancing of multiple nutrients in animal foraging as a situation in which such multiobjective control is achieved in animals in the natural world. We then focus on the nutritional geometry framework, which can quantitatively handle the long-term characteristics of foraging strategies for multiple nutrients in nutritional biology, and construct a similar verification environment to show experimentally that homeostatic RL agents exhibit long-term foraging characteristics seen in animals in nature. Furthermore, numerical simulation results show that the long-term foraging characteristics of the agent can be controlled by changing the weighting for the agent's multiobjective motivation. These results show that the long-term behavioral characteristics of homeostatic RL agents that perform behavioral emergence at the motor control level can be predicted and designed based on the internal dynamics of the body and the weighting of motivation, which change in real time.

The Organization of Anxiety Symptoms Along the Threat Imminence Continuum

Pathological anxiety is highly prevalent, impairing, and often chronic. Yet, despite considerable research, mechanistic understanding of anxiety and its translation to clinical practice remain limited. Here, we first highlight two foundational complications that contribute to this gap: a reliance on a phenomenology-driven definition of pathological anxiety in neurobiological mechanistic research, and a limited understanding of the chronicity of anxiety symptom expression. We then posit that anxiety symptoms may reflect aberrant expression of otherwise normative defensive responses. Accordingly, we propose that threat imminence, an organizing dimension for normative defensive responses observed across species, may be applied to organize and understand anxiety symptoms along a temporal dimension of expression. Empirical evidence linking distinct anxiety symptoms and the aberrant expression of imminence-dependent defensive responses is reviewed, alongside the neural mechanisms which may underpin these cognitive, physiological, and behavioral responses. Drawing from extensive translational and clinical research, we suggest that understanding anxiety symptoms through this neurobiologically-informed framework may begin to overcome the conceptual complications hindering advancement in mechanistic research and clinical interventions for pathological anxiety.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions, with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, the OFC leads the processing of symbolic RL in the ventral frontostriatal circuitry.

Transcriptomic diversity of amygdalar subdivisions across humans and nonhuman primates

The amygdaloid complex mediates learning, memory, and emotions. Understanding the cellular and anatomical features that are specialized in the amygdala of primates versus other vertebrates requires a systematic, anatomically-resolved molecular analysis of constituent cell populations. We analyzed five nuclear subdivisions of the primate amygdala with single-nucleus RNA sequencing in macaques, baboons, and humans to examine gene expression profiles for excitatory and inhibitory neurons and confirmed our results with single-molecule FISH analysis. We identified distinct subtypes of FOXP2 + interneurons in the intercalated cell masses and protein-kinase C-δ interneurons in the central nucleus. We also establish that glutamatergic, pyramidal-like neurons are transcriptionally specialized within the basal, lateral, or accessory basal nuclei. Understanding the molecular heterogeneity of anatomically-resolved amygdalar neuron types provides a cellular framework for improving existing models of how amygdalar neural circuits contribute to cognition and mental health in humans by using nonhuman primates as a translational bridge.

Nonlinear age-related differences in probabilistic learning in mice: A 5-armed bandit task study

This study explores the impact of aging on reinforcement learning in mice, focusing on changes in learning rates and behavioral strategies. A 5-armed bandit task (5-ABT) and a computational Q-learning model were used to evaluate the positive and negative learning rates and the inverse temperature across three age groups (3, 12, and 18 months). Results showed a significant decline in the negative learning rate of 18-month-old mice, which was not observed for the positive learning rate. This suggests that older mice maintain the ability to learn from successful experiences while decreasing the ability to learn from negative outcomes. We also observed a significant age-dependent variation in inverse temperature, reflecting a shift in action selection policy. Middle-aged mice (12 months) exhibited higher inverse temperature, indicating a higher reliance on previous rewarding experiences and reduced exploratory behaviors, when compared to both younger and older mice. This study provides new insights into aging research by demonstrating that there are age-related differences in specific components of reinforcement learning, which exhibit a non-linear pattern.

Hypothalamic connectivities and self-evaluated aggression in young adults

Introduction

The hypothalamus plays a pivotal role in supporting motivated behavior, including aggression. Previous work suggested differential roles of the medial hypothalamus (MH) and lateral hypothalamus (LH) in aggressive behaviors, but little is known about how their resting-state functional connectivity (rsFC) may relate to aggression in humans.

Methods

We employed the data from the Human Connectome Project (HCP) and examined the rsFC's of LH and MH in 745 young adults (393 women). We also explored sex differences in the rsFC's. We processed the imaging data with published routines and evaluated the results of voxel-wise regression on aggression score, as obtained from Achenbach Adult Self Report, with a corrected threshold.

Results

The analysis revealed significant rsFC between the LH and clusters in the middle temporal and occipital gyri across all subjects and in the thalamus for men, both in negative correlation with aggression score. Slope test confirmed sex differences in the correlation between LH-thalamus rsFC and aggression score. No significant rsFC was observed for MH.

Conclusions

These findings suggest a role of LH rsFCs and sex differences in LH-thalamus rsFC in the manifestation of aggression in humans. The findings highlight the need for further research into sex-specific neural pathways in aggression and other related behavioral traits of importance to mental health.

Drinking severity mediates the relationship between hypothalamic connectivity and rule-breaking/intrusive behavior differently in young women and men: an exploratory study

Background

The hypothalamus is a key hub of the neural circuits of motivated behavior. Alcohol misuse may lead to hypothalamic dysfunction. Here, we investigated how resting-state hypothalamic functional connectivities are altered in association with the severity of drinking and clinical comorbidities and how men and women differ in this association.

Methods

We employed the data of the Human Connectome Project. A total of 870 subjects were included in data analyses. The severity of alcohol use was quantified for individual subjects with the first principal component (PC1) identified from principal component analyses of all drinking measures. Rule-breaking and intrusive scores were evaluated with the Achenbach Adult Self-Report Scale. We performed a whole-brain regression of hypothalamic connectivities on drinking PC1 in all subjects and men/women separately and evaluated the results at a corrected threshold.

Results

Higher drinking PC1 was associated with greater hypothalamic connectivity with the paracentral lobule (PCL). Hypothalamic PCL connectivity was positively correlated with rule-breaking score in men (r=0.152, P=0.002) but not in women. In women but not men, hypothalamic connectivity with the left temporo-parietal junction (LTPJ) was negatively correlated with drinking PC1 (r=-0.246, P<0.001) and with intrusiveness score (r=-0.127, P=0.006). Mediation analyses showed that drinking PC1 mediated the relationship between hypothalamic PCL connectivity and rule-breaking score in men and between hypothalamic LTPJ connectivity and intrusiveness score bidirectionally in women.

Conclusions

We characterized sex-specific hypothalamic connectivities in link with the severity of alcohol misuse and its comorbidities. These findings extend the literature by elucidating the potential impact of problem drinking on the motivation circuits.

Convergent direct and indirect cortical streams shape avoidance decisions in mice via the midline thalamus

Current concepts of corticothalamic organization in the mammalian brain are mainly based on sensory systems, with less focus on circuits for higher-order cognitive functions. In sensory systems, first-order thalamic relays are driven by subcortical inputs and modulated by cortical feedback, while higher-order relays receive strong excitatory cortical inputs. The applicability of these principles beyond sensory systems is uncertain. We investigated mouse prefronto-thalamic projections to the midline thalamus, revealing distinct top-down control. Unlike sensory systems, this pathway relies on indirect modulation via the thalamic reticular nucleus (TRN). Specifically, the prelimbic area, which influences emotional and motivated behaviors, impacts instrumental avoidance responses through direct and indirect projections to the paraventricular thalamus. Both pathways promote defensive states, but the indirect pathway via the TRN is essential for organizing avoidance decisions through disinhibition. Our findings highlight intra-thalamic circuit dynamics that integrate cortical cognitive signals and their role in shaping complex behaviors.

Neural interactions in the human frontal cortex dissociate reward and punishment learning

How human prefrontal and insular regions interact while maximizing rewards and minimizing punishments is unknown. Capitalizing on human intracranial recordings, we demonstrate that the functional specificity toward reward or punishment learning is better disentangled by interactions compared to local representations. Prefrontal and insular cortices display non-selective neural populations to rewards and punishments. Non-selective responses, however, give rise to context-specific interareal interactions. We identify a reward subsystem with redundant interactions between the orbitofrontal and ventromedial prefrontal cortices, with a driving role of the latter. In addition, we find a punishment subsystem with redundant interactions between the insular and dorsolateral cortices, with a driving role of the insula. Finally, switching between reward and punishment learning is mediated by synergistic interactions between the two subsystems. These results provide a unifying explanation of distributed cortical representations and interactions supporting reward and punishment learning.

Distinct basal ganglia contributions to learning from implicit and explicit value signals in perceptual decision-making

Metacognitive evaluations of confidence provide an estimate of decision accuracy that could guide learning in the absence of explicit feedback. We examine how humans might learn from this implicit feedback in direct comparison with that of explicit feedback, using simultaneous EEG-fMRI. Participants performed a motion direction discrimination task where stimulus difficulty was increased to maintain performance, with intermixed explicit- and no-feedback trials. We isolate single-trial estimates of post-decision confidence using EEG decoding, and find these neural signatures re-emerge at the time of feedback together with separable signatures of explicit feedback. We identified these signatures of implicit versus explicit feedback along a dorsal-ventral gradient in the striatum, a finding uniquely enabled by an EEG-fMRI fusion. These two signals appear to integrate into an aggregate representation in the external globus pallidus, which could broadcast updates to improve cortical decision processing via the thalamus and insular cortex, irrespective of the source of feedback.

Computational Mechanisms Underlying Motivation to Earn Symbolic Reinforcers

Reinforcement learning is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g., money, points) that are later exchanged for primary reinforcers (e.g., food, drink). Although symbolic reinforcers are ubiquitous in our daily lives, widely used in laboratory tasks because they can be motivating, mechanisms by which they become motivating are less understood. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g., the current number of accumulated tokens, choice options, task epoch, trials since the last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to then correlate with the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n = 5 monkeys, three males and two females). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in the state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.

Changes in the dopaminergic circuitry and adult neurogenesis linked to reinforcement learning in corvids

The caudolateral nidopallium (NCL, an analog of the prefrontal cortex) is known to be involved in learning, memory, and discrimination in corvids (a songbird), whereas the involvement of other brain regions in these phenomena is not well explored. We used house crows (Corvus splendens) to explore the neural correlates of learning and decision-making by initially training them on a shape discrimination task followed by immunohistochemistry to study the immediate early gene expression (Arc), a dopaminoceptive neuronal marker (DARPP-32, Dopamine- and cAMP-regulated phosphoprotein, Mr 32 kDa) to understand the involvement of the reward pathway and an immature neuronal marker (DCX, doublecortin) to detect learning-induced changes in adult neurogenesis. We performed neuronal counts and neuronal tracing, followed by morphometric analyses. Our present results have demonstrated that besides NCL, other parts of the caudal nidopallium (NC), avian basal ganglia, and intriguingly, vocal control regions in house crows are involved in visual discrimination. We have also found that training on the visual discrimination task can be correlated with neurite pruning in mature dopaminoceptive neurons and immature DCX-positive neurons in the NC of house crows. Furthermore, there is an increase in the incorporation of new neurons throughout NC and the medial striatum which can also be linked to learning. For the first time, our results demonstrate that a combination of structural changes in mature and immature neurons and adult neurogenesis are linked to learning in corvids.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and the mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, OFC leads the process of symbolic reinforcement learning in the ventral frontostriatal circuitry.

The genetic architecture of the human hypothalamus and its involvement in neuropsychiatric behaviours and disorders

Despite its crucial role in the regulation of vital metabolic and neurological functions, the genetic architecture of the hypothalamus remains unknown. Here we conducted multivariate genome-wide association studies (GWAS) using hypothalamic imaging data from 32,956 individuals to uncover the genetic underpinnings of the hypothalamus and its involvement in neuropsychiatric traits. There were 23 significant loci associated with the whole hypothalamus and its subunits, with functional enrichment for genes involved in intracellular trafficking systems and metabolic processes of steroid-related compounds. The hypothalamus exhibited substantial genetic associations with limbic system structures and neuropsychiatric traits including chronotype, risky behaviour, cognition, satiety and sympathetic-parasympathetic activity. The strongest signal in the primary GWAS, the ADAMTS8 locus, was replicated in three independent datasets (N = 1,685-4,321) and was strengthened after meta-analysis. Exome-wide association analyses added evidence to the association for ADAMTS8, and Mendelian randomization showed lower ADAMTS8 expression with larger hypothalamic volumes. The current study advances our understanding of complex structure-function relationships of the hypothalamus and provides insights into the molecular mechanisms that underlie hypothalamic formation.

Neuromodulation for the treatment of Prader-Willi syndrome - A systematic review

Prader-Willi syndrome (PWS) is a complex, genetic disorder characterized by multisystem involvement, including hyperphagia, maladaptive behaviors and endocrinological derangements. Recent developments in advanced neuroimaging have led to a growing understanding of PWS as a neural circuit disorder, as well as subsequent interests in the application of neuromodulatory therapies. Various non-invasive and invasive device-based neuromodulation methods, including vagus nerve stimulation (VNS), transcranial direct current stimulation (tDCS), repetitive transcranial magnetic stimulation (rTMS), and deep brain stimulation (DBS) have all been reported to be potentially promising treatments for addressing the major symptoms of PWS. In this systematic literature review, we summarize the recent literature that investigated these therapies, discuss the underlying circuits which may underpin symptom manifestations, and cover future directions of the field. Through our comprehensive search, there were a total of 47 patients who had undergone device-based neuromodulation therapy for PWS. Two articles described VNS, 4 tDCS, 1 rTMS and 2 DBS, targeting different symptoms of PWS, including aberrant behavior, hyperphagia and weight. Multi-center and multi-country efforts will be required to advance the field given the low prevalence of PWS. Finally, given the potentially vulnerable population, neuroethical considerations and dialogue should guide the field.

Nuclei-specific hypothalamus networks predict a dimensional marker of stress in humans

The hypothalamus is part of the hypothalamic-pituitary-adrenal axis which activates stress responses through release of cortisol. It is a small but heterogeneous structure comprising multiple nuclei. In vivo human neuroimaging has rarely succeeded in recording signals from individual hypothalamus nuclei. Here we use human resting-state fMRI (n = 498) with high spatial resolution to examine relationships between the functional connectivity of specific hypothalamic nuclei and a dimensional marker of prolonged stress. First, we demonstrate that we can parcellate the human hypothalamus into seven nuclei in vivo. Using the functional connectivity between these nuclei and other subcortical structures including the amygdala, we significantly predict stress scores out-of-sample. Predictions use 0.0015% of all possible brain edges, are specific to stress, and improve when using nucleus-specific compared to whole-hypothalamus connectivity. Thus, stress relates to connectivity changes in precise and functionally meaningful subcortical networks, which may be exploited in future studies using interventions in stress disorders.

Dissociable encoding of motivated behavior by parallel thalamo-striatal projections

The successful pursuit of goals requires the coordinated execution and termination of actions that lead to positive outcomes. This process is thought to rely on motivational states that are guided by internal drivers, such as hunger or fear. However, the mechanisms by which the brain tracks motivational states to shape instrumental actions are not fully understood. The paraventricular nucleus of the thalamus (PVT) is a midline thalamic nucleus that shapes motivated behaviors via its projections to the nucleus accumbens (NAc)1-8 and monitors internal state via interoceptive inputs from the hypothalamus and brainstem3,9-14. Recent studies indicate that the PVT can be subdivided into two major neuronal subpopulations, namely PVTD2(+) and PVTD2(-), which differ in genetic identity, functionality, and anatomical connectivity to other brain regions, including the NAc4,15,16. In this study, we used fiber photometry to investigate the in vivo dynamics of these two distinct PVT neuronal types in mice performing a reward foraging-like behavioral task. We discovered that PVTD2(+) and PVTD2(-) neurons encode the execution and termination of goal-oriented actions, respectively. Furthermore, activity in the PVTD2(+) neuronal population mirrored motivation parameters such as vigor and satiety. Similarly, PVTD2(-) neurons, also mirrored some of these parameters but to a much lesser extent. Importantly, these features were largely preserved when activity in PVT projections to the NAc was selectively assessed. Collectively, our results highlight the existence of two parallel thalamo-striatal projections that participate in the dynamic regulation of goal pursuits and provide insight into the mechanisms by which the brain tracks motivational states to shape instrumental actions.

Computational mechanisms underlying motivation to earn symbolic reinforcers

Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that can later be exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are motivating, little is understood about the neural or computational mechanisms underlying the motivation to earn them. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to capture the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.

Understanding anxiety symptoms as aberrant defensive responding along the threat imminence continuum

Threat-anticipatory defensive responses have evolved to promote survival in a dynamic world. While inherently adaptive, aberrant expression of defensive responses to potential threat could manifest as pathological anxiety, which is prevalent, impairing, and associated with adverse outcomes. Extensive translational neuroscience research indicates that normative defensive responses are organized by threat imminence, such that distinct response patterns are observed in each phase of threat encounter and orchestrated by partially conserved neural circuitry. Anxiety symptoms, such as excessive and pervasive worry, physiological arousal, and avoidance behavior, may reflect aberrant expression of otherwise normative defensive responses, and therefore follow the same imminence-based organization. Here, empirical evidence linking aberrant expression of specific, imminence-dependent defensive responding to distinct anxiety symptoms is reviewed, and plausible contributing neural circuitry is highlighted. Drawing from translational and clinical research, the proposed framework informs our understanding of pathological anxiety by grounding anxiety symptoms in conserved psychobiological mechanisms. Potential implications for research and treatment are discussed.

Nutrient-Sensitive Reinforcement Learning in Monkeys

In reinforcement learning (RL), animals choose by assigning values to options and learn by updating these values from reward outcomes. This framework has been instrumental in identifying fundamental learning variables and their neuronal implementations. However, canonical RL models do not explain how reward values are constructed from biologically critical intrinsic reward components, such as nutrients. From an ecological perspective, animals should adapt their foraging choices in dynamic environments to acquire nutrients that are essential for survival. Here, to advance the biological and ecological validity of RL models, we investigated how (male) monkeys adapt their choices to obtain preferred nutrient rewards under varying reward probabilities. We found that the nutrient composition of rewards strongly influenced learning and choices. Preferences of the animals for specific nutrients (sugar, fat) affected how they adapted to changing reward probabilities; the history of recent rewards influenced choices of the monkeys more strongly if these rewards contained the their preferred nutrients (nutrient-specific reward history). The monkeys also chose preferred nutrients even when they were associated with lower reward probability. A nutrient-sensitive RL model captured these processes; it updated the values of individual sugar and fat components of expected rewards based on experience and integrated them into subjective values that explained the choices of the monkeys. Nutrient-specific reward prediction errors guided this value-updating process. Our results identify nutrients as important reward components that guide learning and choice by influencing the subjective value of choice options. Extending RL models with nutrient-value functions may enhance their biological validity and uncover nutrient-specific learning and decision variables.SIGNIFICANCE STATEMENT RL is an influential framework that formalizes how animals learn from experienced rewards. Although reward is a foundational concept in RL theory, canonical RL models cannot explain how learning depends on specific reward properties, such as nutrients. Intuitively, learning should be sensitive to the nutrient components of the reward to benefit health and survival. Here, we show that the nutrient (fat, sugar) composition of rewards affects how the monkeys choose and learn in an RL paradigm and that key learning variables including reward history and reward prediction error should be modified with nutrient-specific components to account for the choice behavior observed in the monkeys. By incorporating biologically critical nutrient rewards into the RL framework, our findings help advance the ecological validity of RL models.

Anatomical organization of forebrain circuits in the primate

The primate forebrain is a complex structure. Thousands of connections have been identified between cortical areas, and between cortical and sub-cortical areas. Previous work, however, has suggested that a number of principles can be used to reduce this complexity. Here, we integrate four principles that have been put forth previously, including a nested model of neocortical connectivity, gradients of connectivity between frontal cortical areas and the striatum and thalamus, shared patterns of sub-cortical connectivity between connected posterior and frontal cortical areas, and topographic organization of cortical-striatal-pallidal-thalamocortical circuits. We integrate these principles into a single model that accounts for a substantial amount of connectivity in the forebrain. We then suggest that studies in evolution and development can account for these four principles, by assuming that the ancestral vertebrate pallium was dominated by medial, hippocampal and ventral-lateral, pyriform areas, and at most a small dorsal pallium. The small dorsal pallium expanded massively in the lineage leading to primates. During this expansion, topological, adjacency relationships were maintained between pallial and sub-pallial areas. This maintained topology led to the connectivity gradients seen between cortex, striatum, pallidum, and thalamus.

The effects of androgen deprivation on working memory and quality of life in prostate cancer patients: The roles of hypothalamic connectivity

BACKGROUND

Androgen deprivation therapy (ADT) has been associated with adverse effects on the brain. ADT alters testosterone levels via its action on the hypothalamus-pituitary-gonadal axis and may influence hypothalamic functions. Given the wide regional connectivity of the hypothalamus and its role in regulating cognition and behavior, we assessed the effects of ADT on hypothalamic resting state functional connectivity (rsFC) and their cognitive and clinical correlates.

METHODS

In a prospective observational study, 22 men with nonmetastatic prostate cancer receiving ADT and 28 patients not receiving ADT (controls), matched in age, years of education, and Montreal Cognitive Assessment score, participated in N-back task and quality of life (QoL) assessments and brain imaging at baseline and at 6 months. Imaging data were processed with published routines and the results of a group by time flexible factorial analysis were evaluated at a corrected threshold.

RESULTS

ADT and control groups did not differ in N-back performance or QoL across time points. Relative to controls, patients receiving ADT showed significantly higher hypothalamus-right mid-cingulate cortex (MCC) and precentral gyrus (PCG) rsFC during follow-up versus baseline. Further, the changes in MCC and PCG rsFC were correlated positively with the change in QoL score and 0-back correct response rate, respectively, in patients with undergoing ADT.

CONCLUSION

Six-month ADT affects hypothalamic functional connectivity with brain regions critical to cognitive motor and affective functions. Elevated hypothalamic MCC and PCG connectivity likely serve to functionally compensate for the effects of ADT and sustain attention and overall QoL. The longer-term effects of ADT remain to be investigated.

Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System

To effectively behave within ever-changing environments, biological agents must learn and act at varying hierarchical levels such that a complex task may be broken down into more tractable subtasks. Hierarchical reinforcement learning (HRL) is a computational framework that provides an understanding of this process by combining sequential actions into one temporally extended unit called an option. However, there are still open questions within the HRL framework, including how options are formed and how HRL mechanisms might be realized within the brain. In this review, we propose that the existing human motor sequence literature can aid in understanding both of these questions. We give specific emphasis to visuomotor sequence learning tasks such as the discrete sequence production task and the M × N (M steps × N sets) task to understand how hierarchical learning and behavior manifest across sequential action tasks as well as how the dorsal cortical-subcortical circuitry could support this kind of behavior. This review highlights how motor chunks within a motor sequence can function as HRL options. Furthermore, we aim to merge findings from motor sequence literature with reinforcement learning perspectives to inform experimental design in each respective subfield.

Hypothalamic connectivities predict individual differences in ADT-elicited changes in working memory and quality of life in prostate cancer patients

Androgen deprivation therapy (ADT) has been associated with adverse effects on cognition. However, we currently lack understanding of the neurobiology and prognostic markers of these effects. Given that ADT acts via the hypothalamus-pituitary-gonadal axis, we assessed whether baseline hypothalamic resting state functional connectivity (rsFC) could predict changes in working memory and quality of life in prostate cancer patients following androgen deprivation. In a prospective observational study, 28 men with non-metastatic prostate cancer receiving ADT and 38 patients not receiving ADT (controls), matched in age, years of education and Montreal Cognitive Assessment score, participated in brain imaging at baseline, and N-back task and quality-of-life (QoL) assessments at baseline and at 6 months follow-up. Imaging data were processed with published routines and evaluated at a corrected threshold. ADT and control groups did not differ in N-back performance or QoL across time points. In ADT, the changes in 0-back correct response rate (follow-up-baseline) were correlated with baseline hypothalamus-precentral gyrus rsFC; the changes in 1-back correct response rate and reaction time were each correlated with hypothalamus-middle frontal gyrus and superior parietal lobule rsFC. The changes in physical well-being subscore of QoL were correlated with baseline hypothalamus-anterior cingulate and cuneus rsFC. The hypothalamus rsFCs predicted N-back and QoL change with an area under the receiver operating characteristic curve of 0.93 and 0.73, respectively. Baseline hypothalamus-frontoparietal and salience network rsFC's predict inter-subject variations in the changes in working-memory and QoL following 6 months of ADT. Whether and how hypothalamic rsFCs may predict the cognitive and QoL effects with longer-term ADT remain to be investigated.

The neurocomputational bases of explore-exploit decision-making

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

Pruning recurrent neural networks replicates adolescent changes in working memory and reinforcement learning

SignificanceAdolescence is a period during which there are important changes in behavior and the structure of the brain. In this manuscript, we use theoretical modeling to show how improvements in working memory and reinforcement learning that occur during adolescence can be explained by the reduction in synaptic connectivity in prefrontal cortex that occurs during a similar period. We train recurrent neural networks to solve working memory and reinforcement learning tasks and show that when we prune connectivity in these networks, they perform the tasks better. The improvement in task performance, however, can come at the cost of flexibility as the pruned networks are not able to learn some new tasks as well.

Hypothalamic syndrome

Hypothalamic syndrome (HS) is a rare disorder caused by disease-related and/or treatment-related injury to the hypothalamus, most commonly associated with rare, non-cancerous parasellar masses, such as craniopharyngiomas, germ cell tumours, gliomas, cysts of Rathke's pouch and Langerhans cell histiocytosis, as well as with genetic neurodevelopmental syndromes, such as Prader-Willi syndrome and septo-optic dysplasia. HS is characterized by intractable weight gain associated with severe morbid obesity, multiple endocrine abnormalities and memory impairment, attention deficit and reduced impulse control as well as increased risk of cardiovascular and metabolic disorders. Currently, there is no cure for this condition but treatments for general obesity are often used in patients with HS, including surgery, medication and counselling. However, these are mostly ineffective and no medications that are specifically approved for the treatment of HS are available. Specific challenges in HS are because the syndrome represents an adverse effect of different diseases, and that diagnostic criteria, aetiology, pathogenesis and management of HS are not completely defined.

Evolution of behavioural control from chordates to primates

This article outlines a hypothetical sequence of evolutionary innovations, along the lineage that produced humans, which extended behavioural control from simple feedback loops to sophisticated control of diverse species-typical actions. I begin with basic feedback mechanisms of ancient mobile animals and follow the major niche transitions from aquatic to terrestrial life, the retreat into nocturnality in early mammals, the transition to arboreal life and the return to diurnality. Along the way, I propose a sequence of elaboration and diversification of the behavioural repertoire and associated neuroanatomical substrates. This includes midbrain control of approach versus escape actions, telencephalic control of local versus long-range foraging, detection of affordances by the dorsal pallium, diversified control of nocturnal foraging in the mammalian neocortex and expansion of primate frontal, temporal and parietal cortex to support a wide variety of primate-specific behavioural strategies. The result is a proposed functional architecture consisting of parallel control systems, each dedicated to specifying the affordances for guiding particular species-typical actions, which compete against each other through a hierarchy of selection mechanisms. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

Self-tuition as an essential design feature of the brain

We are curious by nature, particularly when young. Evolution has endowed our brain with an inbuilt obligation to educate itself. In this perspectives article, we posit that self-tuition is an evolved principle of vertebrate brain design that is reflected in its basic architecture and critical for its normal development. Self-tuition involves coordination between functionally distinct components of the brain, with one set of areas motivating exploration that leads to the experiences that train another set. We review key hypothalamic and telencephalic structures involved in this interplay, including their anatomical connections and placement within the segmental architecture of conserved forebrain circuits. We discuss the nature of educative behaviours motivated by the hypothalamus, innate stimulus biases, the relationship to survival in early life, and mechanisms by which telencephalic areas gradually accumulate knowledge. We argue that this aspect of brain function is of paramount importance for systems neuroscience, as it confers neural specialization and allows animals to attain far more sophisticated behaviours than would be possible through genetic mechanisms alone. Self-tuition is of particular importance in humans and other primates, whose large brains and complex social cognition rely critically on experience-based learning during a protracted childhood period.

This article is part of the theme issue ‘Systems neuroscience through the lens of evolutionary theory’.

Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior

Goal-directed behavior requires identifying objects in the environment that can satisfy internal needs and executing actions to obtain those objects. The current study examines ventral and dorsal corticostriatal circuits that support complementary aspects of goal-directed behavior. We analyze activity from the amygdala, ventral striatum, orbitofrontal cortex, and lateral prefrontal cortex (LPFC) while monkeys perform a three-armed bandit task. Information about chosen stimuli and their value is primarily encoded in the amygdala, ventral striatum, and orbitofrontal cortex, while the spatial information is primarily encoded in the LPFC. Before the options are presented, information about the to-be-chosen stimulus is represented in the amygdala, ventral striatum, and orbitofrontal cortex; at the time of choice, the information is passed to the LPFC to direct a saccade. Thus, learned value information specifying behavioral goals is maintained throughout the ventral corticostriatal circuit, and it is routed through the dorsal circuit at the time actions are selected.

Prefrontal cortex interactions with the amygdala in primates

This review addresses functional interactions between the primate prefrontal cortex (PFC) and the amygdala, with emphasis on their contributions to behavior and cognition. The interplay between these two telencephalic structures contributes to adaptive behavior and to the evolutionary success of all primate species. In our species, dysfunction in this circuitry creates vulnerabilities to psychopathologies. Here, we describe amygdala-PFC contributions to behaviors that have direct relevance to Darwinian fitness: learned approach and avoidance, foraging, predator defense, and social signaling, which have in common the need for flexibility and sensitivity to specific and rapidly changing contexts. Examples include the prediction of positive outcomes, such as food availability, food desirability, and various social rewards, or of negative outcomes, such as threats of harm from predators or conspecifics. To promote fitness optimally, these stimulus-outcome associations need to be rapidly updated when an associative contingency changes or when the value of a predicted outcome changes. We review evidence from nonhuman primates implicating the PFC, the amygdala, and their functional interactions in these processes, with links to experimental work and clinical findings in humans where possible.

Reinforcement-learning in fronto-striatal circuits

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

Organization of Parieto-Prefrontal and Temporo-Prefrontal Networks in the Macaque

The connectivity among architectonically defined areas of the frontal, parietal, and temporal cortex of the macaque has been extensively mapped through tract tracing methods. To investigate the statistical organization underlying this connectivity, and identify its underlying architecture, we performed a hierarchical cluster analysis on 69 cortical areas based on their anatomically defined inputs. We identified 10 frontal, 4 parietal, and 5 temporal hierarchically related sets of areas (clusters), defined by unique sets of inputs and typically composed of anatomically contiguous areas. Across cortex, clusters that share functional properties were linked by dominant information processing circuits in a topographically organized manner that reflects the organization of the main fiber bundles in the cortex. This led to a dorsal-ventral subdivision of the frontal cortex, where dorsal and ventral clusters showed privileged connectivity with parietal and temporal areas, respectively. Ventrally, temporo-frontal circuits encode information to discriminate objects in the environment, their value, emotional properties, and functions such as memory and spatial navigation. Dorsal parieto-frontal circuits encode information for selecting, generating, and monitoring appropriate actions based on visual-spatial and somatosensory information. This organization may reflect evolutionary antecedents, in which the vertebrate pallium, which is the ancestral cortex, was defined by a ventral and lateral olfactory region and a medial hippocampal region.

A prefrontal network integrates preferences for advance information about uncertain rewards and punishments

Humans and animals can be strongly motivated to seek information to resolve uncertainty about rewards and punishments. In particular, despite its clinical and societal relevance, very little is known about information seeking about punishments. We show that attitudes toward information about punishments and rewards are distinct and separable at both behavioral and neuronal levels. We demonstrate the existence of prefrontal neuronal populations that anticipate opportunities to gain information in a relatively valence-specific manner, separately anticipating information about either punishments or rewards. These neurons are located in anatomically interconnected subregions of anterior cingulate cortex (ACC) and ventrolateral prefrontal cortex (vlPFC) in area 12o/47. Unlike ACC, vlPFC also contains a population of neurons that integrate attitudes toward both reward and punishment information, to encode the overall preference for information in a bivalent manner. This cortical network is well suited to mediate information seeking by integrating the desire to resolve uncertainty about multiple, distinct motivational outcomes.

Effects of Amygdala Lesions on Object-Based Versus Action-Based Learning in Macaques

The neural systems that underlie reinforcement learning (RL) allow animals to adapt to changes in their environment. In the present study, we examined the hypothesis that the amygdala would have a preferential role in learning the values of visual objects. We compared a group of monkeys (Macaca mulatta) with amygdala lesions to a group of unoperated controls on a two-armed bandit reversal learning task. The task had two conditions. In the What condition, the animals had to learn to select a visual object, independent of its location. And in the Where condition, the animals had to learn to saccade to a location, independent of the object at the location. In both conditions choice-outcome mappings reversed in the middle of the block. We found that monkeys with amygdala lesions had learning deficits in both conditions. Monkeys with amygdala lesions did not have deficits in learning to reverse choice-outcome mappings. Rather, amygdala lesions caused the monkeys to become overly sensitive to negative feedback which impaired their ability to consistently select the more highly valued action or object. These results imply that the amygdala is generally necessary for RL.