Distinct Functions of the Primate Putamen Direct and Indirect Pathways in Adaptive Outcome-Based Action Selection

Purposive decision-making task in mice using touchscreen operant apparatus

Purposive decision-making, based on sensory input and memory, is a component of executive functioning. Evaluating executive functioning is crucial for understanding neuropsychiatric disorders and brain injuries. However, there's a lack of mouse tests for this purpose. To address this, we developed a novel touchscreen task to assess purposive decision-making in mice. In the present task, the mice had to touch the correct window (left or right), with a visual stimulus as a cue for decision-making. The mice gradually acquired a relationship between the visual stimuli and the action they should take. Each mouse made the correct choice more than 80% of the time based on the visual cue and memory and knowledge of themselves. We could clearly determine when the mice saw the visual cue. The present task offers a valuable tool for investigating the neural mechanisms behind decision-making.

Dynamic Foraging Behavior Performance Is Not Affected by Scn2a Haploinsufficiency

Dysfunction in the gene SCN2A, which encodes the voltage-gated sodium channel Nav1.2, is strongly associated with neurodevelopmental disorders including autism spectrum disorder and intellectual disability (ASD/ID). This dysfunction typically manifests in these disorders as a haploinsufficiency, where loss of one copy of a gene cannot be compensated for by the other allele. Scn2a haploinsufficiency affects a range of cells and circuits across the brain, including associative neocortical circuits that are important for cognitive flexibility and decision-making behaviors. Here, we tested whether Scn2a haploinsufficiency has any effect on a dynamic foraging task that engages such circuits. Scn2a +/- mice and wild-type (WT) littermates were trained on a choice behavior where the probability of reward between two options varied dynamically across trials and where the location of the high reward underwent uncued reversals. Despite impairments in Scn2a-related neuronal excitability, we found that both male and female Scn2a +/- mice performed these tasks as well as wild-type littermates, with no behavioral difference across genotypes in learning or performance parameters. Varying the number of trials between reversals or probabilities of receiving reward did not result in an observable behavioral difference, either. These data suggest that, despite heterozygous loss of Scn2a, mice can perform relatively complex foraging tasks that make use of higher-order neuronal circuits.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Resting-State Connectivity and Response to Psychotherapy Treatment in Adolescents and Adults With OCD: A Randomized Clinical Trial

OBJECTIVE

Cortical-subcortical hyperconnectivity related to affective-behavioral integration and cortical network hypoconnectivity related to cognitive control have been demonstrated in obsessive-compulsive disorder (OCD); the study objective was to examine whether these connectivity patterns predict treatment response.

METHODS

Adolescents (ages 12-17) and adults (ages 24-45) were randomly assigned to 12 sessions of exposure and response prevention (ERP) or stress management therapy (SMT), an active control. Before treatment, resting-state connectivity of ventromedial prefrontal cortical (vmPFC), cingulo-opercular, frontoparietal, and subcortical regions was assessed with functional MRI. OCD severity was assessed with the Yale-Brown Obsessive Compulsive Scale before, during, and after treatment. Usable fMRI and longitudinal symptom data were obtained from 116 patients (68 female; 54 adolescents; 60 medicated).

RESULTS

ERP produced greater decreases in symptom scores than SMT. ERP was selectively associated with less vmPFC-subcortical (caudate and thalamus) connectivity in both age groups and primarily in unmedicated participants. Greater symptom improvement with both ERP and SMT was associated with greater cognitive-control (cingulo-opercular and frontoparietal) and subcortical (putamen) connectivity across age groups. Developmental specificity was observed across ERP and SMT treatments, such that greater improvements with ERP than SMT were associated with greater frontoparietal-subcortical (nucleus accumbens) connectivity in adolescents but greater connectivity between frontoparietal regions in adults. Comparison of response-predictive connections revealed no significant differences compared with a matched healthy control group.

CONCLUSIONS

The results suggest that less vmPFC-subcortical connectivity related to affect-influenced behavior may be important for ERP engagement, whereas greater cognitive-control and motor circuit connectivity may generally facilitate response to psychotherapy. Finally, neural predictors of treatment response may differ by age.

Conditional Regulation of Blood Pressure in Response to Emotional Stimuli by the Central Nucleus of the Amygdala in Rats

Humans and animals can determine whether a situation is favorable to them and act accordingly. For this, the autonomic tuning of the cardiovascular system to supply energy to active skeletal muscles through the circulatory system is as important as motor control. However, how the autonomic cardiovascular responses are regulated in dynamically changing environments and the neuronal mechanisms underlying these responses remain unclear. To resolve these issues, we recorded the blood pressure and heart rate of head-restrained rats during dynamically changing appetitive and aversive classical conditioning tasks. The rats displayed various associations between conditioned stimuli and unconditioned stimuli in appetitive (sucrose water), neutral (no outcome), and aversive (air puff) blocks. The blood pressure and heart rate in the appetitive block gradually increased in response to the reward-predicting cue and the response to the actual reward vigorously increased. The reward-predictive response was significantly higher than the responses obtained in the neutral and aversive condition blocks. To investigate whether the reward-predictive pressor response was caused by orofacial movements such as anticipatory licking behavior, we separately analyzed high- and low-licking trials. The conditioned pressor response was observed even in trials with low-licking behaviors. Blood pressure and heart rate responses to the air puff-predicting cue in the aversive block were not significantly different from the responses in the neutral block. The conditioned blood pressure response rapidly changed with condition block switching. Furthermore, to examine the contribution of the amygdala as an emotion center to these conditioned responses, we bilaterally microinjected a GABA_A receptor agonist, muscimol, into the central nucleus of the amygdala. Pharmacological inactivation of the central nucleus of the amygdala significantly decreased the reward-predictive pressor responses. These results suggest that the blood pressure is adaptively and rapidly regulated by emotional conditioned stimuli and that the central nucleus of the amygdala participates in regulating the pressor response in dynamically changing situations.

The asymmetric learning rates of murine exploratory behavior in sparse reward environments

Goal-oriented behaviors of animals can be modeled by reinforcement learning algorithms. Such algorithms predict future outcomes of selected actions utilizing action values and updating those values in response to the positive and negative outcomes. In many models of animal behavior, the action values are updated symmetrically based on a common learning rate, that is, in the same way for both positive and negative outcomes. However, animals in environments with scarce rewards may have uneven learning rates. To investigate the asymmetry in learning rates in reward and non-reward, we analyzed the exploration behavior of mice in five-armed bandit tasks using a Q-learning model with differential learning rates for positive and negative outcomes. The positive learning rate was significantly higher in a scarce reward environment than in a rich reward environment, and conversely, the negative learning rate was significantly lower in the scarce environment. The positive to negative learning rate ratio was about 10 in the scarce environment and about 2 in the rich environment. This result suggests that when the reward probability was low, the mice tend to ignore failures and exploit the rare rewards. Computational modeling analysis revealed that the increased learning rates ratio could cause an overestimation of and perseveration on rare-rewarding events, increasing total reward acquisition in the scarce environment but disadvantaging impartial exploration.

Impact of Reck expression and promoter activity in neuronal in vitro differentiation

Reck (REversion-inducing Cysteine-rich protein with Kazal motifs) tumor suppressor gene encodes a multifunctional glycoprotein which inhibits the activity of several matrix metalloproteinases (MMPs), and has the ability to modulate the Notch and canonical Wnt pathways. Reck-deficient neuro-progenitor cells undergo precocious differentiation; however, modulation of Reck expression during progression of the neuronal differentiation process is yet to be characterized. In the present study, we demonstrate that Reck expression levels are increased during in vitro neuronal differentiation of PC12 pheochromocytoma cells and P19 murine teratocarcinoma cells and characterize mouse Reck promoter activity during this process. Increased Reck promoter activity was found upon induction of differentiation in PC12 cells, in accordance with its increased mRNA expression levels in mouse in vitro models. Interestingly, Reck overexpression, prior to the beginning of the differentiation protocol, led to diminished efficiency of the neuronal differentiation process. Taken together, our findings suggest that increased Reck expression at early stages of differentiation diminishes the number of neuron-like cells, which are positive for the beta-3 tubulin marker. Our data highlight the importance of Reck expression evaluation to optimize in vitro neuronal differentiation protocols.

Dynamic decision making and value computations in medial frontal cortex

Dynamic decision making requires an intact medial frontal cortex. Recent work has combined theory and single-neuron measurements in frontal cortex to advance models of decision making. We review behavioral tasks that have been used to study dynamic decision making and algorithmic models of these tasks using reinforcement learning theory. We discuss studies linking neurophysiology and quantitative decision variables. We conclude with hypotheses about the role of other cortical and subcortical structures in dynamic decision making, including ascending neuromodulatory systems.

Topographic distinction in long-term value signals between presumed dopamine neurons and presumed striatal projection neurons in behaving monkeys

Nigrostriatal dopamine (DA) projections are anatomically organized along the dorsolateral-ventromedial axis, conveying long-term value signals to the striatum for shaping actions toward multiple future rewards. The present study examines whether the topographic organization of long-term value signals are observed upon activity of presumed DA neurons and presumed striatal projection neurons (phasically active neurons, PANs), as predicted based on anatomical literature. Our results indicate that DA neurons in the dorsolateral midbrain encode long-term value signals on a short timescale, while ventromedial midbrain DA neurons encode such signals on a relatively longer timescale. Activity of the PANs in the dorsal striatum is more heterogeneous for encoding long-term values, although significant differences in long-term value signals were observed between the caudate nucleus and putamen. These findings suggest that topographic DA signals for long-term values are not simply transferred to striatal neurons, possibly due to the contribution of other projections to the striatum.

Stable Representations of Decision Variables for Flexible Behavior

Decisions occur in dynamic environments. In the framework of reinforcement learning, the probability of performing an action is influenced by decision variables. Discrepancies between predicted and obtained rewards (reward prediction errors) update these variables, but they are otherwise stable between decisions. Although reward prediction errors have been mapped to midbrain dopamine neurons, it is unclear how the brain represents decision variables themselves. We trained mice on a dynamic foraging task in which they chose between alternatives that delivered reward with changing probabilities. Neurons in the medial prefrontal cortex, including projections to the dorsomedial striatum, maintained persistent firing rate changes over long timescales. These changes stably represented relative action values (to bias choices) and total action values (to bias response times) with slow decay. In contrast, decision variables were weakly represented in the anterolateral motor cortex, a region necessary for generating choices. Thus, we define a stable neural mechanism to drive flexible behavior.

Monitoring and Updating of Action Selection for Goal-Directed Behavior through the Striatal Direct and Indirect Pathways

The basal ganglia play key roles in adaptive behaviors guided by reward and punishment. However, despite accumulating knowledge, few studies have tested how heterogeneous signals in the basal ganglia are organized and coordinated for goal-directed behavior. In this study, we investigated neuronal signals of the direct and indirect pathways of the basal ganglia as rats performed a lever push/pull task for a probabilistic reward. In the dorsomedial striatum, we found that optogenetically and electrophysiologically identified direct pathway neurons encoded reward outcomes, whereas indirect pathway neurons encoded no-reward outcome and next-action selection. Outcome coding occurred in association with the chosen action. In support of pathway-specific neuronal coding, light activation induced a bias on repeat selection of the same action in the direct pathway, but on switch selection in the indirect pathway. Our data reveal the mechanisms underlying monitoring and updating of action selection for goal-directed behavior through basal ganglia circuits.