Complementary Control over Habits and Behavioral Vigor by Phasic Activity in the Dorsolateral Striatum

A role for the dorsolateral striatum in prospective action control

The dorsolateral striatum (DLS) is important for performing actions persistently, even when it becomes suboptimal, reflecting a function that is reflexive and habitual. However, there are also ways in which persistent behaviors can result from a more prospective, planning mode of behavior. To help tease apart these possibilities for DLS function, we trained animals to perform a lever press for reward and then inhibited the DLS in key test phases: as the task shifted from a 1-press to a 3-press rule (upshift), as the task was maintained, as the task shifted back to the one-press rule (downshift), and when rewards came independent of pressing. During DLS inhibition, animals always favored their initially learned strategy to press just once, particularly so during the free-reward period. DLS inhibition surprisingly changed performance speed bidirectionally depending on the task shifts. DLS inhibition thus encouraged habitual behavior, suggesting it could normally help adapt to changing conditions.

Making and breaking habits: Revisiting the definitions and behavioral factors that influence habits in animals

Habits have garnered significant interest in studies of associative learning and maladaptive behavior. However, habit research has faced scrutiny and challenges related to the definitions and methods. Differences in the conceptualizations of habits between animal and human studies create difficulties for translational research. Here, we review the definitions and commonly used methods for studying habits in animals and humans and discuss potential alternative ways to assess habits, such as automaticity. To better understand habits, we then focus on the behavioral factors that have been shown to make or break habits in animals, as well as potential mechanisms underlying the influence of these factors. We discuss the evidence that habitual and goal-directed systems learn in parallel and that they seem to interact in competitive and cooperative manners. Finally, we draw parallels between habitual responding and compulsive drug seeking in animals to delineate the similarities and differences in these behaviors.

Characterization of striatal dopamine projections across striatal subregions in behavioral flexibility

Behavioural flexibility is key to survival in a dynamic environmentWhile flexible, goal-directed behaviours are initially dependent on dorsomedial striatum, they become dependent on lateral striatum as behaviours become inflexible. Similarly, lesions of dopamine terminals in lateral striatum disrupt the development of inflexible habits. This work suggests that dopamine release in lateral striatum may drive inflexible behaviours, though few studies have investigated a causative role of subpopulations of striatal dopamine terminals in reversal learning, a measure of flexibility. Here, we performed two optogenetic experiments to activate dopamine terminals in dorsomedial (DMS), dorsolateral (DLS) or ventral (nucleus accumbens [NAc]) striatum in DAT-Cre mice that expressed channelrhodopsin-2 via viral injection (Experiment I) or through transgenic breeding with an Ai32 reporter line (Experiment II) to determine how specific dopamine subpopulations impact reversal learning. Mice performed a reversal task in which they self-stimulated DMS, DLS, or NAc dopamine terminals by pressing one of two levers before action-outcome lever contingencies were reversed. Largely consistent with presumed ventromedial/lateral striatal function, we found that mice self-stimulating medial dopamine terminals reversed lever preference following contingency reversal, while mice self-stimulating NAc showed parial flexibility, and DLS self-stimulation resulted in impaired reversal. Impairments in DLS mice were characterized by more regressive errors and reliance on lose-stay strategies following reversal, as well as reduced within-session learning, suggesting reward insensitivity and overreliance on previously learned actions. This study supports a model of striatal function in which DMS and ventral dopamine facilitate goal-directed responding, and DLS dopamine supports more inflexible responding.

Inhibitory Pedunculopontine Neurons Gate Dopamine-Mediated Motor Actions of Unsigned Valence

BACKGROUND

The pedunculopontine nucleus (PPN) maintains a bidirectional connectivity with the basal ganglia that supports their shared roles in the selection and execution of motor actions. Previous studies identified a role for PPN neurons in goal-directed behavior, but the cellular substrates underlying this function have not been elucidated. We recently revealed the existence of a monosynaptic GABAergic input from the PPN that inhibits dopamine neurons of the substantia nigra. Activation of this pathway interferes with the execution of learned motor sequences when the actions are rewarded, even though the inhibition of dopamine neurons did not shift the value of the action, hence suggesting executive control over the gating of behavior.

OBJECTIVE

To test the attributes of the inhibition of dopamine neurons by the PPN in the context of goal-directed behavior regardless of whether the outcome is positive or negative.

METHODS

We delivered optogenetic stimulation to PPN GABAergic axon terminals in the substantia nigra during a battery of behavioral tasks with positive and negative valence.

RESULTS

Inhibition of dopamine neurons by PPN optogenetic activation during an appetitive task impaired the initiation and overall execution of the behavioral sequence without affecting the consumption of reward. During an active avoidance task, the same activation impaired the ability of mice to avoid a foot shock, but their escape response was unaffected. In addition, responses to potential threats were significantly attenuated.

CONCLUSION

Our results show that PPN GABAergic neurons modulate learned, goal-directed behavior of unsigned valence without affecting overall motor behavior.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Dorsomedial striatum, but not dorsolateral striatum, is necessary for rat category learning

Categorization is an adaptive cognitive function that allows us to generalize knowledge to novel situations. Converging evidence from neuropsychological, neuroimaging, and neurophysiological studies suggest that categorization is mediated by the basal ganglia; however, there is debate regarding the necessity of each subregion of the basal ganglia and their respective functions. The current experiment examined the roles of the dorsomedial striatum (DMS; homologous to the head of the caudate nucleus) and dorsolateral striatum (DLS; homologous to the body and tail of the caudate nucleus) in category learning by combining selective lesions with computational modeling. Using a touchscreen apparatus, rats were trained to categorize distributions of visual stimuli that varied along two continuous dimensions (i.e., spatial frequency and orientation). The tasks either required attention to one stimulus dimension (spatial frequency or orientation; 1D tasks) or both stimulus dimensions (spatial frequency and orientation; 2D tasks). Rats with NMDA lesions of the DMS were impaired on both the 1D tasks and 2D tasks, whereas rats with DLS lesions showed no impairments. The lesions did not affect performance on a discrimination task that had the same trial structure as the categorization tasks, suggesting that the category impairments effected processes relevant to categorization. Model simulations were conducted using a neural network to assess the effect of the DMS lesions on category learning. Together, the results suggest that the DMS is critical to map category representations to appropriate behavioral responses, whereas the DLS is not necessary for categorization.

Task history dictates how the dorsolateral striatum controls action strategy and vigor

The dorsolateral striatum (DLS) is linked to the learning and honing of action routines. However, the DLS is also important for performing behaviors that have been successful in the past. The learning function can be thought of as prospective, helping to plan ongoing actions to be efficient and often optimal. The performance function is more retrospective, helping the animal continue to behave in a way that had worked previously. How the DLS manages this all is curious. What happens when a learned behavior becomes sub-optimal due to environment changes. In this case, the prospective function of the DLS would cause animals to (adaptively) learn and plan more optimal actions. In contrast, the retrospective function would cause animals to (maladaptively) favor the old behavior. Here we find that, during a change in learned task rules, DLS inhibition causes animals to adjust less rapidly to the new task (and to behave less vigorously) in a 'maladaptive' way. Yet, when the task is changed back to the initially learned rules, DLS inhibition instead causes a rapid and vigorous adjustment of behavior in an 'adaptive' way. These results show that inhibiting the DLS biases behavior towards initially acquired strategies, implying a more retrospective outlook in action selection when the DLS is offline. Thus, an active DLS could encourage planning and learning action routines more prospectively. Moreover, the DLS control over behavior can appear to be either advantageous/flexible or disadvantageous/inflexible depending on task context, and its control over vigor can change depending on task context.

Significant Statement

Basal ganglia networks aid behavioral learning (a prospective planning function) but also favor the use of old behaviors (a retrospective performance function), making it unclear what happens when learned behaviors become suboptimal. Here we inhibit the dorsolateral striatum (DLS) as animals encounter a change in task rules, and again when they shift back to those learned task rules. DLS inhibition reduces adjustment to new task rules (and reduces behavioral vigor), but it increases adjustment back to the initially learned task rules later (and increases vigor). Thus, in both cases, DLS inhibition favored the use of the initially learned behavioral strategy, which could appear either maladaptive or adaptive. We suggest that the DLS might promote a prospective orientation of action control.

Lack of action monitoring as a prerequisite for habitual and chunked behavior: Behavioral and neural correlates

We previously reported the rapid development of habitual behavior in a discrete-trials instrumental task in which lever insertion and retraction act as reward-predictive cues delineating sequence execution. Here we asked whether lever cues or performance variables reflective of skill and automaticity might account for habitual behavior in male rats. Behavior in the discrete-trials habit-promoting task was compared with two task variants lacking the sequence-delineating cues of lever extension and retraction. We find that behavior is under goal-directed control in absence of sequence-delineating cues but not in their presence, and that skilled performance does not predict goal-directed vs. habitual behavior. Neural activity recordings revealed an engagement of dorsolateral striatum and a disengagement of dorsomedial striatum during the sequence execution of the habit-promoting task, specifically. Together, these results indicate that sequence delineation cues promote habit and differential engagement of striatal subregions during instrumental responding, a pattern that may reflect cue-elicited behavioral chunking.

Medial septum activation improves strategy switching once strategies are well-learned via bidirectional regulation of dopamine neuron population activity

Strategy switching is a form of cognitive flexibility that requires inhibiting a previously successful strategy and switching to a new strategy of a different categorical modality. It is dependent on dopamine (DA) receptor activation and release in ventral striatum and prefrontal cortex, two primary targets of ventral tegmental area (VTA) DA projections. Although the circuitry that underlies strategy switching early in learning has been studied, few studies have examined it after extended discrimination training. This may be important as DA activity and release patterns change across learning, with several studies demonstrating a critical role for substantia nigra pars compacta (SNc) DA activity and release once behaviors are well-learned. We have demonstrated that medial septum (MS) activation simultaneously increased VTA and decreased SNc DA population activity, as well as improved reversal learning via these actions on DA activity. We hypothesized that MS activation would improve strategy switching both early in learning and after extended training through its ability to increase VTA DA population activity and decrease SNc DA population activity, respectively. We chemogenetically activated the MS of male and female rats and measured their performance on an operant-based strategy switching task following 1, 10, or 15 days of discrimination training. Contrary to our hypothesis, MS activation did not affect strategy switching after 1 day of discrimination training. MS activation improved strategy switching after 10 days of training, but only in females. MS activation improved strategy switching in both sexes after 15 days of training. Infusion of bicuculline into the ventral subiculum (vSub) inhibited the MS-mediated decrease in SNc DA population activity and attenuated the improvement in strategy switching. Intra-vSub infusion of scopolamine inhibited the MS-mediated increase in VTA DA population activity but did not affect the improvement in strategy switching. Intra-vSub infusion of both bicuculline and scopolamine inhibited the MS-mediated effects on DA population activity in both the SNc and VTA and completely prevented the improvement in strategy switching. These data indicate that MS activation improves strategy switching once the original strategy has been sufficiently well-learned, and that this may occur via the MS's regulation of DA neuron responsivity.

Interactions Between Experience, Genotype and Sex in the Development of Individual Coping Strategies

Coping strategies, the first line of defense against adversities, develop through experience. There is consistent evidence that both genotype and sex contribute to the development of dysfunctional coping, leading to maladaptive outcomes of adverse experiences or to adaptive coping that fosters rapid recovery even from severe stress. However, how these factors interact to influence the development of individual coping strategies is just starting to be investigated. In the following review, we will consider evidence that experience, sex, and genotype influence the brain circuits and neurobiological processes involved in coping with adversities and discuss recent results pointing to the specific effects of the interaction between early experiences, genotype, and stress in the development of functional and dysfunctional coping styles.

Micro-habits for life-long learning

Radiology is a demanding career that requires a thorough understanding of evolving knowledge in both medical imaging and technology. Competing interests such as-familial obligations, clinical practice, committee meetings, and research projects-often leave little time for self-care and regular review of current medical literature. Healthy habits can be difficult to maintain, but micro-habits are more manageable and their benefits compound over time. Based on the book, Atomic Habits by James Clear, we discuss a micro-habit toolkit which includes: a two-minute rule, habit-stacking, environmental cues, task prioritization/automatization, habit tracking, and accountability. We offer practical suggestions for radiologists to incorporate this toolkit into their daily lives to become healthy life-long learners.

Dorsolateral Striatal Task-initiation Bursts Represent Past Experiences More than Future Action Plans

The dorsolateral striatum (DLS) is involved in learning and executing procedural actions. Cell ensembles in the DLS, but not the dorsomedial striatum (DMS), exhibit a burst of firing at the start of a well-learned action sequence ("task-bracketing"). However, it is currently unclear what information is contained in these bursts. Some theories suggest that these bursts should represent the procedural action sequence itself (that they should be about future action chains), whereas others suggest that they should contain representations of the current state of the world, taking into account primarily past information. In addition, the DLS local field potential shows transient bursts of power in the 50 Hz range (γ50) around the time a learned action sequence is initiated. However, it is currently unknown how bursts of activity in DLS cell ensembles and bursts of γ50 power in the DLS local field potential are related to each other. We found that DLS bursts at lap initiation in rats represented recently experienced reward locations more than future procedural actions, indicating that task-initiation DLS bursts contain primarily retrospective, rather than prospective, information to guide procedural actions. Furthermore, representations of past reward locations increased during periods of increased γ50 power in the DLS. There was no evidence of task-initiation bursts, increased γ50 power, or retrospective reward location information in the neighboring dorsomedial striatum. These data support a role for the DLS in model-free theories of procedural decision-making over planned action-chain theories, suggesting that procedural actions derive from representations of the current and recent past.SIGNIFICANCE STATEMENT While it is well-established that the dorsolateral striatum (DLS) plays a critical role in procedural decision-making, open questions remain about the kinds of representations contained in DLS ensemble activity that guide procedural actions. We found that DLS, but not DMS, cell ensembles contained nonlocal representations of past reward locations that appear moments before task-initiation DLS bursts. These retrospective representations were temporally linked to a rise in γ50 power that also preceded the characteristic DLS burst at task-initiation. These results support models of procedural decision-making based on associations between available actions and the current state of the world over models based on planning over action-chains.

Mechanisms of Network Interactions for Flexible Cortico-Basal Ganglia-Mediated Action Control

In humans, finely tuned γ synchronization (60-90 Hz) rapidly appears at movement onset in a motor control network involving primary motor cortex, the basal ganglia and motor thalamus. Yet the functional consequences of brief movement-related synchronization are still unclear. Distinct synchronization phenomena have also been linked to different forms of motor inhibition, including relaxing antagonist muscles, rapid movement interruption and stabilizing network dynamics for sustained contractions. Here, I will introduce detailed hypotheses about how intrasite and intersite synchronization could interact with firing rate changes in different parts of the network to enable flexible action control. The here proposed cause-and-effect relationships shine a spotlight on potential key mechanisms of cortico-basal ganglia-thalamo-cortical (CBGTC) communication. Confirming or revising these hypotheses will be critical in understanding the neuronal basis of flexible movement initiation, invigoration and inhibition. Ultimately, the study of more complex cognitive phenomena will also become more tractable once we understand the neuronal mechanisms underlying behavioral readouts.

Spatially restricted inhibition of cholinergic interneurons in the dorsolateral striatum encourages behavioral exploration

When pursuing desirable outcomes, one must make the decision between exploring possible actions to obtain those outcomes and exploiting known strategies to maximize efficiency. The dorsolateral striatum (DLS) has been extensively studied with respect to how actions can develop into habits and has also been implicated as an area involved in governing exploitative behavior. Surprisingly, prior work has shown that DLS cholinergic interneurons (ChIs) are not involved in the canonical habit formation function ascribed to the DLS but are instead modulators of behavioral flexibility after initial learning. To further probe this, we evaluated the role of DLS ChIs in behavioral exploration during a brief instrumental training experiment. Through designer receptors exclusively activated by designer drugs (DREADDs) in ChAT-Cre rats, ChIs in the DLS were inhibited during specific phases of the experiment: instrumental training, free-reward delivery, at both times, or never. Without ChI activity during instrumental training, animals biased their responding toward an "optimal" strategy while continuing to work efficiently. This effect was observed again when contingencies were removed as animals with ChIs offline during that phase, regardless of ChI inhibition previously, decreased responding more than animals with ChIs intact. These findings build upon a growing body of literature implicating ChIs in the striatum as gate-keepers of behavioral flexibility and exploration.