Neurocomputational models of basal ganglia function in learning, memory and choice

Mild Reinforcement Learning Deficits in Patients With First-Episode Psychosis

Numerous studies have identified reinforcement learning (RL) deficits in schizophrenia. Most have focused on chronic patients with longstanding antipsychotic treatment, however, and studies of RL in early-illness patients have produced mixed results, particularly regarding gradual/procedural learning. No study has directly contrasted both rapid and gradual RL in first-episode psychosis (FEP) samples. We examined probabilistic RL in 34 FEP patients and 36 controls, using Go/NoGo (GNG) and Gain vs Loss-Avoidance (GLA) paradigms. Our results were mixed, with FEP patients exhibiting greater impairment in the ability to use positive, as opposed to negative, feedback to drive rapid RL on the GLA, but not the GNG. By contrast, patients and controls showed similar improvement across the acquisition. Finally, we found no significant between-group differences in the postacquisition expression of value-based preference in both tasks. Negative symptoms were modestly associated with RL measures, while the overall bias to engage in Go-responding correlated significantly with psychosis severity in FEP patients, consistent with striatal hyperdopaminergia. Taken together, FEP patients demonstrated more circumscribed RL impairments than previous studies have documented in chronic samples, possibly reflecting differential symptom profiles between first-episode and chronic samples. Our finding of relatively preserved gradual/procedural RL, in briefly medicated FEP patients, might suggest spared or restored basal ganglia function. Our findings of preserved abilities to use representations of expected value to guide decision making, and our mixed results regarding rapid RL, may reflect a lesser degree of prefrontal cortical functional impairment in FEP than in chronic samples. Further longitudinal research, in larger samples, is required.

Modelling ADHD: A review of ADHD theories through their predictions for computational models of decision-making and reinforcement learning

Attention deficit hyperactivity disorder (ADHD) is characterized by altered decision-making (DM) and reinforcement learning (RL), for which competing theories propose alternative explanations. Computational modelling contributes to understanding DM and RL by integrating behavioural and neurobiological findings, and could elucidate pathogenic mechanisms behind ADHD. This review of neurobiological theories of ADHD describes predictions for the effect of ADHD on DM and RL as described by the drift-diffusion model of DM (DDM) and a basic RL model. Empirical studies employing these models are also reviewed. While theories often agree on how ADHD should be reflected in model parameters, each theory implies a unique combination of predictions. Empirical studies agree with the theories' assumptions of a lowered DDM drift rate in ADHD, while findings are less conclusive for boundary separation. The few studies employing RL models support a lower choice sensitivity in ADHD, but not an altered learning rate. The discussion outlines research areas for further theoretical refinement in the ADHD field.

Chronic D2/3 agonist ropinirole treatment increases preference for uncertainty in rats regardless of baseline choice patterns

D_2/3 receptor agonists are effective treatments for Parkinson's disease (PD), but can precipitate impulse control disorders (ICDs) including gambling disorder (GD). The neurobiological mechanisms underlying this devastating side-effect of dopamine agonist replacement therapy (DRT), and any dependence on the dopamine depletion caused by PD, are unclear. It is also unclear whether previous biases towards risk or uncertainty are a risk factor for developing these ICDs. We investigated whether chronic D_2/3 agonist administration (5 mg/kg/day ropinirole for 28 days) altered performance of a rat model of gambling-like behaviour, the rodent betting task (rBT), and examined if baseline behaviour predicted this behavioural change. The rBT captures individual differences in subjective preference for uncertain outcomes: animals choose between guaranteed or probabilistic reinforcement of equal expected value. Chronic ropinirole dramatically increased selection of the uncertain option in two-thirds of animals, regardless of baseline preferences. The effect on choice in the rBT was replicated in a dorsolateral striatal 6-hydroxydopamine (6-OHDA) rat model of early PD. These studies are the first to look at individual differences in response to chronic, rather than pulsatile, dosing of DRT in a rodent model of gambling behaviour. These findings suggest that DRT-induced PG may stem from increases in subjective valuation of uncertainty. Such symptoms likely arise because of changes in dopaminergic striatal signalling caused by DRT rather than from an interaction between pre-morbid behaviours or PD itself.

Functional Relevance of Different Basal Ganglia Pathways Investigated in a Spiking Model with Reward Dependent Plasticity

The brain enables animals to behaviorally adapt in order to survive in a complex and dynamic environment, but how reward-oriented behaviors are achieved and computed by its underlying neural circuitry is an open question. To address this concern, we have developed a spiking model of the basal ganglia (BG) that learns to dis-inhibit the action leading to a reward despite ongoing changes in the reward schedule. The architecture of the network features the two pathways commonly described in BG, the direct (denoted D1) and the indirect (denoted D2) pathway, as well as a loop involving striatum and the dopaminergic system. The activity of these dopaminergic neurons conveys the reward prediction error (RPE), which determines the magnitude of synaptic plasticity within the different pathways. All plastic connections implement a versatile four-factor learning rule derived from Bayesian inference that depends upon pre- and post-synaptic activity, receptor type, and dopamine level. Synaptic weight updates occur in the D1 or D2 pathways depending on the sign of the RPE, and an efference copy informs upstream nuclei about the action selected. We demonstrate successful performance of the system in a multiple-choice learning task with a transiently changing reward schedule. We simulate lesioning of the various pathways and show that a condition without the D2 pathway fares worse than one without D1. Additionally, we simulate the degeneration observed in Parkinson's disease (PD) by decreasing the number of dopaminergic neurons during learning. The results suggest that the D1 pathway impairment in PD might have been overlooked. Furthermore, an analysis of the alterations in the synaptic weights shows that using the absolute reward value instead of the RPE leads to a larger change in D1.

Safety out of control: dopamine and defence

We enjoy a sophisticated understanding of how animals learn to predict appetitive outcomes and direct their behaviour accordingly. This encompasses well-defined learning algorithms and details of how these might be implemented in the brain. Dopamine has played an important part in this unfolding story, appearing to embody a learning signal for predicting rewards and stamping in useful actions, while also being a modulator of behavioural vigour. By contrast, although choosing correct actions and executing them vigorously in the face of adversity is at least as important, our understanding of learning and behaviour in aversive settings is less well developed. We examine aversive processing through the medium of the role of dopamine and targets such as D2 receptors in the striatum. We consider critical factors such as the degree of control that an animal believes it exerts over key aspects of its environment, the distinction between 'better' and 'good' actual or predicted future states, and the potential requirement for a particular form of opponent to dopamine to ensure proper calibration of state values.

Comparison of different speech tasks among adults who stutter and adults who do not stutter

OBJECTIVES

In this study, we compared the performance of both fluent speakers and people who stutter in three different speaking situations: monologue speech, oral reading and choral reading. This study follows the assumption that the neuromotor control of speech can be influenced by external auditory stimuli in both speakers who stutter and speakers who do not stutter.

METHOD

Seventeen adults who stutter and seventeen adults who do not stutter were assessed in three speaking tasks: monologue, oral reading (solo reading aloud) and choral reading (reading in unison with the evaluator). Speech fluency and rate were measured for each task.

RESULTS

The participants who stuttered had a lower frequency of stuttering during choral reading than during monologue and oral reading.

CONCLUSIONS

According to the dual premotor system model, choral speech enhanced fluency by providing external cues for the timing of each syllable compensating for deficient internal cues.

Human Frontal-Subcortical Circuit and Asymmetric Belief Updating

How humans integrate information to form beliefs about reality is a question that has engaged scientists for centuries, yet the biological system supporting this process is not well understood. One of the most salient attributes of information is valence. Whether a piece of news is good or bad is critical in determining whether it will alter our beliefs. Here, we reveal a frontal-subcortical circuit in the left hemisphere that is simultaneously associated with enhanced integration of favorable information into beliefs and impaired integration of unfavorable information. Specifically, for favorable information, stronger white matter connectivity within this system, particularly between the left inferior frontal gyrus (IFG) and left subcortical regions (including the amygdala, hippocampus, thalamus, putamen, and pallidum), as well as insular cortex, is associated with greater change in belief. However, for unfavorable information, stronger connectivity within this system, particularly between the left IFG and left pallidum, putamen, and insular cortex, is associated with reduced change in beliefs. These novel results are consistent with models suggesting that partially separable processes govern learning from favorable and unfavorable information.

SIGNIFICANCE STATEMENT

Beliefs of what may happen in the future are important, because they guide decisions and actions. Here, we illuminate how structural brain connectivity is related to the generation of subjective beliefs. We focus on how the valence of information is related to people's tendency to alter their beliefs. By quantifying the extent to which participants update their beliefs in response to desirable and undesirable information and relating those measures to the strength of white matter connectivity using diffusion tensor imaging, we characterize a left frontal-subcortical system that is associated simultaneously with greater belief updating in response to favorable information and reduced belief updating in response to unfavorable information. This neural architecture may allow valence to be incorporated into belief updating.

Fatty acid amide supplementation decreases impulsivity in young adult heavy drinkers

Compromised dopamine signaling in the striatum has been associated with the expression of impulsive behaviors in addiction, obesity and alcoholism. In rodents, intragastric infusion of the fatty acid amide oleoylethanolamide increases striatal extracellular dopamine levels via vagal afferent signaling. Here we tested whether supplementation with PhosphoLean™, a dietary supplement that contains the precursor of the fatty acid amide oleoylethanolamide (N-oleyl-phosphatidylethanolamine), would reduce impulsive responding and alcohol use in heavy drinking young adults. Twenty-two individuals were assigned to a three-week supplementation regimen with PhosphoLean™ or placebo. Impulsivity was assessed with self-report questionnaires and behavioral tasks pre- and post-supplementation. Although self-report measures of impulsivity did not change, supplementation with PhosphoLean™, but not placebo, significantly reduced false alarm rate on a Go/No-Go task. In addition, an association was found between improved sensitivity on the Go/No-Go task and reduced alcohol intake. These findings provide preliminary evidence that promoting fatty acid derived gut-brain dopamine communication may have therapeutic potential for reducing impulsivity in heavy drinkers.

A Biologically Inspired Computational Model of Basal Ganglia in Action Selection

The basal ganglia (BG) are a subcortical structure implicated in action selection. The aim of this work is to present a new cognitive neuroscience model of the BG, which aspires to represent a parsimonious balance between simplicity and completeness. The model includes the 3 main pathways operating in the BG circuitry, that is, the direct (Go), indirect (NoGo), and hyperdirect pathways. The main original aspects, compared with previous models, are the use of a two-term Hebb rule to train synapses in the striatum, based exclusively on neuronal activity changes caused by dopamine peaks or dips, and the role of the cholinergic interneurons (affected by dopamine themselves) during learning. Some examples are displayed, concerning a few paradigmatic cases: action selection in basal conditions, action selection in the presence of a strong conflict (where the role of the hyperdirect pathway emerges), synapse changes induced by phasic dopamine, and learning new actions based on a previous history of rewards and punishments. Finally, some simulations show model working in conditions of altered dopamine levels, to illustrate pathological cases (dopamine depletion in parkinsonian subjects or dopamine hypermedication). Due to its parsimonious approach, the model may represent a straightforward tool to analyze BG functionality in behavioral experiments.

Response inhibition of cigarette-related cues in male light smokers: behavioral evidence using a two-choice oddball paradigm

Behavioral inhibitory control has been shown to play an important role in a variety of addictive behaviors. A number of studies involving the use of Go/NoGo and stop-signal paradigms have shown that smokers have reduced response inhibition for cigarette-related cues. However, it is not known whether male light smokers' response inhibition for cigarette-related cues is lower than that of non-smokers in the two-choice oddball paradigm. The objective of the current study was to provide further behavioral evidence of male light smokers' impaired response inhibition for cigarette-related cues, using the two-choice oddball paradigm. Sixty-two male students (31 smokers, 31 non-smokers), who were recruited via an advertisement, took part in this two-choice oddball experiment. Cigarette-related pictures (deviant stimuli) and pictures unrelated to cigarettes (standard stimuli) were used. Response inhibition for cigarette-related cues was measured by comparing accuracy (ACC) and reaction time (RT) for deviant and standard stimuli in the two groups of subjects. An analysis of variance (ANOVA) showed that in all the participants, ACC was significantly lower for deviant stimuli than for standard stimuli. For deviant stimuli, the RTs were significantly longer for male light smokers than for male non-smokers; however, there was no significant difference in RTs for standard stimuli. Compared to male non-smokers, male light smokers seem to have a reduced ability to inhibit responses to cigarette-related cues.

Efficient reinforcement learning of a reservoir network model of parametric working memory achieved with a cluster population winner-take-all readout mechanism

The brain often has to make decisions based on information stored in working memory, but the neural circuitry underlying working memory is not fully understood. Many theoretical efforts have been focused on modeling the persistent delay period activity in the prefrontal areas that is believed to represent working memory. Recent experiments reveal that the delay period activity in the prefrontal cortex is neither static nor homogeneous as previously assumed. Models based on reservoir networks have been proposed to model such a dynamical activity pattern. The connections between neurons within a reservoir are random and do not require explicit tuning. Information storage does not depend on the stable states of the network. However, it is not clear how the encoded information can be retrieved for decision making with a biologically realistic algorithm. We therefore built a reservoir-based neural network to model the neuronal responses of the prefrontal cortex in a somatosensory delayed discrimination task. We first illustrate that the neurons in the reservoir exhibit a heterogeneous and dynamical delay period activity observed in previous experiments. Then we show that a cluster population circuit decodes the information from the reservoir with a winner-take-all mechanism and contributes to the decision making. Finally, we show that the model achieves a good performance rapidly by shaping only the readout with reinforcement learning. Our model reproduces important features of previous behavior and neurophysiology data. We illustrate for the first time how task-specific information stored in a reservoir network can be retrieved with a biologically plausible reinforcement learning training scheme.

Rule encoding in dorsal striatum impacts action selection

Cognitive flexibility is a hallmark of prefrontal cortical (PFC) function yet little is known about downstream area involvement. The medial dorsal striatum (mDS) receives major projections from the PFC and is uniquely situated to perform the integration of responses with rule information. In this study, we use a novel rule shifting task in rats that mirrors non-human primate and human studies in its temporal precision and counterbalanced responses. We record activity from single neurons in the mDS while rats switch between different rules for reward. Additionally, we pharmacologically inactivate mDS by infusion of a baclofen/muscimol cocktail. Inactivation of mDS impaired the ability to shift to a new rule and increased the number of regressive errors. While recording in mDS, we identified neurons modulated by direction whose activity reflected the conflict between competing rule information. We show that a subset of these neurons was also rule selective, and that the conflict between competing rule cues was resolved as behavioural performance improved. Other neurons were modulated by rule, but not direction. These neurons became selective before behavioural performance accurately reflected the current rule. These data provide an additional locus for investigating the mechanisms underlying behavioural flexibility. Converging lines of evidence from multiple human psychiatric disorders have implicated dorsal striatum as an important and understudied neural substrate of flexible cognition. Our data confirm the importance of mDS, and suggest a mechanism by which mDS mediates abstract cognition functions.

The ins and outs of the striatum: role in drug addiction

Addiction is a chronic relapsing disorder characterized by the loss of control over drug intake, high motivation to obtain the drug, and a persistent craving for the drug. Accumulating evidence implicates cellular and molecular alterations within cortico-basal ganglia-thalamic circuitry in the development and persistence of this disease. The striatum is a heterogeneous structure that sits at the interface of this circuit, receiving input from a variety of brain regions (e.g., prefrontal cortex, ventral tegmental area) to guide behavioral output, including motor planning, decision-making, motivation and reward. However, the vast interconnectivity of this circuit has made it difficult to isolate how individual projections and cellular subtypes within this circuit modulate each of the facets of addiction. Here, we review the use of new technologies, including optogenetics and DREADDs (Designer Receptors Exclusively Activated by Designer Drugs), in unraveling the role of the striatum in addiction. In particular, we focus on the role of striatal cell populations (i.e., direct and indirect pathway medium spiny neurons) and striatal dopaminergic and glutamatergic afferents in addiction-related plasticity and behaviors.

Identifying the Basal Ganglia network model markers for medication-induced impulsivity in Parkinson's disease patients

Impulsivity, i.e. irresistibility in the execution of actions, may be prominent in Parkinson's disease (PD) patients who are treated with dopamine precursors or dopamine receptor agonists. In this study, we combine clinical investigations with computational modeling to explore whether impulsivity in PD patients on medication may arise as a result of abnormalities in risk, reward and punishment learning. In order to empirically assess learning outcomes involving risk, reward and punishment, four subject groups were examined: healthy controls, ON medication PD patients with impulse control disorder (PD-ON ICD) or without ICD (PD-ON non-ICD), and OFF medication PD patients (PD-OFF). A neural network model of the Basal Ganglia (BG) that has the capacity to predict the dysfunction of both the dopaminergic (DA) and the serotonergic (5HT) neuromodulator systems was developed and used to facilitate the interpretation of experimental results. In the model, the BG action selection dynamics were mimicked using a utility function based decision making framework, with DA controlling reward prediction and 5HT controlling punishment and risk predictions. The striatal model included three pools of Medium Spiny Neurons (MSNs), with D1 receptor (R) alone, D2R alone and co-expressing D1R-D2R. Empirical studies showed that reward optimality was increased in PD-ON ICD patients while punishment optimality was increased in PD-OFF patients. Empirical studies also revealed that PD-ON ICD subjects had lower reaction times (RT) compared to that of the PD-ON non-ICD patients. Computational modeling suggested that PD-OFF patients have higher punishment sensitivity, while healthy controls showed comparatively higher risk sensitivity. A significant decrease in sensitivity to punishment and risk was crucial for explaining behavioral changes observed in PD-ON ICD patients. Our results highlight the power of computational modelling for identifying neuronal circuitry implicated in learning, and its impairment in PD. The results presented here not only show that computational modelling can be used as a valuable tool for understanding and interpreting clinical data, but they also show that computational modeling has the potential to become an invaluable tool to predict the onset of behavioral changes during disease progression.

The C957T polymorphism in the dopamine receptor D₂ gene modulates domain-general category learning

Adaptive learning from reward and punishment is vital for human survival. Striatal and frontal dopaminergic activities are associated with adaptive learning. For example, the C957T single nucleotide polymorphism of the dopamine receptor D2 (DRD2) gene alters striatal D2 receptor availability and affects individuals' adaptive learning ability. Specifically, individuals with the T/T genotype, which is associated with higher striatal D2 availability, show enhanced learning from negative outcomes. Prior work examining DRD2 genetic variability has focused primarily on frontally mediated reflective learning that is under effortful, conscious control. However, less is known about a more automatic, striatally mediated reflexive learning. Here we examined the extent to which this polymorphism differentially influences reflective and reflexive learning across visual and auditory modalities. We employed rule-based (RB) and information-integration (II) category learning paradigms that target reflective and reflexive learning, respectively. Results revealed an advantage in II category learning but poorer RB category learning in T/T homozygotes. The pattern of results was consistent across sensory modalities. These findings suggest that this DRD2 polymorphism exerts opposite influences on domain-general frontally mediated reflective learning and striatally mediated reflexive learning.

Dorsal striatum is necessary for stimulus-value but not action-value learning in humans

Several lines of evidence implicate the striatum in learning from experience on the basis of positive and negative feedback. However, the necessity of the striatum for such learning has been difficult to demonstrate in humans, because brain damage is rarely restricted to this structure. Here we test a rare individual with widespread bilateral damage restricted to the dorsal striatum. His performance was impaired and not significantly different from chance on several classic learning tasks, consistent with current theories regarding the role of the striatum. However, he also exhibited remarkably intact performance on a different subset of learning paradigms. The tasks he could perform can all be solved by learning the value of actions, while those he could not perform can only be solved by learning the value of stimuli. Although dorsal striatum is often thought to play a specific role in action-value learning, we find surprisingly that dorsal striatum is necessary for stimulus-value but not action-value learning in humans.

Neuromodulation of neurons and synapses

Neuromodulation underlies the flexibility of neural circuit operation and behavior. Individual neuromodulators can have divergent actions in a neuron by targeting multiple physiological mechanisms. Conversely, multiple neuromodulators may have convergent actions through overlapping targets. The divergent and convergent neuromodulator actions can be unambiguously synergistic or antagonistic, but neuromodulation often entails balanced adjustment of nonlinear membrane and synaptic properties by targeting ion channel and synaptic dynamics rather than just excitability or synaptic strength. In addition, neuromodulators can exert effects at multiple timescales, from short-term adjustments of neuron and synapse function to persistent long-term regulation. This short review summarizes some highlights of the diverse actions of neuromodulators on ion channel and synaptic properties.

Anatomical and diffusion MRI of deep gray matter in pediatric spina bifida

Individuals with spina bifida myelomeningocele (SBM) exhibit brain abnormalities in cortical thickness, white matter integrity, and cerebellar structure. Little is known about deep gray matter macro- and microstructure in this population. The current study utilized volumetric and diffusion-weighted MRI techniques to examine gray matter volume and microstructure in several subcortical structures: basal ganglia nuclei, thalamus, hippocampus, and amygdala. Sixty-six children and adolescents (ages 8–18; M = 12.0, SD = 2.73) with SBM and typically developing (TD) controls underwent T₁- and diffusion-weighted neuroimaging. Microstructural results indicated that hippocampal volume was disproportionately reduced, whereas the putamen volume was enlarged in the group with SBM. Microstructural analyses indicated increased mean diffusivity (MD) and fractional anisotropy (FA) in the gray matter of most examined structures (i.e., thalamus, caudate, hippocampus), with the putamen exhibiting a unique pattern of decreased MD and increased FA. These results provide further support that SBM differentially disrupts brain regions whereby some structures are volumetrically normal whereas others are reduced or enlarged. In the hippocampus, volumetric reduction coupled with increased MD may imply reduced cellular density and aberrant organization. Alternatively, the enlarged volume and significantly reduced MD in the putamen suggest increased density.

•

Spina bifida resulted in reduced hippocampal and enlarged putamen volumes.

•

Spina bifida resulted in reduced MD and increased FA in the putamen.

•

Periventricular regions were differentiated by increased MD and FA in spina bifida.

•

Spina bifida resulted in greater FA for all regions, except the hippocampus.

The role of dopamine in risk taking: a specific look at Parkinson's disease and gambling

An influential model suggests that dopamine signals the difference between predicted and experienced reward. In this way, dopamine can act as a learning signal that can shape behaviors to maximize rewards and avoid punishments. Dopamine is also thought to invigorate reward seeking behavior. Loss of dopamine signaling is the major abnormality in Parkinson’s disease. Dopamine agonists have been implicated in the occurrence of impulse control disorders in Parkinson’s disease patients, the most common being pathological gambling, compulsive sexual behavior, and compulsive buying. Recently, a number of functional imaging studies investigating impulse control disorders in Parkinson’s disease have been published. Here we review this literature, and attempt to place it within a decision-making framework in which potential gains and losses are evaluated to arrive at optimum choices. We also provide a hypothetical but still incomplete model on the effect of dopamine agonist treatment on these value and risk assessments. Two of the main brain structures thought to be involved in computing aspects of reward and loss are the ventral striatum (VStr) and the insula, both dopamine projection sites. Both structures are consistently implicated in functional brain imaging studies of pathological gambling in Parkinson’s disease.

Behavior modulates effective connectivity between cortex and striatum

It has been notoriously difficult to understand interactions in the basal ganglia because of multiple recurrent loops. Another complication is that activity there is strongly dependent on behavior, suggesting that directional interactions, or effective connections, can dynamically change. A simplifying approach would be to examine just the direct, monosynaptic projections from cortex to striatum and contrast this with the polysynaptic feedback connections from striatum to cortex. Previous work by others on effective connectivity in this pathway indicated that activity in cortex could be used to predict activity in striatum, but that striatal activity could not predict cortical activity. However, this work was conducted in anesthetized or seizing animals, making it impossible to know how free behavior might influence effective connectivity. To address this issue, we applied Granger causality to local field potential signals from cortex and striatum in freely behaving rats. Consistent with previous results, we found that effective connectivity was largely unidirectional, from cortex to striatum, during anesthetized and resting states. Interestingly, we found that effective connectivity became bidirectional during free behaviors. These results are the first to our knowledge to show that striatal influence on cortex can be as strong as cortical influence on striatum. In addition, these findings highlight how behavioral states can affect basal ganglia interactions. Finally, we suggest that this approach may be useful for studies of Parkinson's or Huntington's diseases, in which effective connectivity may change during movement.

Optogenetic stimulation in a computational model of the basal ganglia biases action selection and reward prediction error

Optogenetic stimulation of specific types of medium spiny neurons (MSNs) in the striatum has been shown to bias the selection of mice in a two choices task. This shift is dependent on the localisation and on the intensity of the stimulation but also on the recent reward history. We have implemented a way to simulate this increased activity produced by the optical flash in our computational model of the basal ganglia (BG). This abstract model features the direct and indirect pathways commonly described in biology, and a reward prediction pathway (RP). The framework is similar to Actor-Critic methods and to the ventral/dorsal distinction in the striatum. We thus investigated the impact on the selection caused by an added stimulation in each of the three pathways. We were able to reproduce in our model the bias in action selection observed in mice. Our results also showed that biasing the reward prediction is sufficient to create a modification in the action selection. However, we had to increase the percentage of trials with stimulation relative to that in experiments in order to impact the selection. We found that increasing only the reward prediction had a different effect if the stimulation in RP was action dependent (only for a specific action) or not. We further looked at the evolution of the change in the weights depending on the stage of learning within a block. A bias in RP impacts the plasticity differently depending on that stage but also on the outcome. It remains to experimentally test how the dopaminergic neurons are affected by specific stimulations of neurons in the striatum and to relate data to predictions of our model.

Emergent structured transition from variation to repetition in a biologically-plausible model of learning in basal ganglia

Often, when animals encounter an unexpected sensory event, they transition from executing a variety of movements to repeating the movement(s) that may have caused the event. According to a recent theory of action discovery (Redgrave and Gurney, 2006), repetition allows the animal to represent those movements, and the outcome, as an action for later recruitment. The transition from variation to repetition often follows a non-random, structured, pattern. While the structure of the pattern can be explained by sophisticated cognitive mechanisms, simpler mechanisms based on dopaminergic modulation of basal ganglia (BG) activity are thought to underlie action discovery (Redgrave and Gurney, 2006). In this paper we ask the question: can simple BG-mediated mechanisms account for a structured transition from variation to repetition, or are more sophisticated cognitive mechanisms always necessary? To address this question, we present a computational model of BG-mediated biasing of behavior. In our model, unlike most other models of BG function, the BG biases behavior through modulation of cortical response to excitation; many possible movements are represented by the cortical area; and excitation to the cortical area is topographically-organized. We subject the model to simple reaching tasks, inspired by behavioral studies, in which a location to which to reach must be selected. Locations within a target area elicit a reinforcement signal. A structured transition from variation to repetition emerges from simple BG-mediated biasing of cortical response to excitation. We show how the structured pattern influences behavior in simple and complicated tasks. We also present analyses that describe the structured transition from variation to repetition due to BG-mediated biasing and from biasing that would be expected from a type of cognitive biasing, allowing us to compare behavior resulting from these types of biasing and make connections with future behavioral experiments.

Phosphodiesterase 10A inhibitor MP-10 effects in primates: comparison with risperidone and mechanistic implications

Phosphodiesterase 10A (PDE10A) is highly expressed in striatal medium spiny neurons of both the direct and indirect output pathways. Similar to dopamine D₂ receptor antagonists acting on indirect pathway neurons, PDE10A inhibitors have shown behavioral effects in rodent models that predict antipsychotic efficacy. These findings have supported the clinical investigation of PDE10A inhibitors as a new treatment for schizophrenia. However, PDE10A inhibitors and D₂ antagonists differ in effects on direct pathway and other neurons of the basal ganglia, indicating that these two drug classes may have divergent antipsychotic efficacy and side effect profile. In the present study, we compare the behavioral effects of the selective PDE10A inhibitor MP-10 to those of the clinical standard D₂ antagonist risperidone in rhesus monkeys using a standardized motor disability scale for parkinsonian primates and a newly designed "Drug Effects on Nervous System" scale to assess non-motor effects. Behavioral effects of MP-10 correlated with its plasma levels and its regulation of metabolic activity in striatal and cortical regions as measured by FDG-PET imaging. While MP-10 and risperidone broadly impacted similar behavioral domains in the primate, their effects had a different underlying basis. MP-10-treated animals retained the ability to respond but did not engage tasks, whereas risperidone-treated animals retained the motivation to respond but were unable to perform the intended actions. These findings are discussed in light of what is currently known about the modulation of striatal circuitry by these two classes of compounds, and provide insight into interpreting emerging clinical data with PDE10A inhibitors for the treatment of psychotic symptoms.

Time representation in reinforcement learning models of the basal ganglia

Reinforcement learning (RL) models have been influential in understanding many aspects of basal ganglia function, from reward prediction to action selection. Time plays an important role in these models, but there is still no theoretical consensus about what kind of time representation is used by the basal ganglia. We review several theoretical accounts and their supporting evidence. We then discuss the relationship between RL models and the timing mechanisms that have been attributed to the basal ganglia. We hypothesize that a single computational system may underlie both RL and interval timing—the perception of duration in the range of seconds to hours. This hypothesis, which extends earlier models by incorporating a time-sensitive action selection mechanism, may have important implications for understanding disorders like Parkinson's disease in which both decision making and timing are impaired.

Haloperidol-induced changes in neuronal activity in the striatum of the freely moving rat

The striatum is the main input structure of the basal ganglia, integrating input from the cerebral cortex and the thalamus, which is modulated by midbrain dopaminergic input. Dopamine modulators, including agonists and antagonists, are widely used to relieve motor and psychiatric symptoms in a variety of pathological conditions. Haloperidol, a dopamine D2 antagonist, is commonly used in multiple psychiatric conditions and motor abnormalities. This article reports the effects of haloperidol on the activity of three major striatal subpopulations: medium spiny neurons (MSNs), fast spiking interneurons (FSIs), and tonically active neurons (TANs). We implanted multi-wire electrode arrays in the rat dorsal striatum and recorded the activity of multiple single units in freely moving animals before and after systemic haloperidol injection. Haloperidol decreased the firing rate of FSIs and MSNs while increasing their tendency to fire in an oscillatory manner in the high voltage spindle (HVS) frequency range of 7-9 Hz. Haloperidol led to an increased firing rate of TANs but did not affect their non-oscillatory firing pattern and their typical correlated firing activity. Our results suggest that dopamine plays a key role in tuning both single unit activity and the interactions within and between different subpopulations in the striatum in a differential manner. These findings highlight the heterogeneous striatal effects of tonic dopamine regulation via D2 receptors which potentially enable the treatment of diverse pathological states associated with basal ganglia dysfunction.

Elucidating poor decision-making in a rat gambling task

Although poor decision-making is a hallmark of psychiatric conditions such as attention deficit/hyperactivity disorder, pathological gambling or substance abuse, a fraction of healthy individuals exhibit similar poor decision-making performances in everyday life and specific laboratory tasks such as the Iowa Gambling Task. These particular individuals may provide information on risk factors or common endophenotypes of these mental disorders. In a rodent version of the Iowa gambling task--the Rat Gambling Task (RGT), we identified a population of poor decision makers, and assessed how these rats scored for several behavioral traits relevant to executive disorders: risk taking, reward seeking, behavioral inflexibility, and several aspects of impulsivity. First, we found that poor decision-making could not be well predicted by single behavioral and cognitive characteristics when considered separately. By contrast, a combination of independent traits in the same individual, namely risk taking, reward seeking, behavioral inflexibility, as well as motor impulsivity, was highly predictive of poor decision-making. Second, using a reinforcement-learning model of the RGT, we confirmed that only the combination of extreme scores on these traits could induce maladaptive decision-making. Third, the model suggested that a combination of these behavioral traits results in an inaccurate representation of rewards and penalties and inefficient learning of the environment. Poor decision-making appears as a consequence of the over-valuation of high-reward-high-risk options in the task. Such a specific psychological profile could greatly impair clinically healthy individuals in decision-making tasks and may predispose to mental disorders with similar symptoms.

Striatal abnormalities and spontaneous dyskinesias in non-clinical psychosis

BACKGROUND

Accumulating evidence suggests that individuals experiencing non-clinical psychosis (NCP) represent a critical group for improving understanding of etiological factors underlying the broader psychosis continuum. Although a wealth of evidence supports widespread neural dysfunction in formal psychosis, there has been little empirical evidence to inform our understanding of putative vulnerability markers or brain structure in NCP. In this study, we examined the neural correlates of spontaneous movement abnormalities, a biomarker previously detected in NCP that is linked to abnormalities in the striatal dopamine.

METHODS

We screened a total of 1285 adolescents/young adults, and those scoring in the upper 15th percentile on a NCP scale were invited to participate; 20 of those invited agreed and these individuals were matched with healthy controls. Participants were administered a structural scan, clinical interviews, and an instrumental motor assessment.

RESULTS

The NCP group showed elevated force variability and smaller putamen (but not caudate), and there was a significant relationship between motor dysfunction and striatal abnormalities for the sample. Elevated force variability was associated with both higher positive and negative symptoms, and there was a strong trend (p=.06) to suggest that smaller left putamen volumes were associated with elevated positive symptoms.

CONCLUSIONS

The results are among the first to suggest an association between neural structure and a risk marker in NCP. Findings indicate that vulnerabilities seen in schizophrenia also characterize the lower end of the psychosis spectrum.

The neural basis of responsibility attribution in decision-making

Social responsibility links personal behavior with societal expectations and plays a key role in affecting an agent's emotional state following a decision. However, the neural basis of responsibility attribution remains unclear. In two previous event-related brain potential (ERP) studies we found that personal responsibility modulated outcome evaluation in gambling tasks. Here we conducted a functional magnetic resonance imaging (fMRI) study to identify particular brain regions that mediate responsibility attribution. In a context involving team cooperation, participants completed a task with their teammates and on each trial received feedback about team success and individual success sequentially. We found that brain activity differed between conditions involving team success vs. team failure. Further, different brain regions were associated with reinforcement of behavior by social praise vs. monetary reward. Specifically, right temporoparietal junction (RTPJ) was associated with social pride whereas dorsal striatum and dorsal anterior cingulate cortex (ACC) were related to reinforcement of behaviors leading to personal gain. The present study provides evidence that the RTPJ is an important region for determining whether self-generated behaviors are deserving of praise in a social context.

Olfactory maps, circuits and computations

Sensory information in the visual, auditory and somatosensory systems is organized topographically, with key sensory features ordered in space across neural sheets. Despite the existence of a spatially stereotyped map of odor identity within the olfactory bulb, it is unclear whether the higher olfactory cortex uses topography to organize information about smells. Here, we review recent work on the anatomy, microcircuitry and neuromodulation of two higher-order olfactory areas: the piriform cortex and the olfactory tubercle. The piriform is an archicortical region with an extensive local associational network that constructs representations of odor identity. The olfactory tubercle is an extension of the ventral striatum that may use reward-based learning rules to encode odor valence. We argue that in contrast to brain circuits for other sensory modalities, both the piriform and the olfactory tubercle largely discard any topography present in the bulb and instead use distributive afferent connectivity, local learning rules and input from neuromodulatory centers to build behaviorally relevant representations of olfactory stimuli.

Dopamine on D2-like receptors "reboosts" dopamine D1-like receptor-mediated behavioural activation in rats licking for a isotonic NaCl solution

RATIONALE

We recently suggested that dopamine on D1-like receptors is involved in the activation of goal-directed responses and the level of response activation is "reboosted" on the basis of an evaluation process involving D2-like receptors assessing "response efficacy". A main piece of evidence in support of this hypothesis was the observation of an "extinction mimicry" effect in the time course of licking bursts after dopamine D2-like receptor blockade in rats licking for sucrose.

OBJECTIVES

The aim of this study was to determine whether the pattern of licking observed with sucrose as a reward could be reproduced in rats licking for a different reward (0.9% NaCl).

MATERIALS AND METHODS

We investigated the effects of the dopamine D1-like receptor antagonist SCH 23390 (0.01-0.04 mg/kg) and of the dopamine D2-like receptor antagonist raclopride (0.025-0.25 mg/kg) on the microstructure of licking for a 0.9% NaCl solution in 12-h water-deprived rats in 30-min sessions.

RESULTS

As previously observed with sucrose as a reward, raclopride reduced the size of licking bursts and produced on the burst number time course an "extinction mimicry" effect, while SCH 23390 reduced licking exclusively by reducing burst number.

CONCLUSIONS

These results are consistent with the proposed hypothesis and provide support to the use of the study of licking microstructure as a valid model not only for the investigation of the mechanisms governing ingestive behaviour but also for the investigation of the mechanisms underlying behavioural activation and the related evaluation processes.

Using a hybrid neuron in physiologically inspired models of the basal ganglia

Our current understanding of the basal ganglia (BG) has facilitated the creation of computational models that have contributed novel theories, explored new functional anatomy and demonstrated results complementing physiological experiments. However, the utility of these models extends beyond these applications. Particularly in neuromorphic engineering, where the basal ganglia's role in computation is important for applications such as power efficient autonomous agents and model-based control strategies. The neurons used in existing computational models of the BG, however, are not amenable for many low-power hardware implementations. Motivated by a need for more hardware accessible networks, we replicate four published models of the BG, spanning single neuron and small networks, replacing the more computationally expensive neuron models with an Izhikevich hybrid neuron. This begins with a network modeling action-selection, where the basal activity levels and the ability to appropriately select the most salient input is reproduced. A Parkinson's disease model is then explored under normal conditions, Parkinsonian conditions and during subthalamic nucleus deep brain stimulation (DBS). The resulting network is capable of replicating the loss of thalamic relay capabilities in the Parkinsonian state and its return under DBS. This is also demonstrated using a network capable of action-selection. Finally, a study of correlation transfer under different patterns of Parkinsonian activity is presented. These networks successfully captured the significant results of the originals studies. This not only creates a foundation for neuromorphic hardware implementations but may also support the development of large-scale biophysical models. The former potentially providing a way of improving the efficacy of DBS and the latter allowing for the efficient simulation of larger more comprehensive networks.

A comparison of the subsecond dynamics of neurotransmission of dopamine and serotonin

The neuromodulators dopamine (DA) and serotonin (5-hydroxytryptamine; 5-HT) are similar in a number of ways. Both monoamines can act by volume transmission at metabotropic receptors to modulate synaptic transmission in brain circuits. Presynaptic regulation of 5-HT and DA is governed by parallel processes, and behaviorally, both exert control over emotional processing. However, differences are also apparent: more than twice as many 5-HT receptor subtypes mediate postsynaptic effects than DA receptors and different presynaptic regulation is also emerging. Monoamines are amenable to real-time electrochemical detection using fast scan cyclic voltammetry (FSCV), which allows resolution of the subsecond dynamics of release and reuptake in response to a single action potential. This approach has greatly enriched understanding of DA transmission and has facilitated an integrated view of how DA mediates behavioral control. However, technical challenges are associated with FSCV measurement of 5-HT and understanding of 5-HT transmission at subsecond resolution has not advanced at the same rate. As a result, how the actions of 5-HT at the level of the synapse translate into behavior is poorly understood. Recent technical advances may aid the study of 5-HT in real-time. It is timely, therefore, to compare and contrast what is currently understood of the subsecond characteristics of transmission for DA and 5-HT. In doing so, a number of areas are highlighted as being worthy of exploration for 5-HT.

Corticostriatal connectivity and its role in disease

Corticostriatal projections are essential components of forebrain circuits and are widely involved in motivated behaviour. These axonal projections are formed by two distinct classes of cortical neurons, intratelencephalic (IT) and pyramidal tract (PT) neurons. Convergent evidence points to IT versus PT differentiation of the corticostriatal system at all levels of functional organization, from cellular signalling mechanisms to circuit topology. There is also growing evidence for IT/PT imbalance as an aetiological factor in neurodevelopmental, neuropsychiatric and movement disorders - autism, amyotrophic lateral sclerosis, obsessive-compulsive disorder, schizophrenia, Huntington's and Parkinson's diseases and major depression are highlighted here.

The basal ganglia

Through its connections with widespread cortical areas and with dopaminergic midbrain areas, the basal ganglia are well situated to integrate patterns of cortical input with the dopaminergic reward signal originating in the midbrain. In this review, we consider the functions of the basal ganglia in relation to its gross and cellular anatomy, and discuss how these mechanisms subserve the thresholding and selection of motor and cognitive processes. We also discuss how the dopaminergic reward signal enables flexible task learning through modulation of striatal plasticity, and how reinforcement learning models have been used to account for various aspects of basal ganglia activity. Specifically, we will discuss the important role of the basal ganglia in instrumental learning, cognitive control, sequence learning, and categorization tasks. Finally, we will discuss the neurobiological and cognitive characteristics of Parkinson's disease, Huntington's disease and addiction to illustrate the relationship between the basal ganglia and cognitive function. WIREs Cogn Sci 2013, 4:135-148. doi: 10.1002/wcs.1217 For further resources related to this article, please visit the WIREs website.

Twenty-five lessons from computational neuromodulation

Neural processing faces three rather different, and perniciously tied, communication problems. First, computation is radically distributed, yet point-to-point interconnections are limited. Second, the bulk of these connections are semantically uniform, lacking differentiation at their targets that could tag particular sorts of information. Third, the brain's structure is relatively fixed, and yet different sorts of input, forms of processing, and rules for determining the output are appropriate under different, and possibly rapidly changing, conditions. Neuromodulators address these problems by their multifarious and broad distribution, by enjoying specialized receptor types in partially specific anatomical arrangements, and by their ability to mold the activity and sensitivity of neurons and the strength and plasticity of their synapses. Here, I offer a computationally focused review of algorithmic and implementational motifs associated with neuromodulators, using decision making in the face of uncertainty as a running example.

Reinforcement learning in young adults with developmental language impairment

The aim of the study was to examine reinforcement learning (RL) in young adults with developmental language impairment (DLI) within the context of a neurocomputational model of the basal ganglia-dopamine system (Frank, Seeberger, & O'Reilly, 2004). Two groups of young adults, one with DLI and the other without, were recruited. A probabilistic selection task was used to assess how participants implicitly extracted reinforcement history from the environment based on probabilistic positive/negative feedback. The findings showed impaired RL in individuals with DLI, indicating an altered gating function of the striatum in testing. However, they exploited similar learning strategies as comparison participants at the beginning of training, reflecting relatively intact functions of the prefrontal cortex to rapidly update reinforcement information. Within the context of Frank's model, these results can be interpreted as evidence for alterations in the basal ganglia of individuals with DLI.

Striatal contributions to declarative memory retrieval

Declarative memory is known to depend on the medial temporal lobe memory system. Recently, there has been renewed focus on the relationship between the basal ganglia and declarative memory, including the involvement of striatum. However, the contribution of striatum to declarative memory retrieval remains unknown. Here, we review neuroimaging and neuropsychological evidence for the involvement of the striatum in declarative memory retrieval. From this review, we propose that, along with the prefrontal cortex (PFC), the striatum primarily supports cognitive control of memory retrieval. We conclude by proposing three hypotheses for the specific role of striatum in retrieval: (1) striatum modulates the re-encoding of retrieved items in accord with their expected utility (adaptive encoding), (2) striatum selectively admits information into working memory that is expected to increase the likelihood of successful retrieval (adaptive gating), and (3) striatum enacts adjustments in cognitive control based on the outcome of retrieval (reinforcement learning).

Action selection performance of a reconfigurable basal ganglia inspired model with Hebbian-Bayesian Go-NoGo connectivity

Several studies have shown a strong involvement of the basal ganglia (BG) in action selection and dopamine dependent learning. The dopaminergic signal to striatum, the input stage of the BG, has been commonly described as coding a reward prediction error (RPE), i.e., the difference between the predicted and actual reward. The RPE has been hypothesized to be critical in the modulation of the synaptic plasticity in cortico-striatal synapses in the direct and indirect pathway. We developed an abstract computational model of the BG, with a dual pathway structure functionally corresponding to the direct and indirect pathways, and compared its behavior to biological data as well as other reinforcement learning models. The computations in our model are inspired by Bayesian inference, and the synaptic plasticity changes depend on a three factor Hebbian–Bayesian learning rule based on co-activation of pre- and post-synaptic units and on the value of the RPE. The model builds on a modified Actor-Critic architecture and implements the direct (Go) and the indirect (NoGo) pathway, as well as the reward prediction (RP) system, acting in a complementary fashion. We investigated the performance of the model system when different configurations of the Go, NoGo, and RP system were utilized, e.g., using only the Go, NoGo, or RP system, or combinations of those. Learning performance was investigated in several types of learning paradigms, such as learning-relearning, successive learning, stochastic learning, reversal learning and a two-choice task. The RPE and the activity of the model during learning were similar to monkey electrophysiological and behavioral data. Our results, however, show that there is not a unique best way to configure this BG model to handle well all the learning paradigms tested. We thus suggest that an agent might dynamically configure its action selection mode, possibly depending on task characteristics and also on how much time is available.

Stress modulates reinforcement learning in younger and older adults

Animal research and human neuroimaging studies indicate that stress increases dopamine levels in brain regions involved in reward processing, and stress also appears to increase the attractiveness of addictive drugs. The current study tested the hypothesis that stress increases reward salience, leading to more effective learning about positive than negative outcomes in a probabilistic selection task. Changes to dopamine pathways with age raise the question of whether stress effects on incentive-based learning differ by age. Thus, the present study also examined whether effects of stress on reinforcement learning differed for younger (age 18-34) and older participants (age 65-85). Cold pressor stress was administered to half of the participants in each age group, and salivary cortisol levels were used to confirm biophysiological response to cold stress. After the manipulation, participants completed a probabilistic learning task involving positive and negative feedback. In both younger and older adults, stress enhanced learning about cues that predicted positive outcomes. In addition, during the initial learning phase, stress diminished sensitivity to recent feedback across age groups. These results indicate that stress affects reinforcement learning in both younger and older adults and suggests that stress exerts different effects on specific components of reinforcement learning depending on their neural underpinnings.

Neurocircuitry for modeling drug effects

The identification and functional understanding of the neurocircuitry that mediates alcohol and drug effects that are relevant for the development of addictive behavior is a fundamental challenge in addiction research. Here we introduce an assumption-free construction of a neurocircuitry that mediates acute and chronic drug effects on neurotransmitter dynamics that is solely based on rodent neuroanatomy. Two types of data were considered for constructing the neurocircuitry: (1) information on the cytoarchitecture and neurochemical connectivity of each brain region of interest obtained from different neuroanatomical techniques; (2) information on the functional relevance of each region of interest with respect to alcohol and drug effects. We used mathematical data mining and hierarchical clustering methods to achieve the highest standards in the preprocessing of these data. Using this approach, a dynamical network of high molecular and spatial resolution containing 19 brain regions and seven neurotransmitter systems was obtained. Further graph theoretical analysis suggests that the neurocircuitry is connected and cannot be separated into further components. Our analysis also reveals the existence of a principal core subcircuit comprised of nine brain regions: the prefrontal cortex, insular cortex, nucleus accumbens, hypothalamus, amygdala, thalamus, substantia nigra, ventral tegmental area and raphe nuclei. Finally, by means of algebraic criteria for synchronizability of the neurocircuitry, the suitability for in silico modeling of acute and chronic drug effects is indicated. Indeed, we introduced as an example a dynamical system for modeling the effects of acute ethanol administration in rats and obtained an increase in dopamine release in the nucleus accumbens-a hallmark of drug reinforcement-to an extent similar to that seen in numerous microdialysis studies. We conclude that the present neurocircuitry provides a structural and dynamical framework for large-scale mathematical models and will help to predict chronic drug effects on brain function.

Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources

Accumulating evidence indicates integration of dopamine function with metabolic signals, highlighting a potential role for dopamine in energy balance, frequently construed as modulating reward in response to homeostatic state. Though its precise role remains controversial, the reward perspective of dopamine has dominated investigation of motivational disorders, including obesity. In the hypothesis outlined here, we suggest instead that the primary role of dopamine in behavior is to modulate activity to adapt behavioral energy expenditure to the prevailing environmental energy conditions, with the role of dopamine in reward and motivated behaviors derived from its primary role in energy balance. Dopamine has long been known to modulate activity, exemplified by psychostimulants that act via dopamine. More recently, there has been nascent investigation into the role of dopamine in modulating voluntary activity, with some investigators suggesting that dopamine may serve as a final common pathway that couples energy sensing to regulated voluntary energy expenditure. We suggest that interposed between input from both the internal and external world, dopamine modulates behavioral energy expenditure along two axes: a conserve-expend axis that regulates generalized activity and an explore-exploit axes that regulates the degree to which reward value biases the distribution of activity. In this view, increased dopamine does not promote consumption of tasty food. Instead increased dopamine promotes energy expenditure and exploration while decreased dopamine favors energy conservation and exploitation. This hypothesis provides a mechanistic interpretation to an apparent paradox: the well-established role of dopamine in food seeking and the findings that low dopaminergic functions are associated with obesity. Our hypothesis provides an alternative perspective on the role of dopamine in obesity and reinterprets the "reward deficiency hypothesis" as a perceived energy deficit. We propose that dopamine, by facilitating energy expenditure, should be protective against obesity. We suggest the apparent failure of this protective mechanism in Western societies with high prevalence of obesity arises as a consequence of sedentary lifestyles that thwart energy expenditure.