Self-organization in the basal ganglia with modulation of reinforcement signals

Overt and covert attention to location-based reward

Recent research on the impact of location-based reward on attentional orienting has indicated that reward factors play an influential role in spatial priority maps. The current study investigated whether and how reward associations based on spatial location translate from overt eye movements to covert attention. If reward associations can be tied to locations in space, and if overt and covert attention rely on similar overlapping neuronal populations, then both overt and covert attentional measures should display similar spatial-based reward learning. Our results suggest that location- and reward-based changes in one attentional domain do not lead to similar changes in the other. Specifically, although we found similar improvements at differentially rewarded locations during overt attentional learning, this translated to the least improvement at a highly rewarded location during covert attention. We interpret this as the result of an increased motivational link between the high reward location and the trained eye movement response acquired during learning, leading to a relative slowing during covert attention when the eyes remained fixated and the saccade response was suppressed. In a second experiment participants were not required to keep fixated during the covert attention task and we no longer observed relative slowing at the high reward location. Furthermore, the second experiment revealed no covert spatial priority of rewarded locations. We conclude that the transfer of location-based reward associations is intimately linked with the reward-modulated motor response employed during learning, and alternative attentional and task contexts may interfere with learned spatial priorities.

An attractor-based complexity measurement for Boolean recurrent neural networks

We provide a novel refined attractor-based complexity measurement for Boolean recurrent neural networks that represents an assessment of their computational power in terms of the significance of their attractor dynamics. This complexity measurement is achieved by first proving a computational equivalence between Boolean recurrent neural networks and some specific class of -automata, and then translating the most refined classification of -automata to the Boolean neural network context. As a result, a hierarchical classification of Boolean neural networks based on their attractive dynamics is obtained, thus providing a novel refined attractor-based complexity measurement for Boolean recurrent neural networks. These results provide new theoretical insights to the computational and dynamical capabilities of neural networks according to their attractive potentialities. An application of our findings is illustrated by the analysis of the dynamics of a simplified model of the basal ganglia-thalamocortical network simulated by a Boolean recurrent neural network. This example shows the significance of measuring network complexity, and how our results bear new founding elements for the understanding of the complexity of real brain circuits.

Modeling the role of basal ganglia in saccade generation: is the indirect pathway the explorer?

We model the role played by the Basal Ganglia (BG) in the generation of voluntary saccadic eye movements. The BG model explicitly represents key nuclei like the striatum (caudate), Substantia Nigra pars reticulata (SNr) and compata (SNc), the Subthalamic Nucleus (STN), the two pallidal nuclei and Superior Colliculus. The model is cast within the Reinforcement Learning (RL) framework, with the dopamine representing the temporal difference error, the striatum serving as the critic, and the indirect pathway playing the role of the explorer. Performance of the model is evaluated on a set of tasks such as feature and conjunction searches, directional selectivity and a successive saccade task. Behavioral phenomena such as independence of search time on number of distractors in feature search and linear increase in search time with number of distractors in conjunction search are observed. It is also seen that saccadic reaction times are longer and search efficiency is impaired on diminished BG contribution, which corroborates with reported data obtained from Parkinson's Disease (PD) patients.

Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task

Decision-making studies across different domains suggest that decisions can arise from multiple, parallel systems in the brain: a flexible system utilizing action-outcome expectancies and a more rigid system based on situation-action associations. The hippocampus, ventral striatum, and dorsal striatum make unique contributions to each system, but how information processing in each of these structures supports these systems is unknown. Recent work has shown covert representations of future paths in hippocampus and of future rewards in ventral striatum. We developed analyses in order to use a comparative methodology and apply the same analyses to all three structures. Covert representations of future paths and reward were both absent from the dorsal striatum. In contrast, dorsal striatum slowly developed situation representations that selectively represented action-rich parts of the task. This triple dissociation suggests that the different roles these structures play are due to differences in information-processing mechanisms.

Dynamical model of salience gated working memory, action selection and reinforcement based on basal ganglia and dopamine feedback

A simple working memory model based on recurrent network activation is proposed and its application to selection and reinforcement of an action is demonstrated as a solution to the temporal credit assignment problem. Reactivation of recent salient cue states is generated and maintained as a type of salience gated recurrently active working memory, while lower salience distractors are ignored. Cue reactivation during the action selection period allows the cue to select an action while its reactivation at the reward period allows the reinforcement of the action selected by the reactivated state, which is necessarily the action which led to the reward being found. A down-gating of the external input during the reactivation and maintenance prevents interference. A double winner-take-all system which selects only one cue and only one action allows the targeting of the cue-action allocation to be modified. This targeting works both to reinforce a correct cue-action allocation and to punish the allocation when cue-action allocations change. Here we suggest a firing rate neural network implementation of this system based on the basal ganglia anatomy with input from a cortical association layer where reactivations are generated by signals from the thalamus. Striatum medium spiny neurons represent actions. Auto-catalytic feedback from a dopamine reward signal modulates three-way Hebbian long term potentiation and depression at the cortical-striatal synapses which represent the cue-action associations. The model is illustrated by the numerical simulations of a simple example--that of associating a cue signal to a correct action to obtain reward after a delay period, typical of primate cue reward tasks. Through learning, the model shows a transition from an exploratory phase where actions are generated randomly, to a stable directed phase where the animal always chooses the correct action for each experienced state. When cue-action allocations change, we show that this is noticed by the model, the incorrect cue-action allocations are punished and the correct ones discovered.

Plastic Corticostriatal Circuits for Action Learning: What's Dopamine Got to Do with It?

Reentrant corticobasal ganglia circuits are important for voluntary action and for action selection. In vivo and ex vivo studies show that these circuits can exhibit a plethora of short- and long-lasting plastic changes. Convergent evidence at the molecular, cellular, and circuit levels indicates that corticostriatal circuits are involved in the acquisition and automatization of novel actions. There is strong evidence that activity in corticostriatal circuits changes during the learning of novel actions, but the plastic changes observed during the early stages of learning a novel action are different than those observed after extensive training. A variety of studies indicate that the neural mechanisms and the corticostriatal subcircuits involved in the initial acquisition of actions and skills differ from those involved in their automatization or in the formation of habits. Dopamine, a critical modulator of short- and long-term plasticity in corticostriatal circuits, is differentially involved in early and late stages of action learning. Changes in dopaminergic transmission have several concomitant effects in corticostriatal function, which may be important for action selection and action learning. These diverse effects may subserve different roles for dopamine in reinforcement and action learning.

Basal ganglia orient eyes to reward

Expectation of reward motivates our behaviors and influences our decisions. Indeed, neuronal activity in many brain areas is modulated by expected reward. However, it is still unclear where and how the reward-dependent modulation of neuronal activity occurs and how the reward-modulated signal is transformed into motor outputs. Recent studies suggest an important role of the basal ganglia. Sensorimotor/cognitive activities of neurons in the basal ganglia are strongly modulated by expected reward. Through their abundant outputs to the brain stem motor areas and the thalamocortical circuits, the basal ganglia appear capable of producing body movements based on expected reward. A good behavioral measure to test this hypothesis is saccadic eye movement because its brain stem mechanism has been extensively studied. Studies from our laboratory suggest that the basal ganglia play a key role in guiding the gaze to the location where reward is available. Neurons in the caudate nucleus and the substantia nigra pars reticulata are extremely sensitive to the positional difference in expected reward, which leads to a bias in excitability between the superior colliculi such that the saccade to the to-be-rewarded position occurs more quickly. It is suggested that the reward modulation occurs in the caudate where cortical inputs carrying spatial signals and dopaminergic inputs carrying reward-related signals are integrated. These data support a specific form of reinforcement learning theories, but also suggest further refinement of the theory.

Anatomical funneling, sparse connectivity and redundancy reduction in the neural networks of the basal ganglia

The major anatomical characteristics of the main axis of the basal ganglia are: (1) Numerical reduction in the number of neurons across layers of the feed-forward network, (2) lateral inhibitory connections within the layers, and (3) neuro-modulatory effects of dopamine and acetylcholine, both on the basal ganglia neurons and on the efficacy of information transmission along the basal ganglia axis. We recorded the simultaneous activity of neurons in the output stages of the basal ganglia as well as the activity of dopaminergic and cholinergic neurons during the performance of a probability decision-making task. We found that the functional messages of the cholinergic and dopaminergic neurons differ, and that the cholinergic message is less specific than that of the dopaminergic neurons. The output stage of the basal ganglia showed uncorrelated neuronal activity. We conclude that despite the huge numerical reduction from the cortex to the output nuclei of the basal ganglia, the activity of these nuclei represents an optimally compressed (uncorrelated) version of distinctive features of cortical information.

Modulation of striatal single units by expected reward: a spiny neuron model displaying dopamine-induced bistability

Single-unit activity in the neostriatum of awake monkeys shows a marked dependence on expected reward. Responses to visual cues differ when animals expect primary reinforcements, such as juice rewards, in comparison to secondary reinforcements, such as tones. The mechanism of this reward-dependent modulation has not been established experimentally. To assess the hypothesis that direct neuromodulatory effects of dopamine on spiny neurons can account for this modulation, we develop a computational model based on simplified representations of key ionic currents and their modulation by D1 dopamine receptor activation. This minimal model can be analyzed in detail. We find that D1-mediated increases of inward rectifying potassium and L-type calcium currents cause a bifurcation: the native up/down state behavior of the spiny neuron model becomes truly bistable, which modulates the peak firing rate and the duration of the up state and introduces a dependence of the response on the past state history. These generic consequences of dopamine neuromodulation through bistability can account for both reward-dependent enhancement and suppression of spiny neuron single-unit responses to visual cues. We validate the model by simulating responses to visual targets in a memory-guided saccade task; our results are in close agreement with the main features of the experimental data. Our model provides a conceptual framework for understanding the functional significance of the short-term neuromodulatory actions of dopamine on signal processing in the striatum.