Context coding in the mouse nucleus accumbens modulates motivationally relevant information

Neural activity in the nucleus accumbens (NAc) is thought to track fundamentally value-centric quantities linked to reward and effort. However, the NAc also contributes to flexible behavior in ways that are difficult to explain based on value signals alone, raising the question of if and how nonvalue signals are encoded in NAc. We recorded NAc neural ensembles while head-fixed mice performed an odor-based biconditional discrimination task where an initial discrete cue modulated the behavioral significance of a subsequently presented reward-predictive cue. We extracted single-unit and population-level correlates related to the cues and found value-independent coding for the initial, context-setting cue. This context signal occupied a population-level coding space orthogonal to outcome-related representations and was predictive of subsequent behaviorally relevant responses to the reward-predictive cues. Together, these findings support a gating model for how the NAc contributes to behavioral flexibility and provide a novel population-level perspective from which to view NAc computations.

Between-subject prediction reveals a shared representational geometry in the rodent hippocampus

The rodent hippocampus constructs statistically independent representations across environments ("global remapping") and assigns individual neuron firing fields to locations within an environment in an apparently random fashion, processes thought to contribute to the role of the hippocampus in episodic memory. This random mapping implies that it should be challenging to predict hippocampal encoding of a given experience in one subject based on the encoding of that same experience in another subject. Contrary to this prediction, we find that by constructing a common representational space across rats in which neural activity is aligned using geometric operations (rotation, reflection, and translation; "hyperalignment"), we can predict data of "right" trials (R) on a T-maze in a target rat based on (1) the "left" trials (L) of the target rat and (2) the relationship between L and R trials from a different source rat. These cross-subject predictions relied on ensemble activity patterns, including both firing rate and field location, and outperformed a number of control mappings, such as those based on permuted data that broke the relationship between L and R activity for individual neurons and those based solely on within-subject prediction. This work constitutes proof of principle for successful cross-subject prediction of ensemble activity patterns in the hippocampus and provides new insights in understanding how different experiences are structured, enabling further work identifying what aspects of experience encoding are shared versus unique to an individual.

A framework to identify structured behavioral patterns within rodent spatial trajectories

Animal behavior is highly structured. Yet, structured behavioral patterns-or "statistical ethograms"-are not immediately apparent from the full spatiotemporal data that behavioral scientists usually collect. Here, we introduce a framework to quantitatively characterize rodent behavior during spatial (e.g., maze) navigation, in terms of movement building blocks or motor primitives. The hypothesis that we pursue is that rodent behavior is characterized by a small number of motor primitives, which are combined over time to produce open-ended movements. We assume motor primitives to be organized in terms of two sparsity principles: each movement is controlled using a limited subset of motor primitives (sparse superposition) and each primitive is active only for time-limited, time-contiguous portions of movements (sparse activity). We formalize this hypothesis using a sparse dictionary learning method, which we use to extract motor primitives from rodent position and velocity data collected during spatial navigation, and successively to reconstruct past trajectories and predict novel ones. Three main results validate our approach. First, rodent behavioral trajectories are robustly reconstructed from incomplete data, performing better than approaches based on standard dimensionality reduction methods, such as principal component analysis, or single sparsity. Second, the motor primitives extracted during one experimental session generalize and afford the accurate reconstruction of rodent behavior across successive experimental sessions in the same or in modified mazes. Third, in our approach the number of motor primitives associated with each maze correlates with independent measures of maze complexity, hence showing that our formalism is sensitive to essential aspects of task structure. The framework introduced here can be used by behavioral scientists and neuroscientists as an aid for behavioral and neural data analysis. Indeed, the extracted motor primitives enable the quantitative characterization of the complexity and similarity between different mazes and behavioral patterns across multiple trials (i.e., habit formation). We provide example uses of this computational framework, showing how it can be used to identify behavioural effects of maze complexity, analyze stereotyped behavior, classify behavioral choices and predict place and grid cell displacement in novel environments.

Progress and issues in second-order analysis of hippocampal replay

Patterns of neural activity that occur spontaneously during sharp-wave ripple (SWR) events in the hippocampus are thought to play an important role in memory formation, consolidation and retrieval. Typical studies examining the content of SWRs seek to determine whether the identity and/or temporal order of cell firing is different from chance. Such 'first-order' analyses are focused on a single time point and template (map), and have been used to show, for instance, the existence of preplay. The major methodological challenge in first-order analyses is the construction and interpretation of different chance distributions. By contrast, 'second-order' analyses involve a comparison of SWR content between different time points, and/or between different templates. Typical second-order questions include tests of experience-dependence (replay) that compare SWR content before and after experience, and comparisons or replay between different arms of a maze. Such questions entail additional methodological challenges that can lead to biases in results and associated interpretations. We provide an inventory of analysis challenges for second-order questions about SWR content, and suggest ways of preventing, identifying and addressing possible analysis biases. Given evolving interest in understanding SWR content in more complex experimental scenarios and across different time scales, we expect these issues to become increasingly pervasive. This article is part of the Theo Murphy meeting issue 'Memory reactivation: replaying events past, present and future'.

Internally generated hippocampal sequences as a vantage point to probe future-oriented cognition

Information processing in the rodent hippocampus is fundamentally shaped by internally generated sequences (IGSs), expressed during two different network states: theta sequences, which repeat and reset at the ∼8 Hz theta rhythm associated with active behavior, and punctate sharp wave-ripple (SWR) sequences associated with wakeful rest or slow-wave sleep. A potpourri of diverse functional roles has been proposed for these IGSs, resulting in a fragmented conceptual landscape. Here, we advance a unitary view of IGSs, proposing that they reflect an inferential process that samples a policy from the animal's generative model, supported by hippocampus-specific priors. The same inference affords different cognitive functions when the animal is in distinct dynamical modes, associated with specific functional networks. Theta sequences arise when inference is coupled to the animal's action-perception cycle, supporting online spatial decisions, predictive processing, and episode encoding. SWR sequences arise when the animal is decoupled from the action-perception cycle and may support offline cognitive processing, such as memory consolidation, the prospective simulation of spatial trajectories, and imagination. We discuss the empirical bases of this proposal in relation to rodent studies and highlight how the proposed computational principles can shed light on the mechanisms of future-oriented cognition in humans.

The Head-Direction Signal Plays a Functional Role as a Neural Compass during Navigation

The rat limbic system contains head direction (HD) cells that fire according to heading in the horizontal plane, and these cells are thought to provide animals with an internal compass. Previous work has found that HD cell tuning correlates with behavior on navigational tasks, but a direct, causal link between HD cells and navigation has not been demonstrated. Here, we show that pathway-specific optogenetic inhibition of the nucleus prepositus caused HD cells to become directionally unstable under dark conditions without affecting the animals' locomotion. Then, using the same technique, we found that this decoupling of the HD signal in the absence of visual cues caused the animals to make directional homing errors and that the magnitude and direction of these errors were in a range that corresponded to the degree of instability observed in the HD signal. These results provide evidence that the HD signal plays a causal role as a neural compass in navigation.

Optimizing for generalization in the decoding of internally generated activity in the hippocampus

The decoding of a sensory or motor variable from neural activity benefits from a known ground truth against which decoding performance can be compared. In contrast, the decoding of covert, cognitive neural activity, such as occurs in memory recall or planning, typically cannot be compared to a known ground truth. As a result, it is unclear how decoders of such internally generated activity should be configured in practice. We suggest that if the true code for covert activity is unknown, decoders should be optimized for generalization performance using cross-validation. Using ensemble recording data from hippocampal place cells, we show that this cross-validation approach results in different decoding error, different optimal decoding parameters, and different distributions of error across the decoded variable space. In addition, we show that a minor modification to the commonly used Bayesian decoding procedure, which enables the use of spike density functions, results in substantially lower decoding errors. These results have implications for the interpretation of covert neural activity, and suggest easy-to-implement changes to commonly used procedures across domains, with applications to hippocampal place cells in particular. © 2017 Wiley Periodicals, Inc.

Low- and high-gamma oscillations deviate in opposite directions from zero-phase synchrony in the limbic corticostriatal loop

The loop structure of cortico-striatal anatomy in principle enables both descending (cortico-striatal) and ascending (striato-cortical) influences, but the factors that regulate the flow of information in these loops are not known. We report that low- and high-gamma oscillations (∼50 and ∼80 Hz, respectively) in the local field potential of freely moving rats are highly synchronous between the infralimbic region of the medial prefrontal cortex (mPFC) and the ventral striatum (vStr). Strikingly, high-gamma oscillations in mPFC preceded those in vStr, whereas low-gamma oscillations in mPFC lagged those in vStr, with short (∼1 ms) time lags. These systematic deviations from zero-phase synchrony were consistent across measures based on amplitude cross-correlation and phase slopes and were robustly maintained between behavioral states and different individual subjects. Furthermore, low- and high-gamma oscillations were associated with distinct ensemble spiking patterns in vStr, even when controlling for overt behavioral differences and slow changes in neural activity. These results imply that neural activity in vStr and mPFC is tightly coupled at the gamma timescale and raise the intriguing possibility that frequency-specific deviations from this coupling may signal transient leader-follower switches.

Internally generated sequences in learning and executing goal-directed behavior

A network of brain structures including hippocampus (HC), prefrontal cortex, and striatum controls goal-directed behavior and decision making. However, the neural mechanisms underlying these functions are unknown. Here, we review the role of 'internally generated sequences': structured, multi-neuron firing patterns in the network that are not confined to signaling the current state or location of an agent, but are generated on the basis of internal brain dynamics. Neurophysiological studies suggest that such sequences fulfill functions in memory consolidation, augmentation of representations, internal simulation, and recombination of acquired information. Using computational modeling, we propose that internally generated sequences may be productively considered a component of goal-directed decision systems, implementing a sampling-based inference engine that optimizes goal acquisition at multiple timescales of on-line choice, action control, and learning.

Adaptive properties of differential learning rates for positive and negative outcomes

The concept of the reward prediction error-the difference between reward obtained and reward predicted-continues to be a focal point for much theoretical and experimental work in psychology, cognitive science, and neuroscience. Models that rely on reward prediction errors typically assume a single learning rate for positive and negative prediction errors. However, behavioral data indicate that better-than-expected and worse-than-expected outcomes often do not have symmetric impacts on learning and decision-making. Furthermore, distinct circuits within cortico-striatal loops appear to support learning from positive and negative prediction errors, respectively. Such differential learning rates would be expected to lead to biased reward predictions and therefore suboptimal choice performance. Contrary to this intuition, we show that on static "bandit" choice tasks, differential learning rates can be adaptive. This occurs because asymmetric learning enables a better separation of learned reward probabilities. We show analytically how the optimal learning rate asymmetry depends on the reward distribution and implement a biologically plausible algorithm that adapts the balance of positive and negative learning rates from experience. These results suggest specific adaptive advantages for separate, differential learning rates in simple reinforcement learning settings and provide a novel, normative perspective on the interpretation of associated neural data.

Risk-responsive orbitofrontal neurons track acquired salience

Decision making is impacted by uncertainty and risk (i.e., variance). Activity in the orbitofrontal cortex, an area implicated in decision making, covaries with these quantities. However, this activity could reflect the heightened salience of situations in which multiple outcomes-reward and reward omission-are expected. To resolve these accounts, rats were trained to respond to cues predicting 100%, 67%, 33%, or 0% reward. Consistent with prior reports, some orbitofrontal neurons fired differently in anticipation of uncertain (33% and 67%) versus certain (100% and 0%) reward. However, over 90% of these neurons also fired differently prior to 100% versus 0% reward (or baseline) or prior to 33% versus 67% reward. These responses are inconsistent with risk but fit well with the representation of acquired salience linked to the sum of cue-outcome and cue-no-outcome associative strengths. These results expand our understanding of how the orbitofrontal cortex might regulate learning and behavior.

Ventral striatum: a critical look at models of learning and evaluation

Extensive evidence implicates the ventral striatum in multiple distinct facets of action selection. Early work established a role in modulating ongoing behavior, as engaged by the energizing and directing influences of motivationally relevant cues and the willingness to expend effort in order to obtain reward. More recently, reinforcement learning models have suggested the notion of ventral striatum primarily as an evaluation step during learning, which serves as a critic to update a separate actor. Recent computational and experimental work may provide a resolution to the differences between these two theories through a careful parsing of behavior and the instrinsic heterogeneity that characterizes this complex structure.

Theta phase precession in rat ventral striatum links place and reward information

A functional interaction between the hippocampal formation and the ventral striatum is thought to contribute to the learning and expression of associations between places and rewards. However, the mechanism of how such associations may be learned and used is currently unknown. We recorded neural ensembles and local field potentials from the ventral striatum and CA1 simultaneously as rats ran a modified T-maze. Theta-modulated cells in ventral striatum almost invariably showed firing phase precession relative to the hippocampal theta rhythm. Across the population of ventral striatal cells, phase precession was preferentially associated with an anticipatory ramping of activity up to the reward sites. In contrast, CA1 population activity and phase precession were distributed more uniformly. Ventral striatal phase precession was stronger to hippocampal than ventral striatal theta and was accompanied by increased theta coherence with hippocampus, suggesting that this effect is hippocampally derived. These results suggest that the firing phase of ventral striatal neurons contains motivationally relevant information and that phase precession serves to bind hippocampal place representations to ventral striatal representations of reward.

Integrating early results on ventral striatal gamma oscillations in the rat

A vast literature implicates the ventral striatum in the processing of reward-related information and in mediating the impact of such information on behavior. It is characterized by heterogeneity at the local circuit, connectivity, and functional levels. A tool for dissecting this complex structure that has received relatively little attention until recently is the analysis of ventral striatal local field potential oscillations, which are more prominent in the gamma band compared to the dorsal striatum. Here we review recent results on gamma oscillations recorded from freely moving rats. Ventral striatal gamma separates into distinct frequency bands (gamma-50 and gamma-80) with distinct behavioral correlates, relationships to different inputs, and separate populations of phase-locked putative fast-spiking interneurons. Fast switching between gamma-50 and gamma-80 occurs spontaneously but is influenced by reward delivery as well as the application of dopaminergic drugs. These results provide novel insights into ventral striatal processing and highlight the importance of considering fast-timescale dynamics of ventral striatal activity.

Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task

Decision-making studies across different domains suggest that decisions can arise from multiple, parallel systems in the brain: a flexible system utilizing action-outcome expectancies and a more rigid system based on situation-action associations. The hippocampus, ventral striatum, and dorsal striatum make unique contributions to each system, but how information processing in each of these structures supports these systems is unknown. Recent work has shown covert representations of future paths in hippocampus and of future rewards in ventral striatum. We developed analyses in order to use a comparative methodology and apply the same analyses to all three structures. Covert representations of future paths and reward were both absent from the dorsal striatum. In contrast, dorsal striatum slowly developed situation representations that selectively represented action-rich parts of the task. This triple dissociation suggests that the different roles these structures play are due to differences in information-processing mechanisms.

Evidence for the use of an internal sense of direction in homing

Path integration, the ability to maintain a representation of location and direction on the basis of internal cues, is thought to be important for navigation and the learning of spatial relationships. Representations of location and direction in the brain, such as head direction cells, grid cells, and place cells in the limbic system, are thought to underlie navigation by path integration. While this idea is generally consistent with lesion studies, the relationship between such neural activity and behavior has not been studied on a task where animals demonstrably use a path integration strategy. Here we report the development of such a task in rats: by slowly rotating rats before their return to a trial-unique home base, we could show subjects relied on internal cues only to navigate. To illustrate how this task can be combined with recording, we show examples of simultaneously recorded head direction cells in which neural activity is closely related to rats' homing direction. These results support the notion that rats can navigate by path integration, that this ability depends on head direction cells, and suggest a convenient behavioral paradigm for investigating the neural basis of navigation.

Expectancies in decision making, reinforcement learning, and ventral striatum

Decisions can arise in different ways, such as from a gut feeling, doing what worked last time, or planful deliberation. Different decision-making systems are dissociable behaviorally, map onto distinct brain systems, and have different computational demands. For instance, “model-free” decision strategies use prediction errors to estimate scalar action values from previous experience, while “model-based” strategies leverage internal forward models to generate and evaluate potentially rich outcome expectancies. Animal learning studies indicate that expectancies may arise from different sources, including not only forward models but also Pavlovian associations, and the flexibility with which such representations impact behavior may depend on how they are generated. In the light of these considerations, we review the results of van der Meer and Redish (2009a), who found that ventral striatal neurons that respond to reward delivery can also be activated at other points, notably at a decision point where hippocampal forward representations were also observed. These data suggest the possibility that ventral striatal reward representations contribute to model-based expectancies used in deliberative decision making.

Hippocampal replay is not a simple function of experience

Replay of behavioral sequences in the hippocampus during sharp wave ripple complexes (SWRs) provides a potential mechanism for memory consolidation and the learning of knowledge structures. Current hypotheses imply that replay should straightforwardly reflect recent experience. However, we find these hypotheses to be incompatible with the content of replay on a task with two distinct behavioral sequences (A and B). We observed forward and backward replay of B even when rats had been performing A for >10 min. Furthermore, replay of nonlocal sequence B occurred more often when B was infrequently experienced. Neither forward nor backward sequences preferentially represented highly experienced trajectories within a session. Additionally, we observed the construction of never-experienced novel-path sequences. These observations challenge the idea that sequence activation during SWRs is a simple replay of recent experience. Instead, replay reflected all physically available trajectories within the environment, suggesting a potential role in active learning and maintenance of the cognitive map.

Adaptive integration in the visual cortex by depressing recurrent cortical circuits

Neurons in the visual cortex receive a large amount of input from recurrent connections, yet the functional role of these connections remains unclear. Here we explore networks with strong recurrence in a computational model and show that short-term depression of the synapses in the recurrent loops implements an adaptive filter. This allows the visual system to respond reliably to deteriorated stimuli yet quickly to high-quality stimuli. For low-contrast stimuli, the model predicts long response latencies, whereas latencies are short for high-contrast stimuli. This is consistent with physiological data showing that in higher visual areas, latencies can increase more than 100 ms at low contrast compared to high contrast. Moreover, when presented with briefly flashed stimuli, the model predicts stereotypical responses that outlast the stimulus, again consistent with physiological findings. The adaptive properties of the model suggest that the abundant recurrent connections found in visual cortex serve to adapt the network's time constant in accordance with the stimulus and normalizes neuronal signals such that processing is as fast as possible while maintaining reliability.

Low and High Gamma Oscillations in Rat Ventral Striatum have Distinct Relationships to Behavior, Reward, and Spiking Activity on a Learned Spatial Decision Task

Local field potential (LFP) oscillations in the brain reflect organization thought to be important for perception, attention, movement, and memory. In the basal ganglia, including dorsal striatum, dysfunctional LFP states are associated with Parkinson's disease, while in healthy subjects, dorsal striatal LFPs have been linked to decision-making processes. However, LFPs in ventral striatum have been less studied. We report that in rats running a spatial decision task, prominent gamma-50 (45–55 Hz) and gamma-80 (70–85 Hz) oscillations in ventral striatum had distinct relationships to behavior, task events, and spiking activity. Gamma-50 power increased sharply following reward delivery and before movement initiation, while in contrast, gamma-80 power ramped up gradually to reward locations. Gamma-50 power was low and contained little structure during early learning, but rapidly developed a stable pattern, while gamma-80 power was initially high before returning to a stable level within a similar timeframe. Putative fast-spiking interneurons (FSIs) showed phase, firing rate, and coherence relationships with gamma-50 and gamma-80, indicating that the observed LFP patterns are locally relevant. Furthermore, in a number of FSIs such relationships were specific to gamma-50 or gamma-80, suggesting that partially distinct FSI populations mediate the effects of gamma-50 and gamma-80.

Covert Expectation-of-Reward in Rat Ventral Striatum at Decision Points

Flexible decision-making strategies (such as planning) are a key component of adaptive behavior, yet their neural mechanisms have remained resistant to experimental analysis. Theories of planning require prediction and evaluation of potential future rewards, suggesting that reward signals may covertly appear at decision points. To test this idea, we recorded ensembles of ventral striatal neurons on a spatial decision task, in which hippocampal ensembles are known to represent future possibilities at decision points. We found representations of reward which were not only activated at actual reward delivery sites, but also at a high-cost choice point and before error correction. This expectation-of-reward signal at decision points was apparent at both the single cell and the ensemble level, and vanished with behavioral automation. We conclude that ventral striatal representations of reward are more dynamic than suggested by previous reports of reward- and cue-responsive cells, and may provide the necessary signal for evaluation of internally generated possibilities considered during flexible decision-making.

Exploring the role of context-dependent hippocampal activity in spatial alternation behavior

In a continuous T-maze spatial alternation task, CA1 place cells fire differentially on the stem of the maze as rats are performing left- and right-turn trials (Wood et al. (2000) Neuron 27:623-633). This context-dependent hippocampal activity provides a potential mechanism by which animals could solve the alternation task, as it provides a cue that could prime the appropriate goal choice. The aim of this study was to examine the relationship between context-dependent hippocampal activity and spatial alternation behavior. We report that rats with complete lesions of the hippocampus learn and perform the spatial alternation task as well as controls if there is no delay between trials, suggesting that the observed context-dependent hippocampal activity does not mediate alternation behavior in this task. However lesioned rats are significantly impaired when delays of 2 or 10 s are interposed. Recording experiments reveal that context-dependent hippocampal activity occurs in both the delay and no-delay versions of the task, but that in the delay version it occurs during the delay period, and not on the stem of the maze. These data are consistent with a role for context-dependent hippocampal activity in delayed spatial alternation, but suggest that, according to specific task demands and memory load, the activity may be generated by different mechanisms and/or in different brain structures.

Integrating hippocampus and striatum in decision-making

Learning and memory and navigation literatures emphasize interactions between multiple memory systems: a flexible, planning-based system and a rigid, cached-value system. This has profound implications for decision-making. Recent conceptualizations of flexible decision-making employ prospection and projection arising from a network involving the hippocampus. Recent recordings from rodent hippocampus in decision-making situations have found transient forward-shifted representations. Evaluation of that prediction and subsequent action-selection probably occurs downstream (e.g. in orbitofrontal cortex, in ventral and dorsomedial striatum). Classically, striatum has been identified as a crucial component of the less-flexible, incremental system. Current evidence, however, suggests that striatum is involved in both flexible and stimulus-response decision-making, with dorsolateral striatum involved in stimulus-response strategies and ventral and dorsomedial striatum involved in goal-directed strategies.

Anticipation in the Rodent Head Direction System Can Be Explained by an Interaction of Head Movements and Vestibular Firing Properties

The rodent head-direction (HD) system, which codes for the animal's head direction in the horizontal plane, is thought to be critically involved in spatial navigation. Electrophysiological recording studies have shown that HD cells can anticipate the animal's HD by up to 75–80 ms. The origin of this anticipation is poorly understood. In this modeling study, we provide a novel explanation for HD anticipation that relies on the firing properties of neurons afferent to the HD system. By incorporating spike rate adaptation and postinhibitory rebound as observed in medial vestibular nucleus neurons, our model produces realistic anticipation on a large corpus of rat movement data. In addition, HD anticipation varies between recording sessions of the same cell, between active and passive movement, and between different studies. Such differences do not appear to be correlated with behavioral variables and cannot be accounted for using earlier models. In the present model, anticipation depends on the power spectrum of the head movements. By direct comparison with recording data, we show that the model explains 60–80% of the observed anticipation variability. We conclude that HD afferent dynamics and the statistics of rat head movements are important in generating HD anticipation. This result contributes to understanding the functional circuitry of the HD system and has methodological implications for studies of HD anticipation.