Transition between encoding and consolidation/replay dynamics via cholinergic modulation of CAN current: A modeling study

A robotic model of hippocampal reverse replay for reinforcement learning

Hippocampal reverse replay, a phenomenon in which recently active hippocampal cells reactivate in the reverse order, is thought to contribute to learning, particularly reinforcement learning (RL), in animals. Here, we present a novel computational model which exploits reverse replay to improve stability and performance on a homing task. The model takes inspiration from the hippocampal-striatal network, and learning occurs via a three-factor RL rule. To augment this model with hippocampal reverse replay, we derived a policy gradient learning rule that associates place-cell activity with responses in cells representing actions and a supervised learning rule of the same form, interpreting the replay activity as a 'target' frequency. We evaluated the model using a simulated robot spatial navigation task inspired by the Morris water maze. Results suggest that reverse replay can improve performance stability over multiple trials. Our model exploits reverse reply as an additional source for propagating information about desirable synaptic changes, reducing the requirements for long-time scales in eligibility traces combined with low learning rates. We conclude that reverse replay can positively contribute to RL, although less stable learning is possible in its absence. Analogously, we postulate that reverse replay may enhance RL in the mammalian hippocampal-striatal system rather than provide its core mechanism.

Nonlinear relationship between CAN current and C a 2 + influx underpins synergistic action of muscarinic and NMDA receptors on bursts induction in midbrain dopaminergic neurons

Bursting of midbrain dopamine (DA) neurons is believed to represent an important reward signal that instructs and reinforces goal-directed behaviors. In DA neurons, many afferents, including cholinergic and glutamatergic inputs, induce bursting, and it is suggested that a synergy exists between these afferents in bursting induction. However, the underlying mechanisms of the role and the synergy of muscarinic receptors (mAChRs) and NMDA receptors (NMDARs) in bursting induction remain unclear. Present work bestowed analysis using a mathematical model of DA neurons to demonstrate the underlying mechanisms. Activation of mAChRs, leading to rapid translocation of TRPC channels to cell surface, recruited C a 2 + -activated nonspecific (CAN) current ( I CAN ), meanwhile NMDARs excitation triggered C a 2 + influx, which induced the positive feedback loop of C a 2 + and I CAN , respectively, yielded a robust ramping depolarization with a superimposed high-frequency spiking. In some DA cells, neither NMDARs nor mAChRs induced positive feedback loop unless they were activated simultaneously to induce bursting. Our experimental results verified those theoretical findings. These together unveil the underlying mechanisms of the role and synergy of mAChRs and NMDARs in bursting induction emerge from the nonlinear relationship between C a 2 + influx and I CAN . Given the diverse and complex nature of neural circuitry and the DA neuron heterogeneity, our work provides new insights to understand specific afferents, the synergy between those afferents, and the differences in intrinsic excitability to be integrated by the bursting to accurately characterize the dopamine signals in the valances of reward and reinforcement, and a broad spectrum of neuropsychiatric disorders.

Modeling awake hippocampal reactivations with model-based bidirectional search

Hippocampal offline reactivations during reward-based learning, usually categorized as replay events, have been found to be important for performance improvement over time and for memory consolidation. Recent computational work has linked these phenomena to the need to transform reward information into state-action values for decision making and to propagate it to all relevant states of the environment. Nevertheless, it is still unclear whether an integrated reinforcement learning mechanism could account for the variety of awake hippocampal reactivations, including variety in order (forward and reverse reactivated trajectories) and variety in the location where they occur (reward site or decision-point). Here, we present a model-based bidirectional search model which accounts for a variety of hippocampal reactivations. The model combines forward trajectory sampling from current position and backward sampling through prioritized sweeping from states associated with large reward prediction errors until the two trajectories connect. This is repeated until stabilization of state-action values (convergence), which could explain why hippocampal reactivations drastically diminish when the animal's performance stabilizes. Simulations in a multiple T-maze task show that forward reactivations are prominently found at decision-points while backward reactivations are exclusively generated at reward sites. Finally, the model can generate imaginary trajectories that are not allowed to the agent during task performance. We raise some experimental predictions and implications for future studies of the role of the hippocampo-prefronto-striatal network in learning.

Hippocampal-Cortical Memory Trace Transfer and Reactivation Through Cell-Specific Stimulus and Spontaneous Background Noise

The hippocampus plays important roles in memory formation and retrieval through sharp-wave-ripples. Recent studies have shown that certain neuron populations in the prefrontal cortex (PFC) exhibit coordinated reactivations during awake ripple events. These experimental findings suggest that the awake ripple is an important biomarker, through which the hippocampus interacts with the neocortex to assist memory formation and retrieval. However, the computational mechanisms of this ripple based hippocampal-cortical coordination are still not clear due to the lack of unified models that include both the hippocampal and cortical networks. In this work, using a coupled biophysical model of both CA1 and PFC, we investigate possible mechanisms of hippocampal-cortical memory trace transfer and the conditions that assist reactivation of the transferred memory traces in the PFC. To validate our model, we first show that the local field potentials generated in the hippocampus and PFC exhibit ripple range activities that are consistent with the recent experimental studies. Then we demonstrate that during ripples, sequence replays can successfully transfer the information stored in the hippocampus to the PFC recurrent networks. We investigate possible mechanisms of memory retrieval in PFC networks. Our results suggest that the stored memory traces in the PFC network can be retrieved through two different mechanisms, namely the cell-specific input representing external stimuli and non-specific spontaneous background noise representing spontaneous memory recall events. Importantly, in both cases, the memory reactivation quality is robust to network connection loss. Finally, we find out that the quality of sequence reactivations is enhanced by both increased number of SWRs and an optimal background noise intensity, which tunes the excitability of neurons to a proper level. Our study presents a mechanistic explanation for the memory trace transfer from the hippocampus to neocortex through ripple coupling in awake states and reports two different mechanisms by which the stored memory traces can be successfully retrieved.

Hippocampal Sequences During Exploration: Mechanisms and Functions

Although the hippocampus plays a critical role in spatial and episodic memories, the mechanisms underlying memory formation, stabilization, and recall for adaptive behavior remain relatively unknown. During exploration, within single cycles of the ongoing theta rhythm that dominates hippocampal local field potentials, place cells form precisely ordered sequences of activity. These neural sequences result from the integration of both external inputs conveying sensory-motor information, and intrinsic network dynamics possibly related to memory processes. Their endogenous replay during subsequent sleep is critical for memory consolidation. The present review discusses possible mechanisms and functions of hippocampal theta sequences during exploration. We present several lines of evidence suggesting that these neural sequences play a key role in information processing and support the formation of initial memory traces, and discuss potential functional distinctions between neural sequences emerging during theta vs. awake sharp-wave ripples.

Do TRPC channels support working memory? Comparing modulations of TRPC channels and working memory through G-protein coupled receptors and neuromodulators

Working memory is a crucial ability we use in daily life. However, the cellular mechanisms supporting working memory still remain largely unclear. A key component of working memory is persistent neural firing which is believed to serve short-term (hundreds of milliseconds up to tens of seconds) maintenance of necessary information. In this review, we will focus on the role of transient receptor potential canonical (TRPC) channels as a mechanism underlying persistent firing. Many years of in vitro work have been suggesting a crucial role of TRPC channels in working memory and temporal association tasks. If TRPC channels are indeed a central mechanism for working memory, manipulations which impair or facilitate working memory should have a similar effect on TRPC channel modulation. However, modulations of working memory and TRPC channels were never systematically compared, and it remains unanswered whether TRPC channels indeed contribute to working memory in vivo or not. In this article, we review the effects of G-protein coupled receptors (GPCR) and neuromodulators, including acetylcholine, noradrenalin, serotonin and dopamine, on working memory and TRPC channels. Based on comparisons, we argue that GPCR and downstream signaling pathways that activate TRPC, generally support working memory, while those that suppress TRPC channels impair it. However, depending on the channel types, areas, and systems tested, this is not the case in all studies. Further work to clarify involvement of specific TRPC channels in working memory tasks and how they are affected by neuromodulators is still necessary in the future.

Models of spatial and temporal dimensions of memory

Episodic memory involves coding of the spatial location and time of individual events. Coding of space and time is also relevant to working memory, spatial navigation, and the disambiguation of overlapping memory representations. Neurophysiological data demonstrate that neuronal activity codes the current, past and future location of an animal as well as temporal intervals within a task. Models have addressed how neural coding of space and time for memory function could arise, with both dimensions coded by the same neurons. Neural coding could depend upon network oscillatory and attractor dynamics as well as modulation of neuronal intrinsic properties. These models are relevant to the coding of space and time involving structures including the hippocampus, entorhinal cortex, retrosplenial cortex, striatum and parahippocampal gyrus, which have been implicated in both animal and human studies.

Neuromodulation of the Feedforward Dentate Gyrus-CA3 Microcircuit

The feedforward dentate gyrus-CA3 microcircuit in the hippocampus is thought to activate ensembles of CA3 pyramidal cells and interneurons to encode and retrieve episodic memories. The creation of these CA3 ensembles depends on neuromodulatory input and synaptic plasticity within this microcircuit. Here we review the mechanisms by which the neuromodulators aceylcholine, noradrenaline, dopamine, and serotonin reconfigure this microcircuit and thereby infer the net effect of these modulators on the processes of episodic memory encoding and retrieval.

Entorhinal-hippocampal neuronal circuits bridge temporally discontiguous events

The entorhinal cortex (EC)-hippocampal (HPC) network plays an essential role for episodic memory, which preserves spatial and temporal information about the occurrence of past events. Although there has been significant progress toward understanding the neural circuits underlying the spatial dimension of episodic memory, the relevant circuits subserving the temporal dimension are just beginning to be understood. In this review, we examine the evidence concerning the role of the EC in associating events separated by time--or temporal associative learning--with emphasis on the function of persistent activity in the medial entorhinal cortex layer III (MECIII) and their direct inputs into the CA1 region of HPC. We also discuss the unique role of Island cells in the medial entorhinal cortex layer II (MECII), which is a newly discovered direct feedforward inhibitory circuit to CA1. Finally, we relate the function of these entorhinal cortical circuits to recent findings concerning hippocampal time cells, which may collectively activate in sequence to bridge temporal gaps between discontiguous events in an episode.