Scalar timing in memory: A temporal map in the hippocampus

Time-scale invariant contingency yields one-shot reinforcement learning despite extremely long delays to reinforcement

Reinforcement learning inspires much theorizing in neuroscience, cognitive science, machine learning, and AI. A central question concerns the conditions that produce the perception of a contingency between an action and reinforcement-the assignment-of-credit problem. Contemporary models of associative and reinforcement learning do not leverage the temporal metrics (measured intervals). Our information-theoretic approach formalizes contingency by time-scale invariant temporal mutual information. It predicts that learning may proceed rapidly even with extremely long action-reinforcer delays. We show that rats can learn an action after a single reinforcement, even with a 16-min delay between the action and reinforcement (15-fold longer than any delay previously shown to support such learning). By leveraging metric temporal information, our solution obviates the need for windows of associability, exponentially decaying eligibility traces, microstimuli, or distributions over Bayesian belief states. Its three equations have no free parameters; they predict one-shot learning without iterative simulation.

Resource Allocation in the Noise-Free Striatal Beat Frequency Model of Interval Timing

The Striatal Beat Frequency (SBF) model of interval timing uses many neural oscillators, presumably located in the frontal cortex (FC), to produce beats at a specific criterion time Tc. The coincidence detection produces the beats in the basal ganglia spiny neurons by comparing the current state of the FC neural oscillators against the long-term memory values stored at reinforcement time Tc. The neurobiologically realistic SBF model has been previously used for producing precise and scalar timing in the presence of noise. Here we simplified the SBF model to gain insight into the resource allocation problem in interval timing networks. Specifically, we used a noise-free SBF model to explore the lower limits of the number of neural oscillators required for producing accurate timing. Using abstract sine-wave neural oscillators in the SBF-sin model, we found that the lower limit of the number of oscillators needed is proportional to the criterion time Tc and the frequency span (fmax − fmin) of the FC neural oscillators. Using biophysically realistic Morris–Lecar model neurons in the SBF-ML model, the lower bound increased by one to two orders of magnitude compared to the SBF-sin model.

Ventral hippocampal shock encoding modulates the expression of trace cued fear

The hippocampus is crucial for associative fear learning when the anticipation of threat requires temporal or contextual binding of predictive stimuli as in trace and contextual fear conditioning. Compared with the dorsal hippocampus, far less is known about the contribution of the ventral hippocampus to fear learning. The ventral hippocampus, which is highly interconnected with defensive and emotional networks, has a prominent role in both innate and learned affective behaviors including anxiety, fear, and reward. Lesions or temporary inactivation of the ventral hippocampus impair both cued and contextual fear learning, but whether the ventral hippocampal role in learning is driven by affective processing, associative encoding, or both is not clear. Here, we used trace fear conditioning in mixed sex cohorts to assess the contribution of shock-encoding to the acquisition of cued and contextual fear memories. Trace conditioning requires subjects to associate an auditory conditional stimulus (CS) with a shock unconditional stimulus (UCS) that are separated in time by a 20-s trace interval. We first recorded neuronal activity in the ventral hippocampus during trace fear conditioning and found that ventral CA1 predominantly encoded the shock reinforcer. Potentiated firing to the CS was evident at testing, but no encoding of the trace interval was observed. We then tested the necessity of shock encoding for conditional fear acquisition by optogenetically silencing ventral hippocampal activity during the UCS on each trial of training. Contrary to our predictions, preventing hippocampal shock-evoked firing did not impair associative fear. Instead, it led to a more prolonged expression of CS freezing across test trials, an effect observed in males, but not females. Contextual fear learning was largely intact, although a subset of animals in each sex were differentially affected by shock-silencing. Taken together, the results show that shock encoding in the ventral hippocampus modulates the expression of learned fear in a sex-specific manner.

Five Breakthroughs: A First Approximation of Brain Evolution From Early Bilaterians to Humans

Retracing the evolutionary steps by which human brains evolved can offer insights into the underlying mechanisms of human brain function as well as the phylogenetic origin of various features of human behavior. To this end, this article presents a model for interpreting the physical and behavioral modifications throughout major milestones in human brain evolution. This model introduces the concept of a "breakthrough" as a useful tool for interpreting suites of brain modifications and the various adaptive behaviors these modifications enabled. This offers a unique view into the ordered steps by which human brains evolved and suggests several unique hypotheses on the mechanisms of human brain function.

What Behavioral Abilities Emerged at Key Milestones in Human Brain Evolution? 13 Hypotheses on the 600-Million-Year Phylogenetic History of Human Intelligence

This paper presents 13 hypotheses regarding the specific behavioral abilities that emerged at key milestones during the 600-million-year phylogenetic history from early bilaterians to extant humans. The behavioral, intellectual, and cognitive faculties of humans are complex and varied: we have abilities as diverse as map-based navigation, theory of mind, counterfactual learning, episodic memory, and language. But these faculties, which emerge from the complex human brain, are likely to have evolved from simpler prototypes in the simpler brains of our ancestors. Understanding the order in which behavioral abilities evolved can shed light on how and why our brains evolved. To propose these hypotheses, I review the available data from comparative psychology and evolutionary neuroscience.

The dorsal hippocampus is required for the formation of long-term duration memories in rats

Interval timing—the perception of durations mainly in seconds or minutes—is a ubiquitous behavior in organisms. Animal studies have suggested that the hippocampus plays an essential role in duration memory; however, the memory processes involved are unclear. To clarify the role of the dorsal hippocampus in the acquisition of long‐term duration memories, we adapted the “time‐shift paradigm” to a peak‐interval procedure. After a sufficient number of training with an initial target duration (20 s), the rats underwent “shift sessions” with a new target duration (40 s) under a muscimol (0.5 µg per side) infusion into the bilateral dorsal hippocampus. The memory of the new target duration was then tested in drug‐free “probe sessions,” including trials in which no lever presses were reinforced. In the probe sessions, the mean response rate distribution of the muscimol group was located leftward to the control group, but these two response rate distributions were superimposed on the standardized time axis, suggesting a scalar property. In the session‐by‐session analysis, the mean peak time (an index of timing accuracy) of the muscimol group was lower than that of the control group in the probe sessions, but not in the shift sessions. These findings suggest that the dorsal hippocampus is required for the formation of long‐term duration memories within the range of interval timing.

Is the scalar property of interval timing preserved after hippocampus lesions?

Time perception is fundamental for decision-making, adaptation, and survival. In the peak-interval (PI) paradigm, one of the critical features of time perception is its scale invariance, i.e., the error in time estimation increases linearly with the to-be-timed interval. Brain lesions can profoundly alter time perception, but do they also change its scalar property? In particular, hippocampus (HPC) lesions affect the memory of the reinforced durations. Experiments found that ventral hippocampus (vHPC) lesions shift the perceived durations to longer values while dorsal hippocampus (dHPC) lesions produce opposite effects. Here we used our implementation of the Striatal Beat Frequency (SBFML) model with biophysically realistic Morris-Lecar (ML) model neurons and a topological map of HPC memory to predict analytically and verify numerically the effect of HPC lesions on scalar property. We found that scalar property still holds after both vHPC and dHPC lesions in our SBFML-HPC network simulation. Our numerical results show that PI durations are shifted in the correct direction and match the experimental results. In our simulations, the relative peak shift of the behavioral response curve is controlled by two factors: (1) the lesion size, and (2) the cellular-level memory variance of the temporal durations stored in the HPC. The coefficient of variance (CV) of the behavioral response curve remained constant over the tested durations of PI procedure, which suggests that scalar property is not affected by HPC lesions.

Time as the fourth dimension in the hippocampus

Experiences of animal and human beings are structured by the continuity of space and time coupled with the uni-directionality of time. In addition to its pivotal position in spatial processing and navigation, the hippocampal system also plays a central, multiform role in several types of temporal processing. These include timing and sequence learning, at scales ranging from meso-scales of seconds to macro-scales of minutes, hours, days and beyond, encompassing the classical functions of short term memory, working memory, long term memory, and episodic memories (comprised of information about when, what, and where). This review article highlights the principal findings and behavioral contexts of experiments in rats showing: 1) timing: tracking time during delays by hippocampal 'time cells' and during free behavior by hippocampal-afferent lateral entorhinal cortex ramping cells; 2) 'online' sequence processing: activity coding sequences of events during active behavior; 3) 'off-line' sequence replay: during quiescence or sleep, orderly reactivation of neuronal assemblies coding awake sequences. Studies in humans show neurophysiological correlates of episodic memory comparable to awake replay. Neural mechanisms are discussed, including ion channel properties, plateau and ramping potentials, oscillations of excitation and inhibition of population activity, bursts of high amplitude discharges (sharp wave ripples), as well as short and long term synaptic modifications among and within cell assemblies. Specifically conceived neural network models will suggest processes supporting the emergence of scalar properties (Weber's law), and include different classes of feedforward and recurrent network models, with intrinsic hippocampal coding for 'transitions' (sequencing of events or places).

Activation of cannabinoid type 1 receptors decreases the synchronization of local field potential oscillations in the hippocampus and entorhinal cortex and prolongs the interresponse time during a differential-reinforcement-of-low-rate task

Marijuana intoxication impairs neurocognitive functions. Common side effects of consuming cannabis include time distortion and memory loss. However, the underlying neurophysiological mechanisms involved in these effects remain unclear. We hypothesized that communication between the hippocampal CA1 region and medial entorhinal cortex (MEC) is essential for the transmission of temporal-associated information. We used a differential-reinforcement-of-low-rate (DRL) task, which requires subjects to press a lever at an optimal time point, to correlate the distributions of interresponse time (IRT) with local field potentials (LFPs) recorded in the CA1 and MEC under the effects of a cannabinoid type 1 (CB1) receptor agonist. We used a DRL 10-s schedule and trained the rats to withhold for 10 s before pressing a lever. Our data showed that the percentage of 12.4- to 14-s IRT events rose after activation of CB1 receptors in the MEC. In addition, gamma amplitude synchronization and CA1 theta phase-MEC gamma amplitude coupling decreased during the 6- to 14-s IRT events. These results suggest that activation of CB1 receptors in the MEC disrupt the functional connectivity between the CA1 and the MEC. This inefficient communication may result in increased IRT during a DRL schedule. Overall, we postulate that marijuana intoxication impairs the communication between the CA1 and MEC and influences behavioral performances that require precise timing ability.

The differential role of the dorsal hippocampus in initiating and terminating timed responses: A lesion study using the switch-timing task

This study investigated the role of the dorsal hippocampus (dHPC) in the temporal entrainment of behavior, while addressing limitations of previous evidence from peak procedure experiments. Rats were first trained on a switch-timing task in which food was obtained from one of two concurrently available levers; one lever was effective after 8 s and the other after 16  s. After performance stabilized, rats underwent either bilateral NMDA lesions of the dHPC or sham lesions. After recovery, switch-timing training resumed. In a subsequent condition, the switch-timing task was modified such that food was available after either 8 or 32 s. Although dHPC lesions had subtle and complex effects on when rats stopped seeking for food at the 8-s lever (departures), it more systematically reduced the time when rats started seeking for food at the 16-s and 32-s lever (switches). No systematic effect of dHPC lesions were observed on the coefficient of quartile variation (normalized dispersion) of latencies to switch. Within the context of the pacemaker-accumulator framework of interval timing, these findings suggest that partially or wholly independent mechanisms control the initiation and termination of timed responses, and that the dHPC is primarily involved in encoding the time to start responding.

The hippocampus contributes to temporal duration memory in the context of event sequences: A cross-species perspective

Although a large body of research has implicated the hippocampus in the processing of memory for temporal duration, there is an exigent degree of inconsistency across studies that obfuscates the precise contributions of this structure. To shed light on this issue, the present review article surveys both historical and recent cross-species evidence emanating from a wide variety of experimental paradigms, identifying areas of convergence and divergence. We suggest that while factors such as time-scale (e.g. the length of durations involved) and the nature of memory processing (e.g. prospective vs. retrospective memory) are very helpful in the interpretation of existing data, an additional important consideration is the context in which the duration information is experienced and processed, with the hippocampus being preferentially involved in memory for durations that are embedded within a sequence of events. We consider the mechanisms that may underpin temporal duration memory and how the same mechanisms may contribute to memory for other aspects of event sequences such as temporal order.

Interval timing is disrupted in female 5xFAD mice: An indication of altered memory processes

Temporal information processing in the seconds-to-minutes range is disrupted in patients with Alzheimer's disease (AD). In this study, we investigated the timing behavior of the 5xFAD mouse model of AD in the peak interval (PI) procedure. Nine-month-old female mice were trained with sucrose solution reinforcement for their first response after a fixed-interval (FI) and tested in the inter-mixed non-reinforced PI trials that lasted longer than FI. Timing performance indices were estimated from steady-state timed anticipatory nose-poking responses in the PI trials. We found that the time of maximal reward expectancy (peak time) of the 5xFAD mice was significantly earlier than that of the wild-type (WT) controls with no differences in other indices of timing performance. These behavioral differences corroborate the findings of previous studies on the disruption of temporal associative memory abilities of 5xFAD mice and can be accounted for by the scalar timing theory based on altered long-term memory consolidation of temporal information in the 5xFAD mice. This is the first study to directly show an interval timing phenotype in a genetic mouse model of AD.

A Population-Based Model of the Temporal Memory in the Hippocampus

Spatial and temporal dimensions are fundamental for orientation, adaptation, and survival of organisms. Hippocampus has been identified as the main neuroanatomical structure involved both in space and time perception and their internal representation. Dorsal hippocampus lesions showed a leftward shift (toward shorter durations) in peak-interval procedures, whereas ventral lesions shifted the peak time toward longer durations. We previously explained hippocampus lesion experimental findings by assuming a topological map model of the hippocampus with shorter durations memorized ventrally and longer durations more dorsal. Here we suggested a possible connection between the abstract topological maps model of the hippocampus that stored reinforcement times in a spatially ordered memory register and the "time cells" of the hippocampus. In this new model, the time cells provide a uniformly distributed time basis that covers the entire to-be-learned temporal duration. We hypothesized that the topological map of the hippocampus stores the weights that reflect the contribution of each time cell to the average temporal field that determines the behavioral response. The temporal distance between the to-be-learned criterion time and the time of the peak activity of each time cell provides the error signal that determines the corresponding weight correction. Long-term potentiation/depression could enhance/weaken the weights associated to the time cells that peak closer/farther to the criterion time. A coincidence detector mechanism, possibly under the control of the dopaminergic system, could be involved in our suggested error minimization and learning algorithm.

The Medial Prefrontal Cortex Shapes Dopamine Reward Prediction Errors under State Uncertainty

Animals make predictions based on currently available information. In natural settings, sensory cues may not reveal complete information, requiring the animal to infer the "hidden state" of the environment. The brain structures important in hidden state inference remain unknown. A previous study showed that midbrain dopamine neurons exhibit distinct response patterns depending on whether reward is delivered in 100% (task 1) or 90% of trials (task 2) in a classical conditioning task. Here we found that inactivation of the medial prefrontal cortex (mPFC) affected dopaminergic signaling in task 2, in which the hidden state must be inferred ("will reward come or not?"), but not in task 1, where the state was known with certainty. Computational modeling suggests that the effects of inactivation are best explained by a circuit in which the mPFC conveys inference over hidden states to the dopamine system. VIDEO ABSTRACT.