An adaptive drift-diffusion model of interval timing dynamics

Complementary cognitive roles for D2-MSNs and D1-MSNs in interval timing

The role of striatal pathways in cognitive processing is unclear. We studied dorsomedial striatal cognitive processing during interval timing, an elementary cognitive task that requires mice to estimate intervals of several seconds, which involves working memory for temporal rules as well as attention to the passage of time. We harnessed optogenetic tagging to record from striatal D2-dopamine receptor-expressing medium spiny neurons (D2-MSNs) in the indirect pathway and from D1-dopamine receptor-expressing MSNs (D1-MSNs) in the direct pathway. We found that D2-MSNs and D1-MSNs exhibited opposing dynamics over temporal intervals as quantified by principal component analyses and trial-by-trial generalized linear models. MSN recordings helped construct and constrain a four-parameter drift-diffusion computational model. This model predicted that disrupting either D2-MSN or D1-MSNs would increase interval timing response times and alter MSN firing. In line with this prediction, we found that optogenetic inhibition or pharmacological disruption of either D2-MSNs or D1-MSNs increased response times. Pharmacologically disrupting D2-MSNs or D1-MSNs also increased response times, shifted MSN dynamics, and degraded trial-by-trial temporal decoding. Together, our findings demonstrate that D2-MSNs and D1-MSNs make complementary contributions to interval timing despite opposing dynamics, implying that striatal direct and indirect pathways work together to shape temporal control of action. These data provide novel insight into basal ganglia cognitive operations beyond movement and have implications for a broad range of human striatal diseases and for therapies targeting striatal pathways.

Oscillation/Coincidence-Detection Models of Reward-Related Timing in Corticostriatal Circuits

The major tenets of beat-frequency/coincidence-detection models of reward-related timing are reviewed in light of recent behavioral and neurobiological findings. This includes the emphasis on a core timing network embedded in the motor system that is comprised of a corticothalamic-basal ganglia circuit. Therein, a central hub provides timing pulses (i.e., predictive signals) to the entire brain, including a set of distributed satellite regions in the cerebellum, cortex, amygdala, and hippocampus that are selectively engaged in timing in a manner that is more dependent upon the specific sensory, behavioral, and contextual requirements of the task. Oscillation/coincidence-detection models also emphasize the importance of a tuned ‘perception’ learning and memory system whereby target durations are detected by striatal networks of medium spiny neurons (MSNs) through the coincidental activation of different neural populations, typically utilizing patterns of oscillatory input from the cortex and thalamus or derivations thereof (e.g., population coding) as a time base. The measure of success of beat-frequency/coincidence-detection accounts, such as the Striatal Beat-Frequency model of reward-related timing (SBF), is their ability to accommodate new experimental findings while maintaining their original framework, thereby making testable experimental predictions concerning diagnosis and treatment of issues related to a variety of dopamine-dependent basal ganglia disorders, including Huntington’s and Parkinson’s disease.

Rodents monitor their error in self-generated duration on a single trial basis

A fundamental question in neuroscience is what type of internal representation leads to complex, adaptive behavior. When faced with a deadline, individuals' behavior suggests that they represent the mean and the uncertainty of an internal timer to make near-optimal, time-dependent decisions. Whether this ability relies on simple trial-and-error adjustments or whether it involves richer representations is unknown. Richer representations suggest a possibility of error monitoring, that is, the ability for an individual to assess its internal representation of the world and estimate discrepancy in the absence of external feedback. While rodents show timing behavior, whether they can represent and report temporal errors in their own produced duration on a single-trial basis is unknown. We designed a paradigm requiring rats to produce a target time interval and, subsequently, evaluate its error. Rats received a reward in a given location depending on the magnitude of their timing errors. During the test trials, rats had to choose a port corresponding to the error magnitude of their just-produced duration to receive a reward. High-choice accuracy demonstrates that rats kept track of the values of the timing variables on which they based their decision. Additionally, the rats kept a representation of the mapping between those timing values and the target value, as well as the history of the reinforcements. These findings demonstrate error-monitoring abilities in evaluating self-generated timing in rodents. Together, these findings suggest an explicit representation of produced duration and the possibility to evaluate its relation to the desired target duration.

Medial prefrontal cortex and the temporal control of action

Across species, the medial prefrontal cortex guides actions in time. This process can be studied using behavioral paradigms such as simple reaction-time and interval-timing tasks. Temporal control of action can be influenced by prefrontal neurotransmitters such as dopamine and acetylcholine and is highly relevant to human diseases such as Parkinson's disease, schizophrenia, and attention-deficit hyperactivity disorder (ADHD). We review evidence that across species, medial prefrontal lesions impair the temporal control of action. We then consider neurophysiological correlates in humans, primates, and rodents that might encode temporal processing and relate to cognitive-control mechanisms. These data have informed brain-stimulation studies in rodents and humans that can compensate for timing deficits. This line of work illuminates basic mechanisms of temporal control of action in the medial prefrontal cortex, which underlies a range of high-level cognitive processing and could contribute to new biomarkers and therapies for human brain diseases.

A New Timing Error Cost Function for Binary Time Series Prediction

The ability to make predictions is central to the artificial intelligence problem. While machine learning algorithms have difficulty in learning to predict events with hundreds of time-step dependencies, animals can learn event timing within tens of trials across a broad spectrum of time scales. This suggests strongly a need for new perspectives on the forecasting problem. This paper focuses on binary time series that can be predicted within some temporal precision. We demonstrate that the sum of squared errors (SSE) calculated at every time step is not appropriate for this problem. Next, we look at the advantages and shortcomings of using a dynamic time warping (DTW) cost function. Then, we propose the squared timing error (STE) that uses DTW on the event space and applies SSE on the timing error instead of at each time step. We evaluate all three cost functions on different types of timing errors, such as phase shift, warping, and missing events, on synthetic and real-world binary time series (heartbeats, finance, and music). The results show that STE provides more information about timing error, is differentiable, and can be computed online efficiently. Finally, we devise a gradient descent algorithm for STE on a simplified recurrent neural network. We then compare the performance of the STE-based algorithm to SSE- and logit-based gradient descent algorithms on the same network architecture. The results in real-world binary time series show that the STE algorithm generally outperforms all the other cost functions considered.

Adapting the flow of time with dopamine

The modulation of interval timing by dopamine (DA) has been well established over decades of research. The nature of this modulation, however, has remained controversial: Although the pharmacological evidence has largely suggested that time intervals are overestimated with higher DA levels, more recent optogenetic work has shown the opposite effect. In addition, a large body of work has asserted DA's role as a "reward prediction error" (RPE), or a teaching signal that allows the basal ganglia to learn to predict future rewards in reinforcement learning tasks. Whether these two seemingly disparate accounts of DA may be related has remained an open question. By taking a reinforcement learning-based approach to interval timing, we show here that the RPE interpretation of DA naturally extends to its role as a modulator of timekeeping and furthermore that this view reconciles the seemingly conflicting observations. We derive a biologically plausible, DA-dependent plasticity rule that can modulate the rate of timekeeping in either direction and whose effect depends on the timing of the DA signal itself. This bidirectional update rule can account for the results from pharmacology and optogenetics as well as the behavioral effects of reward rate on interval timing and the temporal selectivity of striatal neurons. Hence, by adopting a single RPE interpretation of DA, our results take a step toward unifying computational theories of reinforcement learning and interval timing. NEW & NOTEWORTHY How does dopamine (DA) influence interval timing? A large body of pharmacological evidence has suggested that DA accelerates timekeeping mechanisms. However, recent optogenetic work has shown exactly the opposite effect. In this article, we relate DA's role in timekeeping to its most established role, as a critical component of reinforcement learning. This allows us to derive a neurobiologically plausible framework that reconciles a large body of DA's temporal effects, including pharmacological, behavioral, electrophysiological, and optogenetic.

Minimal Circuit Model of Reward Prediction Error Computations and Effects of Nicotinic Modulations

Dopamine (DA) neurons in the ventral tegmental area (VTA) are thought to encode reward prediction errors (RPE) by comparing actual and expected rewards. In recent years, much work has been done to identify how the brain uses and computes this signal. While several lines of evidence suggest the interplay of the DA and the inhibitory interneurons in the VTA implements the RPE computation, it still remains unclear how the DA neurons learn key quantities, for example the amplitude and the timing of primary rewards during conditioning tasks. Furthermore, endogenous acetylcholine and exogenous nicotine, also likely affect these computations by acting on both VTA DA and GABA (γ -aminobutyric acid) neurons via nicotinic-acetylcholine receptors (nAChRs). To explore the potential circuit-level mechanisms for RPE computations during classical-conditioning tasks, we developed a minimal computational model of the VTA circuitry. The model was designed to account for several reward-related properties of VTA afferents and recent findings on VTA GABA neuron dynamics during conditioning. With our minimal model, we showed that the RPE can be learned by a two-speed process computing reward timing and magnitude. By including models of nAChR-mediated currents in the VTA DA-GABA circuit, we showed that nicotine should reduce the acetylcholine action on the VTA GABA neurons by receptor desensitization and potentially boost DA responses to reward-related signals in a non-trivial manner. Together, our results delineate the mechanisms by which RPE are computed in the brain, and suggest a hypothesis on nicotine-mediated effects on reward-related perception and decision-making.

A Population-Based Model of the Temporal Memory in the Hippocampus

Spatial and temporal dimensions are fundamental for orientation, adaptation, and survival of organisms. Hippocampus has been identified as the main neuroanatomical structure involved both in space and time perception and their internal representation. Dorsal hippocampus lesions showed a leftward shift (toward shorter durations) in peak-interval procedures, whereas ventral lesions shifted the peak time toward longer durations. We previously explained hippocampus lesion experimental findings by assuming a topological map model of the hippocampus with shorter durations memorized ventrally and longer durations more dorsal. Here we suggested a possible connection between the abstract topological maps model of the hippocampus that stored reinforcement times in a spatially ordered memory register and the "time cells" of the hippocampus. In this new model, the time cells provide a uniformly distributed time basis that covers the entire to-be-learned temporal duration. We hypothesized that the topological map of the hippocampus stores the weights that reflect the contribution of each time cell to the average temporal field that determines the behavioral response. The temporal distance between the to-be-learned criterion time and the time of the peak activity of each time cell provides the error signal that determines the corresponding weight correction. Long-term potentiation/depression could enhance/weaken the weights associated to the time cells that peak closer/farther to the criterion time. A coincidence detector mechanism, possibly under the control of the dopaminergic system, could be involved in our suggested error minimization and learning algorithm.

Between-session memory degradation accounts for within-session changes in fixed-interval performance

A common assumption in the study of fixed-interval (FI) timing is that FI performance is largely stable within sessions, once it is stable between sessions. Within-session changes in FI performance were examined in published data (Daniels and Sanabria, 2017), wherein some rats were trained on a FI 30-s schedule of food reinforcement (FI30) and others on a FI 90-s schedule (FI90). Following stability, FI90 rats were pre-fed for five sessions. Response rates declined as a function of trial, due more to latency lengthening than to run-rate reduction. Latencies were best described by a dynamic gamma-exponential mixture distribution, in which latency lengthening was driven by the growth of the criterion pulse count for a response and not by a reduction in the speed of an endogenous clock. The speed of the clock was selectively sensitive to the length of the FI; the prevalence and length of exponentially-distributed latencies were selectively sensitive to pre-feeding. These findings reveal (a) that parameters governing FI latencies are selectively sensitive to a range of manipulations, (b) a potential degradation of the criterion pulse count between consecutive sessions, and (c) a subsequent recovery of the criterion pulse count within sessions.

Temporal Processing of Joyful and Disgusting Food Pictures by Women With an Eating Disorder

The present study used the presentation of food pictures and judgements about their duration to assess the emotions elicited by food in women suffering from an eating disorder (ED). Twenty-three women diagnosed with an ED, namely anorexia (AN) or bulimia nervosa (BN), and 23 healthy controls (HC) completed a temporal bisection task and a duration discrimination task. Intervals were marked with emotionally pre-rated pictures of joyful and disgusting food, and pictures of neutral objects. The results showed that, in the bisection task, AN women overestimated the duration of food pictures in comparison to neutral ones. Also, compared to participants with BN, they perceived the duration of joyful food pictures as longer, and tended to overestimate the duration of the disgusting ones. These effects on perceived duration suggest that AN women experienced an intense reaction of fear when they were confronted to food pictures. More precisely, by having elevated the arousal level and activated the defensive system, food pictures seemed to have speeded up the rhythm of the AN participants’ internal clock, which led to an overestimation of images’ duration. In addition, the results revealed that, in both tasks, ED women presented a lower temporal sensitivity than HC, which was related to their ED symptomatology (i.e., BMI, restraint and concern) and, particularly, to their weaker cognitive abilities in terms of attention, processing speed and working memory. Considered all together, the findings of the present experiment highlight the role of fear and anxiety in the manifestations of AN and point out the importance of considering non-temporal factors in the interpretation of time perception performance.

A Rescorla-Wagner drift-diffusion model of conditioning and timing

Computational models of classical conditioning have made significant contributions to the theoretic understanding of associative learning, yet they still struggle when the temporal aspects of conditioning are taken into account. Interval timing models have contributed a rich variety of time representations and provided accurate predictions for the timing of responses, but they usually have little to say about associative learning. In this article we present a unified model of conditioning and timing that is based on the influential Rescorla-Wagner conditioning model and the more recently developed Timing Drift-Diffusion model. We test the model by simulating 10 experimental phenomena and show that it can provide an adequate account for 8, and a partial account for the other 2. We argue that the model can account for more phenomena in the chosen set than these other similar in scope models: CSC-TD, MS-TD, Learning to Time and Modular Theory. A comparison and analysis of the mechanisms in these models is provided, with a focus on the types of time representation and associative learning rule used.

Perceived Duration Increases with Contrast, but Only a Little

Recent adaptation studies provide evidence for early visual areas playing a role in duration perception. One explanation for the pronounced duration compression commonly found with adaptation is that it reflects adaptation-driven stimulus-specific reduction in neural activity in early visual areas. If this level of stimulus-associated neural activity does drive duration, then we would expect a strong effect of contrast on perceived duration as electrophysiological studies shows neural activity in early visual areas to be strongly related to contrast. We employed a spatially isotropic noise stimulus where the luminance of each noise element was independently sinusoidally modulated at 4 Hz. Participants matched the perceived duration of a high (0.9) or low (0.1) contrast stimulus to a previously presented standard stimulus (600 ms, contrast = 0.3). To achieve perceptually equivalent durations, the low contrast stimulus had to be presented for longer than the high contrast stimulus. This occurred when we controlled for stimulus size and when we adjusted for individual differences in perceived temporal frequency. Further, we show that the effect cannot be explained by shifts in perceived onset and offset and is not explained by a simple contrast-driven response bias. The direction of our results is clearly consistent with the idea that level of neural activity drives duration. However, the magnitude of the effect (~10% duration difference over a 0.9-0.1 contrast reduction) is in marked contrast to the larger duration distortions that can be found with repetition suppression and the oddball effect; particularly when these may be associated with smaller differences in neural activity than that expected from our contrast difference. Taken together, these results indicate that level of stimulus-related neural activity in early visual areas is unlikely to provide a general mechanism for explaining differences in perceived duration.

Single trial beta oscillations index time estimation

Recent work shows that putamen-originating beta power oscillations serve as a carrier for temporal information during tapping tasks, with higher beta power associated with longer temporal reproductions. However, given the nature of tapping tasks, it is difficult to determine whether beta power dynamics observed in these tasks are linked to the generation or execution of motor programs or to the internal representation of time. To assess whether recent findings in animals generalize to human studies we reanalyzed existing EEG data of participants who estimated a 2.5s time interval with self-paced onset and offset keypresses. The results showed that the trial-to-trial beta power measured after the onset predicts the produced duration, such that higher beta power indexes longer produced durations. Moreover, although beta power measured before the first key-press also influenced the estimated interval, it did so independently from post-first-keypress beta power. These results suggest that initial motor inhibition plays an important role in interval production, and that this inhibition can be interpreted as a biased starting point of the decision processes involved in time estimation.

Revisiting the effect of nicotine on interval timing

This paper reviews the evidence for nicotine-induced acceleration of the internal clock when timing in the seconds-to-minutes timescale, and proposes an alternative explanation to this evidence: that nicotine reduces the threshold for responses that result in more reinforcement. These two hypotheses were tested in male Wistar rats using a novel timing task. In this task, rats were trained to seek food at one location after 8s since trial onset and at a different location after 16s. Some rats received the same reward at both times (group SAME); some received a larger reward at 16s (group DIFF). Steady baseline performance was followed by 3 days of subcutaneous nicotine administration (0.3mg/kg), baseline recovery, and an antagonist challenge (mecamylamine, 1.0mg/kg). Nicotine induced a larger, immediate reduction in latencies to switch (LTS) in group DIFF than in group SAME. This effect was sustained throughout nicotine administration. Mecamylamine pretreatment and nicotine discontinuation rapidly recovered baseline performance. These results support a response-threshold account of nicotinic disruption of timing performance, possibly mediated by nicotinic acetylcholine receptors. A detailed analysis of the distribution of LTSs suggests that anomalous effects of nicotine on LTS dispersion may be due to loss of temporal control of behavior.

Conditioning and time representation in long short-term memory networks

Dopaminergic models based on the temporal-difference learning algorithm usually do not differentiate trace from delay conditioning. Instead, they use a fixed temporal representation of elapsed time since conditioned stimulus onset. Recently, a new model was proposed in which timing is learned within a long short-term memory (LSTM) artificial neural network representing the cerebral cortex (Rivest et al. in J Comput Neurosci 28(1):107-130, 2010). In this paper, that model's ability to reproduce and explain relevant data, as well as its ability to make interesting new predictions, are evaluated. The model reveals a strikingly different temporal representation between trace and delay conditioning since trace conditioning requires working memory to remember the past conditioned stimulus while delay conditioning does not. On the other hand, the model predicts no important difference in DA responses between those two conditions when trained on one conditioning paradigm and tested on the other. The model predicts that in trace conditioning, animal timing starts with the conditioned stimulus offset as opposed to its onset. In classical conditioning, it predicts that if the conditioned stimulus does not disappear after the reward, the animal may expect a second reward. Finally, the last simulation reveals that the buildup of activity of some units in the networks can adapt to new delays by adjusting their rate of integration. Most importantly, the paper shows that it is possible, with the proposed architecture, to acquire discharge patterns similar to those observed in dopaminergic neurons and in the cerebral cortex on those tasks simply by minimizing a predictive cost function.

Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area

Neural activity in dopaminergic areas such as the ventral tegmental area is influenced by timing processes, in particular by the temporal expectation of rewards during Pavlovian conditioning. Receipt of a reward at the expected time allows to compute reward-prediction errors which can drive learning in motor or cognitive structures. Reciprocally, dopamine plays an important role in the timing of external events. Several models of the dopaminergic system exist, but the substrate of temporal learning is rather unclear. In this article, we propose a neuro-computational model of the afferent network to the ventral tegmental area, including the lateral hypothalamus, the pedunculopontine nucleus, the amygdala, the ventromedial prefrontal cortex, the ventral basal ganglia (including the nucleus accumbens and the ventral pallidum), as well as the lateral habenula and the rostromedial tegmental nucleus. Based on a plausible connectivity and realistic learning rules, this neuro-computational model reproduces several experimental observations, such as the progressive cancelation of dopaminergic bursts at reward delivery, the appearance of bursts at the onset of reward-predicting cues or the influence of reward magnitude on activity in the amygdala and ventral tegmental area. While associative learning occurs primarily in the amygdala, learning of the temporal relationship between the cue and the associated reward is implemented as a dopamine-modulated coincidence detection mechanism in the nucleus accumbens.

Interactions of timing and prediction error learning

Timing and prediction error learning have historically been treated as independent processes, but growing evidence has indicated that they are not orthogonal. Timing emerges at the earliest time point when conditioned responses are observed, and temporal variables modulate prediction error learning in both simple conditioning and cue competition paradigms. In addition, prediction errors, through changes in reward magnitude or value alter timing of behavior. Thus, there appears to be a bi-directional interaction between timing and prediction error learning. Modern theories have attempted to integrate the two processes with mixed success. A neurocomputational approach to theory development is espoused, which draws on neurobiological evidence to guide and constrain computational model development. Heuristics for future model development are presented with the goal of sparking new approaches to theory development in the timing and prediction error fields.

Adaptation of timing behavior to a regular change in criterion

This study examined how operant behavior adapted to an abrupt but regular change in the timing of reinforcement. Pigeons were trained on a fixed interval (FI) 15-s schedule of reinforcement during half of each experimental session, and on an FI 45-s (Experiment 1), FI 60-s (Experiment 2), or extinction schedule (Experiment 3) during the other half. FI performance was well characterized by a mixture of two gamma-shaped distributions of responses. When a longer FI schedule was in effect in the first half of the session (Experiment 1), a constant interference by the shorter FI was observed. When a shorter FI schedule was in effect in the first half of the session (Experiments 1, 2, and 3), the transition between schedules involved a decline in responding and a progressive rightward shift in the mode of the response distribution initially centered around the short FI. These findings are discussed in terms of the constraints they impose to quantitative models of timing, and in relation to the implications for information-based models of associative learning. This article is part of a Special Issue entitled: Associative and Temporal Learning.

Timescale Invariance in the Pacemaker-Accumulator Family of Timing Models

Pacemaker-accumulator (PA) systems have been the most popular kind of timing model in the half-century since their introduction by Treisman (1963). Many alternative timing models have been designed predicated on different assumptions, though the dominant PA model during this period — Gibbon and Church’s Scalar Expectancy Theory (SET) — invokes most of them. As in Treisman, SET’s implementation assumes a fixed-rate clock-pulse generator and encodes durations by storing average pulse counts; unlike Treisman’s model, SET’s decision process invokes Weber’s law of magnitude-comparison to account for timescale-invariant temporal precision in animal behavior. This is one way to deal with the ‘Poisson timing’ issue, in which relative temporal precision increases for longer durations, contrafactually, in a simplified version of Treisman’s model. First, we review the fact that this problem does not afflict Treisman’s model itself due to a key assumption not shared by SET. Second, we develop a contrasting PA model, an extension of Killeen and Fetterman’s Behavioral Theory of Timing that accumulates Poisson pulses up to a fixed criterion level, with pulse rates adapting to time different intervals. Like Treisman’s model, this time-adaptive, opponent Poisson, drift–diffusion model accounts for timescale invariance without first assuming Weber’s law. It also makes new predictions about response times and learning speed and connects interval timing to the popular drift–diffusion model of perceptual decision making. With at least three different routes to timescale invariance, the PA model family can provide a more compelling account of timed behavior than may be generally appreciated.