Neural Encoding and Representation of Time for Sensorimotor Control and Learning

Common neural mechanisms supporting time judgements in humans and monkeys

There has been an increasing interest in identifying the biological underpinnings of human time perception, for which purpose research in non-human primates (NHP) is common. Although previous work, based on behaviour, suggests that similar mechanisms support time perception across species, the neural correlates of time estimation in humans and NHP have not been directly compared. In this study, we assess whether brain evoked responses during a time categorization task are similar across species. Specifically, we assess putative differences in post-interval evoked potentials as a function of perceived duration in human EEG (N = 24) and local field potential (LFP) and spike recordings in pre-supplementary motor area (pre-SMA) of one monkey. Event-related potentials (ERPs) differed significantly after the presentation of the temporal interval between "short" and "long" perceived durations in both species, even when the objective duration of the stimuli was the same. Interestingly, the polarity of the reported ERPs was reversed for incorrect trials (i.e., the ERP of a "long" stimulus looked like the ERP of a "short" stimulus when a time categorization error was made). Hence, our results show that post-interval potentials reflect the perceived (rather than the objective) duration of the presented time interval in both NHP and humans. In addition, firing rates in monkey's pre-SMA also differed significantly between short and long perceived durations and were reversed in incorrect trials. Together, our results show that common neural mechanisms support time categorization in NHP and humans, thereby suggesting that NHP are a good model for investigating human time perception.

Population coding of strategic variables during foraging in freely moving macaques

Until now, it has been difficult to examine the neural bases of foraging in naturalistic environments because previous approaches have relied on restrained animals performing trial-based foraging tasks. Here we allowed unrestrained monkeys to freely interact with concurrent reward options while we wirelessly recorded population activity in the dorsolateral prefrontal cortex. The animals decided when and where to forage based on whether their prediction of reward was fulfilled or violated. This prediction was not solely based on a history of reward delivery, but also on the understanding that waiting longer improves the chance of reward. The task variables were continuously represented in a subspace of the high-dimensional population activity, and this compressed representation predicted the animal's subsequent choices better than the true task variables and as well as the raw neural activity. Our results indicate that monkeys' foraging strategies are based on a cortical model of reward dynamics as animals freely explore their environment.

Embodying Time in the Brain: A Multi-Dimensional Neuroimaging Meta-Analysis of 95 Duration Processing Studies

Time is an omnipresent aspect of almost everything we experience internally or in the external world. The experience of time occurs through such an extensive set of contextual factors that, after decades of research, a unified understanding of its neural substrates is still elusive. In this study, following the recent best-practice guidelines, we conducted a coordinate-based meta-analysis of 95 carefully-selected neuroimaging papers of duration processing. We categorized the included papers into 14 classes of temporal features according to six categorical dimensions. Then, using the activation likelihood estimation (ALE) technique we investigated the convergent activation patterns of each class with a cluster-level family-wise error correction at p < 0.05. The regions most consistently activated across the various timing contexts were the pre-SMA and bilateral insula, consistent with an embodied theory of timing in which abstract representations of duration are rooted in sensorimotor and interoceptive experience, respectively. Moreover, class-specific patterns of activation could be roughly divided according to whether participants were timing auditory sequential stimuli, which additionally activated the dorsal striatum and SMA-proper, or visual single interval stimuli, which additionally activated the right middle frontal and inferior parietal cortices. We conclude that temporal cognition is so entangled with our everyday experience that timing stereotypically common combinations of stimulus characteristics reactivates the sensorimotor systems with which they were first experienced.

Direct contribution of the sensory cortex to the judgment of stimulus duration

Decision making frequently depends on monitoring the duration of sensory events. To determine whether, and how, the perception of elapsed time derives from the neuronal representation of the stimulus itself, we recorded and optogenetically modulated vibrissal somatosensory cortical activity as male rats judged vibration duration. Perceived duration was dilated by optogenetic excitation. A second set of rats judged vibration intensity; here, optogenetic excitation amplified the intensity percept, demonstrating sensory cortex to be the common gateway both to time and to stimulus feature processing. A model beginning with the membrane currents evoked by vibrissal and optogenetic drive and culminating in the representation of perceived time successfully replicated rats' choices. Time perception is thus as deeply intermeshed within the sensory processing pathway as is the sense of touch itself, suggesting that the experience of time may be further investigated with the toolbox of sensory coding.

Pre-supplementary Motor Cortex Mediates Learning Transfer from Perceptual to Motor Timing

When we intensively train a timing skill, such as learning to play the piano, we not only produce brain changes associated with task-specific learning but also improve our performance in other temporal behaviors that depend on these tuned neural resources. Since the neural basis of time learning and generalization is still unknown, we measured the changes in neural activity associated with the transfer of learning from perceptual to motor timing in a large sample of subjects (n = 65; 39 women). We found that intense training in an interval discrimination task increased the acuity of time perception in a group of subjects that also exhibited learning transfer, expressed as a reduction in inter-tap interval variability during an internally driven periodic motor task. In addition, we found subjects with no learning and/or generalization effects. Notably, functional imaging showed an increase in pre-supplementary motor area and caudate-putamen activity between the post- and pre-training sessions of the tapping task. This increase was specific to the subjects that generalized their timing acuity from the perceptual to the motor context. These results emphasize the central role of the cortico-basal ganglia circuit in the generalization of timing abilities between tasks.

Learned response dynamics reflect stimulus timing and encode temporal expectation violations in superficial layers of mouse V1

The ability to recognize ordered event sequences is a fundamental component of sensory cognition and underlies the capacity to generate temporally specific expectations of future events based on previous experience. Various lines of evidence suggest that the primary visual cortex participates in some form of predictive processing, but many details remain ambiguous. Here we use two-photon calcium imaging in layer 2/3 (L2/3) of the mouse primary visual cortex (V1) to study changes to neural activity under a multi-day sequence learning paradigm with respect to prediction error responses, stimulus encoding, and time. We find increased neural activity at the time an expected, but omitted, stimulus would have occurred but no significant prediction error responses following an unexpected stimulus substitution. Sequence representations became sparser and less correlated with training, although these changes had no effect on decoding accuracy of stimulus identity or timing. Additionally, we find that experience modifies the temporal structure of stimulus responses to produce a bias towards predictive stimulus-locked activity. Finally, we find significant temporal structure during intersequence rest periods that was largely unchanged by training.

Amodal population clock in the primate medial premotor system for rhythmic tapping

The neural substrate for beat extraction and response entrainment to rhythms is not fully understood. Here we analyze the activity of medial premotor neurons in monkeys performing isochronous tapping guided by brief flashing stimuli or auditory tones. The population dynamics shared the following properties across modalities: the circular dynamics of the neural trajectories form a regenerating loop for every produced interval; the trajectories converge in similar state space at tapping times resetting the clock; and the tempo of the synchronized tapping is encoded in the trajectories by a combination of amplitude modulation and temporal scaling. Notably, the modality induces displacement in the neural trajectories in the auditory and visual subspaces without greatly altering the time-keeping mechanism. These results suggest that the interaction between the medial premotor cortex's amodal internal representation of pulse and a modality-specific external input generates a neural rhythmic clock whose dynamics govern rhythmic tapping execution across senses.

Rhythmic tapping to a moving beat motion kinematics overrules natural gravity

Beat induction is the cognitive ability that allows humans to listen to a regular pulse in music and move in synchrony with it. Although auditory rhythmic cues induce more consistent synchronization than flashing visual metronomes, this auditory-visual asymmetry can be canceled by visual moving stimuli. Here, we investigated whether the naturalness of visual motion or its kinematics could provide a synchronization advantage over flashing metronomes. Subjects were asked to tap in sync with visual metronomes defined by vertically accelerating/decelerating motion, either congruent or not with natural gravity; horizontally accelerating/decelerating motion; or flashing stimuli. We found that motion kinematics was the predominant factor determining rhythm synchronization, as accelerating moving metronomes in any cardinal direction produced more precise and predictive tapping than decelerating or flashing conditions. Our results support the notion that accelerating visual metronomes convey a strong sense of beat, as seen in the cueing movements of an orchestra director.

Auditory-motor synchronization varies among individuals and is critically shaped by acoustic features

The ability to synchronize body movements with quasi-regular auditory stimuli represents a fundamental trait in humans at the core of speech and music. Despite the long trajectory of the study of such ability, little attention has been paid to how acoustic features of the stimuli and individual differences can modulate auditory-motor synchrony. Here, by exploring auditory-motor synchronization abilities across different effectors and types of stimuli, we revealed that this capability is more restricted than previously assumed. While the general population can synchronize to sequences composed of the repetitions of the same acoustic unit, the synchrony in a subgroup of participants is impaired when the unit's identity varies across the sequence. In addition, synchronization in this group can be temporarily restored by being primed by a facilitator stimulus. Auditory-motor integration is stable across effectors, supporting the hypothesis of a central clock mechanism subserving the different articulators but critically shaped by the acoustic features of the stimulus and individual abilities.

The role of consciously timed movements in shaping and improving auditory timing

Our subjective sense of time is intertwined with a plethora of perceptual, cognitive and motor functions, and likewise, the brain is equipped to expertly filter, weight and combine these signals for seamless interactions with a dynamic world. Until relatively recently, the literature on time perception has excluded the influence of simultaneous motor activity, yet it has been found that motor circuits in the brain are at the core of most timing functions. Several studies have now identified that concurrent movements exert robust effects on perceptual timing estimates, but critically have not assessed how humans consciously judge the duration of their own movements. This creates a gap in our understanding of the mechanisms driving movement-related effects on sensory timing. We sought to address this gap by administering a sensorimotor timing task in which we explicitly compared the timing of isolated auditory tones and arm movements, or both simultaneously. We contextualized our findings within a Bayesian cue combination framework, in which separate sources of temporal information are weighted by their reliability and integrated into a unitary time estimate that is more precise than either unisensory estimate. Our results revealed differences in accuracy between auditory, movement and combined trials, and (crucially) that combined trials were the most accurately timed. Under the Bayesian framework, we found that participants' combined estimates were more precise than isolated estimates, yet were sub-optimal when compared with the model's prediction, on average. These findings elucidate previously unknown qualities of conscious motor timing and propose computational mechanisms that can describe how movements combine with perceptual signals to create unified, multimodal experiences of time.

Reciprocal facilitation between mental and visuomotor rotations

Humans exhibit remarkably complex cognitive abilities and adaptive behavior in daily life. Cognitive operation in the "mental workspace," such as mentally rotating a piece of luggage to fit into fixed trunk space, helps us maintain and manipulate information on a moment-to-moment basis. Skill acquisition in the "sensorimotor workspace," such as learning a new mapping between the magnitude of new vehicle movement and wheel turn, allows us to adjust our behavior to changing environmental or internal demands to maintain appropriate motor performance. While this cognitive and sensorimotor synergy is at the root of adaptive behavior in the real world, their interplay has been understudied due to a divide-and-conquer approach. We evaluated whether a separate domain-specific or common domain-general operation drives mental and sensorimotor rotational transformations. We observed that participants improved the efficiency of mental rotation speed after the visuomotor rotation training, and their learning rate for visuomotor adaptation also improved after their mental rotation training. Such bidirectional transfer between two widely different tasks highlights the remarkable reciprocal plasticity and demonstrates a common transformation mechanism between two intertwined workspaces. Our findings urge the necessity of an explicitly integrated approach to enhance our understanding of the dynamic interdependence between cognitive and sensorimotor mechanisms.

Neurocognitive analyses reveal that video game players exhibit enhanced implicit temporal processing

Winning in action video games requires to predict timed events in order to react fast enough. In these games, repeated waiting for enemies may help to develop implicit (incidental) preparation mechanisms. We compared action video game players and non-video game players in a reaction time task involving both implicit time preparations and explicit (conscious) temporal attention cues. Participants were immersed in virtual reality and instructed to respond to a visual target appearing at variable delays after a warning signal. In half of the trials, an explicit cue indicated when the target would occur after the warning signal. Behavioral, oculomotor and EEG data consistently indicate that, compared with non-video game players, video game players better prepare in time using implicit mechanisms. This sheds light on the neglected role of implicit timing and related electrophysiological mechanisms in gaming research. The results further suggest that game-based interventions may help remediate implicit timing disorders found in psychiatric populations.

A cross-sectional EEG study reveals that individuals who consistently play action video games exhibit improved performance in a reaction time task involving implicit time preparations, compared with participants who did not normally play video games

Time Perception for Musical Rhythms: Sensorimotor Perspectives on Entrainment, Simulation, and Prediction

Neural mechanisms supporting time perception in continuously changing sensory environments may be relevant to a broader understanding of how the human brain utilizes time in cognition and action. In this review, we describe current theories of sensorimotor engagement in the support of subsecond timing. We focus on musical timing due to the extensive literature surrounding movement with and perception of musical rhythms. First, we define commonly used but ambiguous concepts including neural entrainment, simulation, and prediction in the context of musical timing. Next, we summarize the literature on sensorimotor timing during perception and performance and describe current theories of sensorimotor engagement in the support of subsecond timing. We review the evidence supporting that sensorimotor engagement is critical in accurate time perception. Finally, potential clinical implications for a sensorimotor perspective of timing are highlighted.

Efficient Temporal Coding in the Early Visual System: Existing Evidence and Future Directions

While it is universally accepted that the brain makes predictions, there is little agreement about how this is accomplished and under which conditions. Accurate prediction requires neural circuits to learn and store spatiotemporal patterns observed in the natural environment, but it is not obvious how such information should be stored, or encoded. Information theory provides a mathematical formalism that can be used to measure the efficiency and utility of different coding schemes for data transfer and storage. This theory shows that codes become efficient when they remove predictable, redundant spatial and temporal information. Efficient coding has been used to understand retinal computations and may also be relevant to understanding more complicated temporal processing in visual cortex. However, the literature on efficient coding in cortex is varied and can be confusing since the same terms are used to mean different things in different experimental and theoretical contexts. In this work, we attempt to provide a clear summary of the theoretical relationship between efficient coding and temporal prediction, and review evidence that efficient coding principles explain computations in the retina. We then apply the same framework to computations occurring in early visuocortical areas, arguing that data from rodents is largely consistent with the predictions of this model. Finally, we review and respond to criticisms of efficient coding and suggest ways that this theory might be used to design future experiments, with particular focus on understanding the extent to which neural circuits make predictions from efficient representations of environmental statistics.

No behavioural evidence for rhythmic facilitation of perceptual discrimination

It has been hypothesized that internal oscillations can synchronize (i.e., entrain) to external environmental rhythms, thereby facilitating perception and behaviour. To date, evidence for the link between the phase of neural oscillations and behaviour has been scarce and contradictory; moreover, it remains an open question whether the brain can use this tentative mechanism for active temporal prediction. In our present study, we conducted a series of auditory pitch discrimination tasks with 181 healthy participants in an effort to shed light on the proposed behavioural benefits of rhythmic cueing and entrainment. In the three versions of our task, we observed no perceptual benefit of purported entrainment: targets occurring in-phase with a rhythmic cue provided no perceptual benefits in terms of discrimination accuracy or reaction time when compared with targets occurring out-of-phase or targets occurring randomly, nor did we find performance differences for targets preceded by rhythmic versus random cues. However, we found a surprising effect of cueing frequency on reaction time, in which participants showed faster responses to cue rhythms presented at higher frequencies. We therefore provide no evidence of entrainment, but instead a tentative effect of covert active sensing in which a faster external rhythm leads to a faster communication rate between motor and sensory cortices, allowing for sensory inputs to be sampled earlier in time.

No behavioural evidence for rhythmic facilitation of perceptual discrimination

It has been hypothesized that internal oscillations can synchronize (i.e., entrain) to external environmental rhythms, thereby facilitating perception and behaviour. To date, evidence for the link between the phase of neural oscillations and behaviour has been scarce and contradictory; moreover, it remains an open question whether the brain can use this tentative mechanism for active temporal prediction. In our present study, we conducted a series of auditory pitch discrimination tasks with 181 healthy participants in an effort to shed light on the proposed behavioural benefits of rhythmic cueing and entrainment. In the three versions of our task, we observed no perceptual benefit of purported entrainment: targets occurring in-phase with a rhythmic cue provided no perceptual benefits in terms of discrimination accuracy or reaction time when compared with targets occurring out-of-phase or targets occurring randomly, nor did we find performance differences for targets preceded by rhythmic versus random cues. However, we found a surprising effect of cueing frequency on reaction time, in which participants showed faster responses to cue rhythms presented at higher frequencies. We therefore provide no evidence of entrainment, but instead a tentative effect of covert active sensing in which a faster external rhythm leads to a faster communication rate between motor and sensory cortices, allowing for sensory inputs to be sampled earlier in time.

How movements shape the perception of time

In order to keep up with a changing environment, mobile organisms must be capable of deciding both where and when to move. This precision necessitates a strong sense of time, as otherwise we would fail in many of our movement goals. Yet, despite this intrinsic link, only recently have researchers begun to understand how these two features interact. Primarily, two effects have been observed: movements can bias time estimates, but they can also make them more precise. Here we review this literature and propose that both effects can be explained by a Bayesian cue combination framework, in which movement itself affords the most precise representation of time, which can influence perception in either feedforward or active sensing modes.

Mapping between sound, brain and behaviour: four-level framework for understanding rhythm processing in humans and non-human primates

Humans perceive and spontaneously move to one or several levels of periodic pulses (a meter, for short) when listening to musical rhythm, even when the sensory input does not provide prominent periodic cues to their temporal location. Here, we review a multi-levelled framework to understanding how external rhythmic inputs are mapped onto internally represented metric pulses. This mapping is studied using an approach to quantify and directly compare representations of metric pulses in signals corresponding to sensory inputs, neural activity and behaviour (typically body movement). Based on this approach, recent empirical evidence can be drawn together into a conceptual framework that unpacks the phenomenon of meter into four levels. Each level highlights specific functional processes that critically enable and shape the mapping from sensory input to internal meter. We discuss the nature, constraints and neural substrates of these processes, starting with fundamental mechanisms investigated in macaque monkeys that enable basic forms of mapping between simple rhythmic stimuli and internally represented metric pulse. We propose that human evolution has gradually built a robust and flexible system upon these fundamental processes, allowing more complex levels of mapping to emerge in musical behaviours. This approach opens promising avenues to understand the many facets of rhythmic behaviours across individuals and species. This article is part of the theme issue 'Synchrony and rhythm interaction: from the brain to behavioural ecology'.

Vocal learning as a preadaptation for the evolution of human beat perception and synchronization

The human capacity to synchronize movements to an auditory beat is central to musical behaviour and to debates over the evolution of human musicality. Have humans evolved any neural specializations for music processing, or does music rely entirely on brain circuits that evolved for other reasons? The vocal learning and rhythmic synchronization hypothesis proposes that our ability to move in time with an auditory beat in a precise, predictive and tempo-flexible manner originated in the neural circuitry for complex vocal learning. In the 15 years, since the hypothesis was proposed a variety of studies have supported it. However, one study has provided a significant challenge to the hypothesis. Furthermore, it is increasingly clear that vocal learning is not a binary trait animals have or lack, but varies more continuously across species. In the light of these developments and of recent progress in the neurobiology of beat processing and of vocal learning, the current paper revises the vocal learning hypothesis. It argues that an advanced form of vocal learning acts as a preadaptation for sporadic beat perception and synchronization (BPS), providing intrinsic rewards for predicting the temporal structure of complex acoustic sequences. It further proposes that in humans, mechanisms of gene-culture coevolution transformed this preadaptation into a genuine neural adaptation for sustained BPS. The larger significance of this proposal is that it outlines a hypothesis of cognitive gene-culture coevolution which makes testable predictions for neuroscience, cross-species studies and genetics. This article is part of the theme issue 'Synchrony and rhythm interaction: from the brain to behavioural ecology'.

Competitive state of movements during planning predicts sequence performance

Humans can learn and produce skilled movement sequences from memory, yet the nature of sequence planning is not well understood. Previous computational and neurophysiological work suggests that movements in a sequence are planned as parallel graded activations and selected for output through competition. However, the relevance of this planning pattern to sequence production fluency and accuracy, as opposed to the temporal structure of sequences, is unclear. To resolve this question, we assessed the relative availability of constituent movements behaviorally during the preparation of motor sequences from memory. In three separate multisession experiments, healthy participants were trained to retrieve and produce four-element finger press sequences with particular timing according to an abstract sequence cue. We evaluated reaction time (RT) and error rate as markers of movement availability to constituent movement probes. Our results demonstrate that longer preparation time produces more pronounced differences in availability between adjacent sequence elements, whereas no effect was found for sequence speed or temporal grouping. Further, participants with larger position-dependent differences in movement availability tended to initiate correct sequences faster and with a higher temporal accuracy. Our results suggest that competitive preactivation is established gradually during sequence planning and predicts sequence skill, rather than the temporal structure of the motor sequence.NEW & NOTEWORTHY Sequence planning is an integral part of motor sequence control. Here, we demonstrate that the competitive state of sequential movements during sequence planning can be read out behaviorally through movement probes. We show that position-dependent differences in movement availability during planning reflect sequence preparedness and skill but not the timing of the planned sequence. Behavioral access to the preparatory state of movements may serve as a marker of sequence planning capacity.