Discovering Precise Temporal Patterns in Large-Scale Neural Recordings through Robust and Interpretable Time Warping

A magnitude-independent neural code for linguistic information during sentence production

Humans are the only species with the ability to convey an unbounded number of novel thoughts by combining words into sentences. This process is guided by complex syntactic and semantic processes. Despite their centrality to human cognition, the neural mechanisms underlying these systems remain obscured by inherent limitations of non-invasive brain measures and a near total focus on comprehension paradigms. Here, we address these limitations with high-resolution neurosurgical recordings (electrocorticography) and a controlled sentence production experiment. We uncover distinct cortical networks encoding word-level, semantic, and syntactic information. These networks are broadly distributed across traditional language areas, but with focal sensitivity to syntactic structure in middle and inferior frontal gyri. In contrast to previous findings from comprehension studies, we find that these networks are largely non-overlapping, each specialized for just one of the three linguistic constructs we investigate. Most strikingly, our data reveal an unexpected property of syntax and semantics: it is encoded independent of neural activity levels. We hypothesize that this "magnitude-independent" coding scheme likely reflects the distributed nature of these higher-order cognitive constructs.

Dynamic modeling of EEG responses to natural speech reveals earlier processing of predictable words

In recent years, it has become clear that EEG indexes the comprehension of natural, narrative speech. One particularly compelling demonstration of this fact can be seen by regressing EEG responses to speech against measures of how individual words in that speech linguistically relate to their preceding context. This approach produces a so-called temporal response function that displays a centro-parietal negativity reminiscent of the classic N400 component of the event-related potential. One shortcoming of previous implementations of this approach is that they have typically assumed a linear, time-invariant relationship between the linguistic speech features and the EEG responses. In other words, the analysis typically assumes that the response has the same shape and timing for every word - and only varies (linearly) in terms of its amplitude. In the present work, we relax this assumption under the hypothesis that responses to individual words may be processed more rapidly when they are predictable. Specifically, we introduce a framework wherein the standard linear temporal response function can be modulated in terms of its amplitude, latency, and temporal scale based on the predictability of the current and prior words. We use the proposed approach to model EEG recorded from a set of participants who listened to an audiobook narrated by a single talker, and a separate set of participants who attended to one of two concurrently presented audiobooks. We show that expected words are processed faster - evoking lower amplitude N400-like responses with earlier peaks - and that this effect is driven both by the word's own predictability and the predictability of the immediately preceding word. Additional analysis suggests that this finding is not simply explained based on how quickly words can be disambiguated from their phonetic neighbors. As such, our study demonstrates that the timing and amplitude of brain responses to words in natural speech depend on their predictability. By accounting for these effects, our framework also improves the accuracy with which neural responses to natural speech can be modeled.

The geometry of the neural state space of decisions

How do populations of neurons collectively encode and process information during cognitive tasks? We analyze high-yield population recordings from the macaque lateral intraparietal area (LIP) during a reaction-time random-dot-motion direction-discrimination task. We find that the trajectories of neural population activity patterns during single decisions lie within a curved two-dimensional manifold. The reaction time of trajectories systematically varies along one dimension, such that slow and fast decisions trace distinct activity patterns. Trajectories transition from a deliberation stage, in which they are noisy and remain similar between the choices, to a commitment stage, in which they are far less noisy and diverge sharply for the different choices. The deliberation phase is pronounced for slower decisions and gradually diminishes as reaction time decreases. A mechanistic circuit model provides an explanation for the observed properties, and suggests the transition between stages represents a transition from more sensory-driven to more circuit-driven dynamics. It yields two striking predictions we verify in the data. First, whether neurons are more choice selective for slow or fast trials varies systematically with the retinotopic location of their response fields. Second, the slower the trial, the more saccades undershoot the choice target. The results highlight the roles of distributed and dynamic activity patterns and intrinsic circuit dynamics in the population implementation of a cognitive task.

Rarely categorical, always high-dimensional: how the neural code changes along the cortical hierarchy

A long-standing debate in neuroscience concerns whether individual neurons are organized into functionally distinct populations that encode information differently ("categorical" representations [1-3]) and the implications for neural computation. Here, we systematically analyzed how cortical neurons encode cognitive, sensory, and movement variables across 43 cortical regions during a complex task (14,000+ units from the International Brain Laboratory public Brain-wide Map data set [4]) and studied how these properties change across the sensory-cognitive cortical hierarchy [5]. We found that the structure of the neural code was scale-dependent: on a whole-cortex scale, neural selectivity was categorical and organized across regions in a way that reflected their anatomical connectivity. However, within individual regions, categorical representations were rare and limited to primary sensory areas. Remarkably, the degree of categorical clustering of neural selectivity was inversely correlated to the dimensionality of neural representations, suggesting a link between single-neuron selectivity and computational properties of population codes that we explained in a mathematical model. Finally, we found that the fraction of linearly separable combinations of experimental conditions ("Shattering Dimensionality" [6]) was near maximal across all areas, indicating a robust and uniform ability for flexible information encoding throughout the cortex. In conclusion, our results provide systematic evidence for a non-categorical, high-dimensional neural code in all but the lower levels of the cortical hierarchy.

An emerging view of neural geometry in motor cortex supports high-performance decoding

Decoders for brain-computer interfaces (BCIs) assume constraints on neural activity, chosen to reflect scientific beliefs while yielding tractable computations. Recent scientific advances suggest that the true constraints on neural activity, especially its geometry, may be quite different from those assumed by most decoders. We designed a decoder, MINT, to embrace statistical constraints that are potentially more appropriate. If those constraints are accurate, MINT should outperform standard methods that explicitly make different assumptions. Additionally, MINT should be competitive with expressive machine learning methods that can implicitly learn constraints from data. MINT performed well across tasks, suggesting its assumptions are well-matched to the data. MINT outperformed other interpretable methods in every comparison we made. MINT outperformed expressive machine learning methods in 37 of 42 comparisons. MINT's computations are simple, scale favorably with increasing neuron counts, and yield interpretable quantities such as data likelihoods. MINT's performance and simplicity suggest it may be a strong candidate for many BCI applications.

Decoding words during sentence production: Syntactic role encoding and structure-dependent dynamics revealed by ECoG

Sentence production is the uniquely human ability to transform complex thoughts into strings of words. Despite the importance of this process, language production research has primarily focused on single words. However, it remains a largely untested assumption that the principles of word production generalize to more naturalistic utterances like sentences. Here, we investigate this using high-resolution neurosurgical recordings (ECoG) and an overt production experiment where patients produced six words in isolation (picture naming) and in sentences (scene description). We trained machine learning classifiers to identify the unique brain activity patterns for each word during picture naming, and used these patterns to decode which words patients were processing while they produced sentences. Our findings confirm that words share cortical representations across tasks, but reveal a division of labor within the language network. In sensorimotor cortex, words were consistently activated in the order in which they were said in the sentence. However, in inferior and middle frontal gyri (IFG and MFG), the order in which words were processed depended on the syntactic structure of the sentence. Deeper analysis of this pattern revealed a spatial code for representing a word's position in the sentence, with subjects selectively encoded in IFG and objects in MFG. Finally, we argue that the processes we observe in prefrontal cortex may impose a subtle pressure on language evolution, explaining why nearly all the world's languages position subjects before objects.

A mathematical language for linking fine-scale structure in spikes from hundreds to thousands of neurons with behaviour

Recent advances in neural recording technology allow simultaneously recording action potentials from hundreds to thousands of neurons in awake, behaving animals. However, characterizing spike patterns in the resulting data, and linking these patterns to behaviour, remains a challenging task. The lack of a rigorous mathematical language for variable numbers of events (spikes) emitted by multiple agents (neurons) is an important limiting factor. We introduce a new mathematical operation to decompose complex spike patterns into a set of simple, structured elements. This creates a mathematical language that allows comparing spike patterns across trials, detecting sub-patterns, and making links to behaviour via a clear distance measure. We first demonstrate the method using Neuropixel recordings from macaque motor cortex. We then apply the method to dual Utah array recordings from macaque prefrontal cortex, where this technique reveals previously unseen structure that can predict both memory-guided decisions and errors in a virtual-reality working memory task. These results demonstrate that this technique provides a powerful new approach to understand structure in the spike times of neural populations, at a scale that will continue to grow more and more rapidly in upcoming years.

Aggression experience and observation promote shared behavioral and neural changes

The ability to observe the social behavior of others and use observed information to bias future action is a fundamental building block of social cognition1,2. A foundational question is whether social observation and experience engage common circuit mechanisms that enable behavioral change. While classic studies on social learning have shown that aggressive behaviors can be learned through observation3, it remains unclear whether aggression observation promotes persistent neural changes that generalize to new contexts. Here, to directly compare the effects of aggression experience and observation at brain-wide scale, we develop a strategy to perform large-scale cell-type specific recordings across subcortical networks for social behavior control and learning. We record longitudinally while animals "train" through direct experience or observation, then probe shared differences in behavior and neural activity in a novel "hard" aggression context. Using supervised and unsupervised methods for behavioral quantification, we detect unique signatures of a shared behavioral strategy not present in animals with no training. During observation, we find widespread activation that mimics experience in networks for behavior generation, with critical differences in signals associated with reward and threat learning. After observation, we observe that changes persist into the novel aggression context, with increased similarity in the neural dynamics between experience and observation groups. Network-level modeling reveals persistent shared changes to a core aggression network, with widespread decoupling of inhibition from a key hypothalamic output region. This demonstrates that "experience-like" activity during observation can recruit a shared plasticity mechanism that biases behavior toward adaptive defensive strategies in new contexts.

Neuronal Sequences and dynamic coding of water-sucrose categorization in rat gustatory cortices

The gustatory system allows us to perceive and distinguish sweetness from water. We studied this phenomenon by recording neural activity in rats' anterior insular (aIC) and orbitofrontal (OFC) cortices while they categorized varying sucrose concentrations against water. Neurons in both aIC and OFC encoded the categorical distinction between sucrose and water rather than specific sucrose concentrations. Notably, aIC encoded this distinction faster than OFC. Conversely, the OFC slightly preceded the aIC in representing choice information, although both cortices encoded the rat's choices in parallel. Further analyses revealed dynamic and sequential encoding of sensory and categorical decisions, forming brief sequences of encoding neurons throughout the trial rather than long-lasting neuronal representations. Our findings, supported by single-cell, population decoding, and principal-component analysis (PCA), demonstrate that gustatory cortices employ neuronal sequences to compute sensorimotor transformations, from taste detection to categorical decisions, and continuously update this process as new taste information emerges using dynamic coding.

Accurate neural control of a hand prosthesis by posture-related activity in the primate grasping circuit

Brain-computer interfaces (BCIs) have the potential to restore hand movement for people with paralysis, but current devices still lack the fine control required to interact with objects of daily living. Following our understanding of cortical activity during arm reaches, hand BCI studies have focused primarily on velocity control. However, mounting evidence suggests that posture, and not velocity, dominates in hand-related areas. To explore whether this signal can causally control a prosthesis, we developed a BCI training paradigm centered on the reproduction of posture transitions. Monkeys trained with this protocol were able to control a multidimensional hand prosthesis with high accuracy, including execution of the very intricate precision grip. Analysis revealed that the posture signal in the target grasping areas was the main contributor to control. We present, for the first time, neural posture control of a multidimensional hand prosthesis, opening the door for future interfaces to leverage this additional information channel.

The genetically programmed rhythmic alteration of diurnal gene expression in the aged Arabidopsis leaves

The circadian clock regulates the daily pattern of temporal gene expression. In Arabidopsis, aging is associated with a shortening of the endogenous period of circadian rhythms under circadian conditions. However, the functional link between the circadian clock and aging under diurnal conditions and its physiological relevance remain elusive. In this study, we investigate and characterize the effect of aging on the waveforms of rhythmic gene expression patterns under light/dark cycles. Our analysis revealed that the diurnal rhythmic patterns of core clock genes undergo significant rhythmic alteration with phase shift and change of waveforms in aged plants compared to younger plants. Transcriptomic analysis indicated that this age-dependent rhythmic alteration occurs not only in core clock genes but also globally. Due to the rhythmic alteration patterns of the diurnal rhythmic gene expression, aged plants experience subjectively a shorter day and longer night. We also observed that genetic mutants of core clock component genes exhibited broadly yet distinctively altered changes in diurnal rhythmic gene expression patterns as aging progresses. Collectively, our findings support that age-dependent rhythmic alteration of diurnal gene expression rhythms reprograms the timetable of daily gene expression, leading to the physiological changes required for plant senescence.

The flow of reward information through neuronal ensembles in the accumbens

The nucleus accumbens shell (NAcSh) integrates reward information through diverse and specialized neuronal ensembles, influencing decision-making. By training rats in a probabilistic choice task and recording NAcSh neuronal activity, we found that rats adapt their choices based solely on the presence or absence of a sucrose reward, suggesting they build an internal representation of reward likelihood. We further demonstrate that NAcSh ensembles dynamically process different aspects of reward-guided behavior, with changes in composition and functional connections observed throughout the reinforcement learning process. The NAcSh forms a highly connected network characterized by a heavy-tailed distribution and the presence of neuronal hubs, facilitating efficient information flow. Reward delivery enhances mutual information, indicating increased communication between ensembles and network synchronization, whereas reward omission decreases it. Our findings reveal how reward information flows through dynamic NAcSh ensembles, whose flexible membership adapts as the rat learns to obtain rewards (energy) in an ever-changing environment.

A click-based electrocorticographic brain-computer interface enables long-term high-performance switch scan spelling

BACKGROUND

Brain-computer interfaces (BCIs) can restore communication for movement- and/or speech-impaired individuals by enabling neural control of computer typing applications. Single command click detectors provide a basic yet highly functional capability.

METHODS

We sought to test the performance and long-term stability of click decoding using a chronically implanted high density electrocorticographic (ECoG) BCI with coverage of the sensorimotor cortex in a human clinical trial participant (ClinicalTrials.gov, NCT03567213) with amyotrophic lateral sclerosis. We trained the participant's click detector using a small amount of training data (<44 min across 4 days) collected up to 21 days prior to BCI use, and then tested it over a period of 90 days without any retraining or updating.

RESULTS

Using a click detector to navigate a switch scanning speller interface, the study participant can maintain a median spelling rate of 10.2 characters per min. Though a transient reduction in signal power modulation can interrupt usage of a fixed model, a new click detector can achieve comparable performance despite being trained with even less data (<15 min, within 1 day).

CONCLUSIONS

These results demonstrate that a click detector can be trained with a small ECoG dataset while retaining robust performance for extended periods, providing functional text-based communication to BCI users.

Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks

Understanding the dynamical transformation of neural activity to behavior requires new capabilities to nonlinearly model, dissociate and prioritize behaviorally relevant neural dynamics and test hypotheses about the origin of nonlinearity. We present dissociative prioritized analysis of dynamics (DPAD), a nonlinear dynamical modeling approach that enables these capabilities with a multisection neural network architecture and training approach. Analyzing cortical spiking and local field potential activity across four movement tasks, we demonstrate five use-cases. DPAD enabled more accurate neural-behavioral prediction. It identified nonlinear dynamical transformations of local field potentials that were more behavior predictive than traditional power features. Further, DPAD achieved behavior-predictive nonlinear neural dimensionality reduction. It enabled hypothesis testing regarding nonlinearities in neural-behavioral transformation, revealing that, in our datasets, nonlinearities could largely be isolated to the mapping from latent cortical dynamics to behavior. Finally, DPAD extended across continuous, intermittently sampled and categorical behaviors. DPAD provides a powerful tool for nonlinear dynamical modeling and investigation of neural-behavioral data.

The design and implementation of multi-character classification scheme based on EEG signals of visual imagery

In visual-imagery-based brain-computer interface (VI-BCI), there are problems of singleness of imagination task and insufficient description of feature information, which seriously hinder the development and application of VI-BCI technology in the field of restoring communication. In this paper, we design and optimize a multi-character classification scheme based on electroencephalogram (EEG) signals of visual imagery (VI), which is used to classify 29 characters including 26 lowercase English letters and three punctuation marks. Firstly, a new paradigm of randomly presenting characters and including preparation stage is designed to acquire EEG signals and construct a multi-character dataset, which can eliminate the influence between VI tasks. Secondly, tensor data is obtained by the Morlet wavelet transform, and a feature extraction algorithm based on tensor-uncorrelated multilinear principal component analysis is used to extract high-quality features. Finally, three classifiers, namely support vector machine, K-nearest neighbor, and extreme learning machine, are employed for classifying multi-character, and the results are compared. The experimental results demonstrate that, the proposed scheme effectively extracts character features with minimal redundancy, weak correlation, and strong representation capability, and successfully achieves an average classification accuracy 97.59% for 29 characters, surpassing existing research in terms of both accuracy and quantity of classification. The present study designs a new paradigm for acquiring EEG signals of VI, and combines the Morlet wavelet transform and UMPCA algorithm to extract the character features, enabling multi-character classification in various classifiers. This research paves a novel pathway for establishing direct brain-to-world communication.

An instantaneous voice synthesis neuroprosthesis

Brain computer interfaces (BCIs) have the potential to restore communication to people who have lost the ability to speak due to neurological disease or injury. BCIs have been used to translate the neural correlates of attempted speech into text1-3. However, text communication fails to capture the nuances of human speech such as prosody, intonation and immediately hearing one's own voice. Here, we demonstrate a "brain-to-voice" neuroprosthesis that instantaneously synthesizes voice with closed-loop audio feedback by decoding neural activity from 256 microelectrodes implanted into the ventral precentral gyrus of a man with amyotrophic lateral sclerosis and severe dysarthria. We overcame the challenge of lacking ground-truth speech for training the neural decoder and were able to accurately synthesize his voice. Along with phonemic content, we were also able to decode paralinguistic features from intracortical activity, enabling the participant to modulate his BCI-synthesized voice in real-time to change intonation, emphasize words, and sing short melodies. These results demonstrate the feasibility of enabling people with paralysis to speak intelligibly and expressively through a BCI.

A method to find temporal structure of neuronal coactivity patterns with across-trial correlations

BACKGROUND

The across-trial correlation of neurons' coactivity patterns emerges to be important for information coding, but methods for finding their temporal structures remain largely unexplored.

NEW METHOD

In the present study, we propose a method to find time clusters in which coactivity patterns of neurons are correlated across trials. We transform the multidimensional neural activity at each timing into a coactivity pattern of binary states, and predict the coactivity patterns at different timings. We devise a method suitable for these coactivity pattern predictions, call general event prediction. Cross-temporal prediction accuracy is then used to estimate across-trial correlations between coactivity patterns at two timings. We extract time clusters from the cross-temporal prediction accuracy by a modified k-means algorithm.

RESULTS

The feasibility of the proposed method is verified through simulations based on ground truth. We apply the proposed method to a calcium imaging dataset recorded from the motor cortex of mice, and demonstrate time clusters of motor cortical coactivity patterns during a motor task.

COMPARISON WITH EXISTING METHODS

While the existing cosine similarity method, which does not account for across-trial correlation, shows temporal structures only for contralateral neural responses, the proposed method reveals those for both contralateral and ipsilateral neural responses, demonstrating the effect of across-trial correlations.

CONCLUSIONS

This study introduces a novel method for measuring the temporal structure of neuronal ensemble activity.

Real-time detection of motifs from semi-synthetic calcium imaging data with various noise types

The hippocampus is an important region responsible for memory formation. Some research evidence suggests that neurons in the hippocampus discharge in a sequential order to form specific activity patterns. Multiple replayed spike sequences may represent the recurrence of an individual's memory, and scientists refer to these replayed spatiotemporal sequences as motifs. Identification and online detection of these replayed spike trains are important for studying the mechanisms of memory formation. This work aims to evaluate the detection accuracy of motifs in Calcium imaging for semisynthetic data where manually identified motifs (originating from real data) are embedded into synthetic data containing various types of noise. It allows us to accurately pinpoint the timing of motif occurrences, thereby facilitating the evaluation of our method's real-time detection performance.The action potential of a neuron is typically completed within three milliseconds, while the replay time of a complete motif usually spans several hundred milliseconds. Ideally, motifs should be detected before they end in order to investigate their functions causally through intervention. For detecting motifs, we used a non-negative matrix factorization algorithm seqNMF. We conducted tests on our method's real-time detection performance for replay events. The method exhibited the ability to perform real-time detection of replayed spike sequences within a few tens of milliseconds. It was more effective in detecting large motifs than small motifs. However, we also found that the method is susceptible to increased jitter, suggesting the importance of time-precise recording for retrieving motifs robustly.

Beta: bursts of cognition

Beta oscillations are linked to the control of goal-directed processing of sensory information and the timing of motor output. Recent evidence demonstrates they are not sustained but organized into intermittent high-power bursts mediating timely functional inhibition. This implies there is a considerable moment-to-moment variation in the neural dynamics supporting cognition. Beta bursts thus offer new opportunities for studying how sensory inputs are selectively processed, reshaped by inhibitory cognitive operations and ultimately result in motor actions. Recent method advances reveal diversity in beta bursts that provide deeper insights into their function and the underlying neural circuit activity motifs. We propose that brain-wide, spatiotemporal patterns of beta bursting reflect various cognitive operations and that their dynamics reveal nonlinear aspects of cortical processing.

Temporally aligned segmentation and clustering (TASC) framework for behavior time series analysis

Behavior exhibits a complex spatiotemporal structure consisting of discrete sub-behaviors, or motifs. Continuous behavior data requires segmentation and clustering to reveal these embedded motifs. The popularity of automatic behavior quantification is growing, but existing solutions are often tailored to specific needs and are not designed for the time scale and precision required in many experimental and clinical settings. Here we propose a generalized framework with an iterative approach to refine both segmentation and clustering. Temporally aligned segmentation and clustering (TASC) uses temporal linear alignment to compute distances between and align the recurring behavior motifs in a multidimensional time series, enabling precise segmentation and clustering. We introduce an alternating-step process: evaluation of temporal neighbors against current cluster centroids using linear alignment, alternating with selecting the best non-overlapping segments and their subsequent re-clustering. The framework is evaluated on semi-synthetic and real-world experimental and clinical data, demonstrating enhanced segmentation and clustering, offering a better foundation for consequent research. The framework may be used to extend existing tools in the field of behavior research and may be applied to other domains requiring high precision of time series segmentation.

Dimensionality reduction beyond neural subspaces with slice tensor component analysis

Recent work has argued that large-scale neural recordings are often well described by patterns of coactivation across neurons. Yet the view that neural variability is constrained to a fixed, low-dimensional subspace may overlook higher-dimensional structure, including stereotyped neural sequences or slowly evolving latent spaces. Here we argue that task-relevant variability in neural data can also cofluctuate over trials or time, defining distinct 'covariability classes' that may co-occur within the same dataset. To demix these covariability classes, we develop sliceTCA (slice tensor component analysis), a new unsupervised dimensionality reduction method for neural data tensors. In three example datasets, including motor cortical activity during a classic reaching task in primates and recent multiregion recordings in mice, we show that sliceTCA can capture more task-relevant structure in neural data using fewer components than traditional methods. Overall, our theoretical framework extends the classic view of low-dimensional population activity by incorporating additional classes of latent variables capturing higher-dimensional structure.

Calibrating Bayesian Decoders of Neural Spiking Activity

Accurately decoding external variables from observations of neural activity is a major challenge in systems neuroscience. Bayesian decoders, which provide probabilistic estimates, are some of the most widely used. Here we show how, in many common settings, the probabilistic predictions made by traditional Bayesian decoders are overconfident. That is, the estimates for the decoded stimulus or movement variables are more certain than they should be. We then show how Bayesian decoding with latent variables, taking account of low-dimensional shared variability in the observations, can improve calibration, although additional correction for overconfidence is still needed. Using data from males, we examine (1) decoding the direction of grating stimuli from spike recordings in the primary visual cortex in monkeys, (2) decoding movement direction from recordings in the primary motor cortex in monkeys, (3) decoding natural images from multiregion recordings in mice, and (4) decoding position from hippocampal recordings in rats. For each setting, we characterize the overconfidence, and we describe a possible method to correct miscalibration post hoc. Properly calibrated Bayesian decoders may alter theoretical results on probabilistic population coding and lead to brain-machine interfaces that more accurately reflect confidence levels when identifying external variables.

Low frequency independent components: Internal neuromarkers linking cortical LFPs to behavior

Local field potentials (LFPs) in the primate motor cortex have been shown to reflect information related to volitional movements. However, LFPs are composite signals that receive contributions from multiple neural sources, producing a complex mix of component signals. Using a blind source separation approach, we examined the components of neural activity recorded using multielectrode arrays in motor areas of macaque monkeys during a grasping and lifting task. We found a set of independent components in the low-frequency LFP with high temporal and spatial consistency associated with each task stage. We observed that ICs often arise from electrodes distributed across multiple cortical areas and provide complementary information to external behavioral markers, specifically in task stage detection and trial alignment. Taken together, our results show that it is possible to separate useful independent components of the LFP associated with specific task-related events, potentially representing internal markers of transition between cortical network states.

A design and implementation of multi-character classification scheme based on motor imagery EEG signals

In the field of brain-to-text communication, it is difficult to finish highly dexterous behaviors of writing multi-character by motor-imagery-based brain-computer interface (MI-BCI), setting a barrier to restore communication in people who have lost the ability to move and speak. In this paper, we design and implement a multi-character classification scheme based on 29 characters of motor imagery (MI) electroencephalogram (EEG) signals, which contains 26 English letters and 3 punctuation marks. Firstly, we design a novel experimental paradigm to increase the variety of BCI inputs by asking subjects to imagine the movement of writing 29 characters instead of gross motor skills such as reaching or grasping. Secondly, because of the high dimension of EEG signals, we adopt power spectral density (PSD), principal components analysis (PCA), kernel principal components analysis (KPCA) respectively to decompose EEG signals and extract feature, and then test the results with pearson product-moment correlation coefficient (PCCs). Thirdly, we respectively employ k-nearest neighbor (kNN), support vector machine (SVM), extreme learning machine (ELM) and light gradient boosting machine (LightGBM) to classify 29 characters and compare the results. We have implemented a complete scheme, including paradigm design, signal acquisition, feature extraction and classification, which can effectively classify 29 characters. The experimental results show that the KPCA has the best feature extraction effect and the kNN has the highest classification accuracy, with the final classification accuracy reaching 96.2%, which is better than other studies.

Transient oscillations as computations for cognition: Analysis, modeling and function

Our view of neural oscillations is currently changing. The dominant picture of sustained oscillations is now often replaced by transient oscillations occurring in bursts. This phenomenon seems to be quite comprehensive, as it has been reported for different oscillation frequencies, including the theta, beta, and gamma bands, as well as cortical and subcortical regions in a variety of cognitive tasks and species. Here we review recent developments in their analysis, computational modeling, and functional roles. For the analysis of transient oscillations methods using lagged coherence and Hidden Markov Models have been developed and applied in recent studies to ascertain their transient nature and study their contribution to cognitive functions. Furthermore, computational models have been developed that account for their stochastic nature, which poses interesting functional constraints. Finally, as transient oscillations have been observed across many species, they are likely of functional significance and we consider challenges in characterizing their function.

Inaccessible time to visual awareness during attentional blinks in macaques and humans

Even when we attend to successive visual events, we often cannot notice an event occurring during a certain temporal window. Such an inaccessible time for visual awareness is known as "attentional blink" (AB). Whether AB is a phenomenon unique to humans or exists also in other animals is unclear. Using a dual-task paradigm shared between macaques and humans, we here demonstrate a nonhuman primate model of AB. Although macaques also showed behavioral signatures of AB, their AB effect lasted longer than that of humans. To map the relation between macaque and human ABs, we introduced a time warping analysis. The analysis revealed a formal structure behind the interspecies difference of AB; the temporal window of macaque AB was scaled from that of human AB. The present study opens the door to combining the approaches of neuroscience, psychophysics, and theoretical models to further identify a scale-invariant biological substrate of visual awareness.

A click-based electrocorticographic brain-computer interface enables long-term high-performance switch-scan spelling

Background

Brain-computer interfaces (BCIs) can restore communication in movement- and/or speech-impaired individuals by enabling neural control of computer typing applications. Single command "click" decoders provide a basic yet highly functional capability.

Methods

We sought to test the performance and long-term stability of click-decoding using a chronically implanted high density electrocorticographic (ECoG) BCI with coverage of the sensorimotor cortex in a human clinical trial participant (ClinicalTrials.gov, NCT03567213) with amyotrophic lateral sclerosis (ALS). We trained the participant's click decoder using a small amount of training data (< 44 minutes across four days) collected up to 21 days prior to BCI use, and then tested it over a period of 90 days without any retraining or updating.

Results

Using this click decoder to navigate a switch-scanning spelling interface, the study participant was able to maintain a median spelling rate of 10.2 characters per min. Though a transient reduction in signal power modulation interrupted testing with this fixed model, a new click decoder achieved comparable performance despite being trained with even less data (< 15 min, within one day).

Conclusion

These results demonstrate that a click decoder can be trained with a small ECoG dataset while retaining robust performance for extended periods, providing functional text-based communication to BCI users.

Post-stimulus encoding of decision confidence in EEG: toward a brain-computer interface for decision making

Objective.When making decisions, humans can evaluate how likely they are to be correct. If this subjective confidence could be reliably decoded from brain activity, it would be possible to build a brain-computer interface (BCI) that improves decision performance by automatically providing more information to the user if needed based on their confidence. But this possibility depends on whether confidence can be decoded right after stimulus presentation and before the response so that a corrective action can be taken in time. Although prior work has shown that decision confidence is represented in brain signals, it is unclear if the representation is stimulus-locked or response-locked, and whether stimulus-locked pre-response decoding is sufficiently accurate for enabling such a BCI.Approach.We investigate the neural correlates of confidence by collecting high-density electroencephalography (EEG) during a perceptual decision task with realistic stimuli. Importantly, we design our task to include a post-stimulus gap that prevents the confounding of stimulus-locked activity by response-locked activity and vice versa, and then compare with a task without this gap.Main results.We perform event-related potential and source-localization analyses. Our analyses suggest that the neural correlates of confidence are stimulus-locked, and that an absence of a post-stimulus gap could cause these correlates to incorrectly appear as response-locked. By preventing response-locked activity from confounding stimulus-locked activity, we then show that confidence can be reliably decoded from single-trial stimulus-locked pre-response EEG alone. We also identify a high-performance classification algorithm by comparing a battery of algorithms. Lastly, we design a simulated BCI framework to show that the EEG classification is accurate enough to build a BCI and that the decoded confidence could be used to improve decision making performance particularly when the task difficulty and cost of errors are high.Significance.Our results show feasibility of non-invasive EEG-based BCIs to improve human decision making.

Transient Oscillations of Neural Firing Rate Associated With Routing of Evidence in a Perceptual Decision

To form a perceptual decision, the brain must acquire samples of evidence from the environment and incorporate them in computations that mediate choice behavior. While much is known about the neural circuits that process sensory information and those that form decisions, less is known about the mechanisms that establish the functional linkage between them. We trained monkeys of both sexes to make difficult decisions about the net direction of visual motion under conditions that required trial-by-trial control of functional connectivity. In one condition, the motion appeared at different locations on different trials. In the other, two motion patches appeared, only one of which was informative. Neurons in the parietal cortex produced brief oscillations in their firing rate at the time routing was established: upon onset of the motion display when its location was unpredictable across trials, and upon onset of an attention cue that indicated in which of two locations an informative patch of dots would appear. The oscillation was absent when the stimulus location was fixed across trials. We interpret the oscillation as a manifestation of the mechanism that establishes the source and destination of flexibly routed information, but not the transmission of the information per se Significance Statement It has often been suggested that oscillations in neural activity might serve a role in routing information appropriately. We observe an oscillation in neural firing rate in the lateral intraparietal area consistent with such a role. The oscillations are transient. They coincide with the establishment of routing, but they do not appear to play a role in the transmission (or conveyance) of the routed information itself.

A neural mechanism for terminating decisions

The brain makes decisions by accumulating evidence until there is enough to stop and choose. Neural mechanisms of evidence accumulation are established in association cortex, but the site and mechanism of termination are unknown. Here, we show that the superior colliculus (SC) plays a causal role in terminating decisions, and we provide evidence for a mechanism by which this occurs. We recorded simultaneously from neurons in the lateral intraparietal area (LIP) and SC while monkeys made perceptual decisions. Despite similar trial-averaged activity, we found distinct single-trial dynamics in the two areas: LIP displayed drift-diffusion dynamics and SC displayed bursting dynamics. We hypothesized that the bursts manifest a threshold mechanism applied to signals represented in LIP to terminate the decision. Consistent with this hypothesis, SC inactivation produced behavioral effects diagnostic of an impaired threshold sensor and prolonged the buildup of activity in LIP. The results reveal the transformation from deliberation to commitment.

SpikeShip: A method for fast, unsupervised discovery of high-dimensional neural spiking patterns

Neural coding and memory formation depend on temporal spiking sequences that span high-dimensional neural ensembles. The unsupervised discovery and characterization of these spiking sequences requires a suitable dissimilarity measure to spiking patterns, which can then be used for clustering and decoding. Here, we present a new dissimilarity measure based on optimal transport theory called SpikeShip, which compares multi-neuron spiking patterns based on all the relative spike-timing relationships among neurons. SpikeShip computes the optimal transport cost to make all the relative spike-timing relationships (across neurons) identical between two spiking patterns. We show that this transport cost can be decomposed into a temporal rigid translation term, which captures global latency shifts, and a vector of neuron-specific transport flows, which reflect inter-neuronal spike timing differences. SpikeShip can be effectively computed for high-dimensional neuronal ensembles, has a low (linear) computational cost that has the same order as the spike count, and is sensitive to higher-order correlations. Furthermore, SpikeShip is binless, can handle any form of spike time distributions, is not affected by firing rate fluctuations, can detect patterns with a low signal-to-noise ratio, and can be effectively combined with a sliding window approach. We compare the advantages and differences between SpikeShip and other measures like SPIKE and Victor-Purpura distance. We applied SpikeShip to large-scale Neuropixel recordings during spontaneous activity and visual encoding. We show that high-dimensional spiking sequences detected via SpikeShip reliably distinguish between different natural images and different behavioral states. These spiking sequences carried complementary information to conventional firing rate codes. SpikeShip opens new avenues for studying neural coding and memory consolidation by rapid and unsupervised detection of temporal spiking patterns in high-dimensional neural ensembles.

How our understanding of memory replay evolves

Memory reactivations and replay, widely reported in the hippocampus and cortex across species, have been implicated in memory consolidation, planning, and spatial and skill learning. Technological advances in electrophysiology, calcium imaging, and human neuroimaging techniques have enabled neuroscientists to measure large-scale neural activity with increasing spatiotemporal resolution and have provided opportunities for developing robust analytic methods to identify memory replay. In this article, we first review a large body of historically important and representative memory replay studies from the animal and human literature. We then discuss our current understanding of memory replay functions in learning, planning, and memory consolidation and further discuss the progress in computational modeling that has contributed to these improvements. Next, we review past and present analytic methods for replay analyses and discuss their limitations and challenges. Finally, looking ahead, we discuss some promising analytic methods for detecting nonstereotypical, behaviorally nondecodable structures from large-scale neural recordings. We argue that seamless integration of multisite recordings, real-time replay decoding, and closed-loop manipulation experiments will be essential for delineating the role of memory replay in a wide range of cognitive and motor functions.

Dimensionality reduction of calcium-imaged neuronal population activity

Calcium imaging has been widely adopted for its ability to record from large neuronal populations. To summarize the time course of neural activity, dimensionality reduction methods, which have been applied extensively to population spiking activity, may be particularly useful. However, it is unclear if the dimensionality reduction methods applied to spiking activity are appropriate for calcium imaging. We thus carried out a systematic study of design choices based on standard dimensionality reduction methods. We also developed a method to perform deconvolution and dimensionality reduction simultaneously (Calcium Imaging Linear Dynamical System, CILDS). CILDS most accurately recovered the single-trial, low-dimensional time courses from simulated calcium imaging data. CILDS also outperformed the other methods on calcium imaging recordings from larval zebrafish and mice. More broadly, this study represents a foundation for summarizing calcium imaging recordings of large neuronal populations using dimensionality reduction in diverse experimental settings.

Long-term stability of single neuron activity in the motor system

How an established behavior is retained and consistently produced by a nervous system in constant flux remains a mystery. One possible solution to ensure long-term stability in motor output is to fix the activity patterns of single neurons in the relevant circuits. Alternatively, activity in single cells could drift over time provided that the population dynamics are constrained to produce the same behavior. To arbitrate between these possibilities, we recorded single-unit activity in motor cortex and striatum continuously for several weeks as rats performed stereotyped motor behaviors-both learned and innate. We found long-term stability in single neuron activity patterns across both brain regions. A small amount of drift in neural activity, observed over weeks of recording, could be explained by concomitant changes in task-irrelevant aspects of the behavior. These results suggest that long-term stable behaviors are generated by single neuron activity patterns that are themselves highly stable.

Temporal scaling of human scalp-recorded potentials

Much of human behavior is governed by common processes that unfold over varying timescales. Standard event-related potential analysis assumes fixed-duration responses relative to experimental events. However, recent single-unit recordings in animals have revealed neural activity scales to span different durations during behaviors demanding flexible timing. Here, we employed a general linear modeling approach using a combination of fixed-duration and variable-duration regressors to unmix fixed-time and scaled-time components in human magneto-/electroencephalography (M/EEG) data. We use this to reveal consistent temporal scaling of human scalp-recorded potentials across four independent electroencephalogram (EEG) datasets, including interval perception, production, prediction, and value-based decision making. Between-trial variation in the temporally scaled response predicts between-trial variation in subject reaction times, demonstrating the relevance of this temporally scaled signal for temporal variation in behavior. Our results provide a general approach for studying flexibly timed behavior in the human brain.

The brain time toolbox, a software library to retune electrophysiology data to brain dynamics

Human thought is highly flexible, achieved by evolving patterns of brain activity across groups of cells. Neuroscience aims to understand cognition in the brain by analysing these intricate patterns. We argue that this goal is impeded by the time format of our data-clock time. The brain is a system with its own dynamics and regime of time, with no intrinsic concern for the human-invented second. Here, we present the Brain Time Toolbox, a software library that retunes electrophysiology data in line with oscillations that orchestrate neural patterns of cognition. These oscillations continually slow down, speed up and undergo abrupt changes, introducing a disharmony between the brain's internal regime and clock time. The toolbox overcomes this disharmony by warping the data to the dynamics of coordinating oscillations, setting oscillatory cycles as the data's new time axis. This enables the study of neural patterns as they unfold in the brain, aiding neuroscientific enquiry into dynamic cognition. In support of this, we demonstrate that the toolbox can reveal results that are absent in a default clock time format.

Synaptic Plasticity Is Predicted by Spatiotemporal Firing Rate Patterns and Robust to In Vivo-like Variability

Synaptic plasticity, the experience-induced change in connections between neurons, underlies learning and memory in the brain. Most of our understanding of synaptic plasticity derives from in vitro experiments with precisely repeated stimulus patterns; however, neurons exhibit significant variability in vivo during repeated experiences. Further, the spatial pattern of synaptic inputs to the dendritic tree influences synaptic plasticity, yet is not considered in most synaptic plasticity rules. Here, we investigate how spatiotemporal synaptic input patterns produce plasticity with in vivo-like conditions using a data-driven computational model with a plasticity rule based on calcium dynamics. Using in vivo spike train recordings as inputs to different size clusters of spines, we show that plasticity is strongly robust to trial-to-trial variability of spike timing. In addition, we derive general synaptic plasticity rules describing how spatiotemporal patterns of synaptic inputs control the magnitude and direction of plasticity. Synapses that strongly potentiated have greater firing rates and calcium concentration later in the trial, whereas strongly depressing synapses have hiring firing rates early in the trial. The neighboring synaptic activity influences the direction and magnitude of synaptic plasticity, with small clusters of spines producing the greatest increase in synaptic strength. Together, our results reveal that calcium dynamics can unify diverse plasticity rules and reveal how spatiotemporal firing rate patterns control synaptic plasticity.

Estimating muscle activation from EMG using deep learning-based dynamical systems models

Objective. To study the neural control of movement, it is often necessary to estimate how muscles are activated across a variety of behavioral conditions. One approach is to try extracting the underlying neural command signal to muscles by applying latent variable modeling methods to electromyographic (EMG) recordings. However, estimating the latent command signal that underlies muscle activation is challenging due to its complex relation with recorded EMG signals. Common approaches estimate each muscle's activation independently or require manual tuning of model hyperparameters to preserve behaviorally-relevant features.Approach. Here, we adapted AutoLFADS, a large-scale, unsupervised deep learning approach originally designed to de-noise cortical spiking data, to estimate muscle activation from multi-muscle EMG signals. AutoLFADS uses recurrent neural networks to model the spatial and temporal regularities that underlie multi-muscle activation.Main results. We first tested AutoLFADS on muscle activity from the rat hindlimb during locomotion and found that it dynamically adjusts its frequency response characteristics across different phases of behavior. The model produced single-trial estimates of muscle activation that improved prediction of joint kinematics as compared to low-pass or Bayesian filtering. We also applied AutoLFADS to monkey forearm muscle activity recorded during an isometric wrist force task. AutoLFADS uncovered previously uncharacterized high-frequency oscillations in the EMG that enhanced the correlation with measured force. The AutoLFADS-inferred estimates of muscle activation were also more closely correlated with simultaneously-recorded motor cortical activity than were other tested approaches.Significance.This method leverages dynamical systems modeling and artificial neural networks to provide estimates of muscle activation for multiple muscles. Ultimately, the approach can be used for further studies of multi-muscle coordination and its control by upstream brain areas, and for improving brain-machine interfaces that rely on myoelectric control signals.

A reawakening of Machine Learning Application in Unmanned Aerial Vehicle: Future Research Motivation

Machine learning (ML) entails artificial procedures that improve robotically through experience and using data. Supervised, unsupervised, semi-supervised, and Reinforcement Learning (RL) are the main types of ML. This study mainly focuses on RL and Deep learning, since necessitates mainly sequential and consecutive decision-making context. This is a comparison to supervised and non-supervised learning due to the interactive nature of the environment. Exploiting a forthcoming accumulative compensation and its stimulus of machines, complex policy decisions. The study further analyses and presents ML perspectives depicting state-of-the-art developments with advancement, relatively depicting the future trend of RL based on its applicability in technology. It's a challenge to an Internet of Things (IoT) and demonstrates what possibly can be adopted as a solution. This study presented a summarized perspective on identified arenas on the analysis of RL. The study scrutinized that a reasonable number of the techniques engrossed in alternating policy values instead of modifying other gears in an exact state of intellectual. The study presented a strong foundation for the current studies to be adopted by the researchers from different research backgrounds to develop models, and architectures that are relevant.

Removing nonlinear misalignment in neuronal spike trains using the Fisher-Rao registration framework

BACKGROUND

The temporal precision in neural spike train data is critically important for understanding functional mechanism in the nervous systems. However, the timing variability of spiking activity can be highly nonlinear in practical observations due to behavioral variability or unobserved/unobservable cognitive states.

NEW METHOD

In this study, we propose to adopt a powerful nonlinear method, referred to as the Fisher-Rao Registration (FRR), to remove such nonlinear phase variability in discrete neuronal spike trains. We also develop a smoothing procedure on the discrete spike train data in order to use the FRR framework.

COMPARISON WITH EXISTING METHODS

We systematically compare the FRR with the state-of-the-art linear and nonlinear methods in terms of model efficiency and effectiveness.

RESULTS

We show that the FRR has superior performance and the advantages are well illustrated with simulation and real experimental data.

CONCLUSIONS

It is found the FRR framework provides more appropriate alignment performance to understand the temporal variability in neuronal spike trains.

Acceleration of the SPADE Method Using a Custom-Tailored FP-Growth Implementation

The SPADE (spatio-temporal Spike PAttern Detection and Evaluation) method was developed to find reoccurring spatio-temporal patterns in neuronal spike activity (parallel spike trains). However, depending on the number of spike trains and the length of recording, this method can exhibit long runtimes. Based on a realistic benchmark data set, we identified that the combination of pattern mining (using the FP-Growth algorithm) and the result filtering account for 85–90% of the method's total runtime. Therefore, in this paper, we propose a customized FP-Growth implementation tailored to the requirements of SPADE, which significantly accelerates pattern mining and result filtering. Our version allows for parallel and distributed execution, and due to the improvements made, an execution on heterogeneous and low-power embedded devices is now also possible. The implementation has been evaluated using a traditional workstation based on an Intel Broadwell Xeon E5-1650 v4 as a baseline. Furthermore, the heterogeneous microserver platform RECS|Box has been used for evaluating the implementation on two HiSilicon Hi1616 (Kunpeng 916), an Intel Coffee Lake-ER Xeon E-2276ME, an Intel Broadwell Xeon D-D1577, and three NVIDIA Tegra devices (Jetson AGX Xavier, Jetson Xavier NX, and Jetson TX2). Depending on the platform, our implementation is between 27 and 200 times faster than the original implementation. At the same time, the energy consumption was reduced by up to two orders of magnitude.

Statistical neuroscience in the single trial limit

Individual neurons often produce highly variable responses over nominally identical trials, reflecting a mixture of intrinsic 'noise' and systematic changes in the animal's cognitive and behavioral state. Disentangling these sources of variability is of great scientific interest in its own right, but it is also increasingly inescapable as neuroscientists aspire to study more complex and naturalistic animal behaviors. In these settings, behavioral actions never repeat themselves exactly and may rarely do so even approximately. Thus, new statistical methods that extract reliable features of neural activity using few, if any, repeated trials are needed. Accurate statistical modeling in this severely trial-limited regime is challenging, but still possible if simplifying structure in neural data can be exploited. We review recent works that have identified different forms of simplifying structure - including shared gain modulations across neural subpopulations, temporal smoothness in neural firing rates, and correlations in responses across behavioral conditions - and exploited them to reveal novel insights into the trial-by-trial operation of neural circuits.

Selection of Essential Neural Activity Timesteps for Intracortical Brain-Computer Interface Based on Recurrent Neural Network

Intracortical brain–computer interfaces (iBCIs) translate neural activity into control commands, thereby allowing paralyzed persons to control devices via their brain signals. Recurrent neural networks (RNNs) are widely used as neural decoders because they can learn neural response dynamics from continuous neural activity. Nevertheless, excessively long or short input neural activity for an RNN may decrease its decoding performance. Based on the temporal attention module exploiting relations in features over time, we propose a temporal attention-aware timestep selection (TTS) method that improves the interpretability of the salience of each timestep in an input neural activity. Furthermore, TTS determines the appropriate input neural activity length for accurate neural decoding. Experimental results show that the proposed TTS efficiently selects 28 essential timesteps for RNN-based neural decoders, outperforming state-of-the-art neural decoders on two nonhuman primate datasets (R2=0.76±0.05 for monkey Indy and CC=0.91±0.01 for monkey N). In addition, it reduces the computation time for offline training (reducing 5–12%) and online prediction (reducing 16–18%). When visualizing the attention mechanism in TTS, the preparatory neural activity is consecutively highlighted during arm movement, and the most recent neural activity is highlighted during the resting state in nonhuman primates. Selecting only a few essential timesteps for an RNN-based neural decoder provides sufficient decoding performance and requires only a short computation time.

Clustering and control for adaptation uncovers time-warped spike time patterns in cortical networks in vivo

How information in the nervous system is encoded by patterns of action potentials (i.e. spikes) remains an open question. Multi-neuron patterns of single spikes are a prime candidate for spike time encoding but their temporal variability requires further characterisation. Here we show how known sources of spike count variability affect stimulus-evoked spike time patterns between neurons separated over multiple layers and columns of adult rat somatosensory cortex in vivo. On subsets of trials (clusters) and after controlling for stimulus-response adaptation, spike time differences between pairs of neurons are “time-warped” (compressed/stretched) by trial-to-trial changes in shared excitability, explaining why fixed spike time patterns and noise correlations are seldom reported. We show that predicted cortical state is correlated between groups of 4 neurons, introducing the possibility of spike time pattern modulation by population-wide trial-to-trial changes in excitability (i.e. cortical state). Under the assumption of state-dependent coding, we propose an improved potential encoding capacity.

Urgent Decision Making: Resolving Visuomotor Interactions at High Temporal Resolution

Measuring when exactly perceptual decisions are made is crucial for defining how the activation of specific neurons contributes to behavior. However, in traditional, nonurgent visuomotor tasks, the uncertainty of this temporal measurement is very large. This is a problem not only for delimiting the capacity of perception, but also for correctly interpreting the functional roles ascribed to choice-related neuronal responses. In this article, we review psychophysical, neurophysiological, and modeling work based on urgent visuomotor tasks in which this temporal uncertainty can be effectively overcome. The cornerstone of this work is a novel behavioral metric that describes the evolution of the subject's perceptual judgment moment by moment, allowing us to resolve numerous perceptual events that unfold within a few tens of milliseconds. In this framework, the neural distinction between perceptual evaluation and motor selection processes becomes particularly clear, as the conclusion of one is not contingent on that of the other. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Supporting generalization in non-human primate behavior by tapping into structural knowledge: Examples from sensorimotor mappings, inference, and decision-making

The complex behaviors we ultimately wish to understand are far from those currently used in systems neuroscience laboratories. A salient difference are the closed loops between action and perception prominently present in natural but not laboratory behaviors. The framework of reinforcement learning and control naturally wades across action and perception, and thus is poised to inform the neurosciences of tomorrow, not only from a data analyses and modeling framework, but also in guiding experimental design. We argue that this theoretical framework emphasizes active sensing, dynamical planning, and the leveraging of structural regularities as key operations for intelligent behavior within uncertain, time-varying environments. Similarly, we argue that we may study natural task strategies and their neural circuits without over-training animals when the tasks we use tap into our animal's structural knowledge. As proof-of-principle, we teach animals to navigate through a virtual environment - i.e., explore a well-defined and repetitive structure governed by the laws of physics - using a joystick. Once these animals have learned to 'drive', without further training they naturally (i) show zero- or one-shot learning of novel sensorimotor contingencies, (ii) infer the evolving path of dynamically changing latent variables, and (iii) make decisions consistent with maximizing reward rate. Such task designs allow for the study of flexible and generalizable, yet controlled, behaviors. In turn, they allow for the exploitation of pillars of intelligence - flexibility, prediction, and generalization -, properties whose neural underpinning have remained elusive.

Low-dimensional learned feature spaces quantify individual and group differences in vocal repertoires

Increases in the scale and complexity of behavioral data pose an increasing challenge for data analysis. A common strategy involves replacing entire behaviors with small numbers of handpicked, domain-specific features, but this approach suffers from several crucial limitations. For example, handpicked features may miss important dimensions of variability, and correlations among them complicate statistical testing. Here, by contrast, we apply the variational autoencoder (VAE), an unsupervised learning method, to learn features directly from data and quantify the vocal behavior of two model species: the laboratory mouse and the zebra finch. The VAE converges on a parsimonious representation that outperforms handpicked features on a variety of common analysis tasks, enables the measurement of moment-by-moment vocal variability on the timescale of tens of milliseconds in the zebra finch, provides strong evidence that mouse ultrasonic vocalizations do not cluster as is commonly believed, and captures the similarity of tutor and pupil birdsong with qualitatively higher fidelity than previous approaches. In all, we demonstrate the utility of modern unsupervised learning approaches to the quantification of complex and high-dimensional vocal behavior.

High-performance brain-to-text communication via handwriting

Brain-computer interfaces (BCIs) can restore communication to people who have lost the ability to move or speak. So far, a major focus of BCI research has been on restoring gross motor skills, such as reaching and grasping^1-5 or point-and-click typing with a computer cursor^6,7. However, rapid sequences of highly dexterous behaviours, such as handwriting or touch typing, might enable faster rates of communication. Here we developed an intracortical BCI that decodes attempted handwriting movements from neural activity in the motor cortex and translates it to text in real time, using a recurrent neural network decoding approach. With this BCI, our study participant, whose hand was paralysed from spinal cord injury, achieved typing speeds of 90 characters per minute with 94.1% raw accuracy online, and greater than 99% accuracy offline with a general-purpose autocorrect. To our knowledge, these typing speeds exceed those reported for any other BCI, and are comparable to typical smartphone typing speeds of individuals in the age group of our participant (115 characters per minute)⁸. Finally, theoretical considerations explain why temporally complex movements, such as handwriting, may be fundamentally easier to decode than point-to-point movements. Our results open a new approach for BCIs and demonstrate the feasibility of accurately decoding rapid, dexterous movements years after paralysis.

Odor sampling strategies in mice with genetically altered olfactory responses

Peripheral sensory cells and the central neuronal circuits that monitor environmental changes to drive behaviors should be adapted to match the behaviorally relevant kinetics of incoming stimuli, be it the detection of sound frequencies, the speed of moving objects or local temperature changes. Detection of odorants begins with the activation of olfactory receptor neurons in the nasal cavity following inhalation of air and airborne odorants carried therein. Thus, olfactory receptor neurons are stimulated in a rhythmic and repeated fashion that is determined by the breathing or sniffing frequency that can be controlled and altered by the animal. This raises the question of how the response kinetics of olfactory receptor neurons are matched to the imposed stimulation frequency and if, vice versa, the kinetics of olfactory receptor neuron responses determine the sniffing frequency. We addressed this question by using a mouse model that lacks the K⁺-dependent Na⁺/Ca²⁺ exchanger 4 (NCKX4), which results in markedly slowed response termination of olfactory receptor neuron responses and hence changes the temporal response kinetics of these neurons. We monitored sniffing behaviors of freely moving wildtype and NCKX4 knockout mice while they performed olfactory Go/NoGo discrimination tasks. Knockout mice performed with similar or, surprisingly, better accuracy compared to wildtype mice, but chose, depending on the task, different odorant sampling durations depending on the behavioral demands of the odorant identification task. Similarly, depending on the demands of the behavioral task, knockout mice displayed a lower basal breathing frequency prior to odorant sampling, a possible mechanism to increase the dynamic range for changes in sniffing frequency during odorant sampling. Overall, changes in sniffing behavior between wildtype and NCKX4 knockout mice were subtle, suggesting that, at least for the particular odorant-driven task we used, slowed response termination of the odorant-induced receptor neuron response either has a limited detrimental effect on odorant-driven behavior or mice are able to compensate via an as yet unknown mechanism.

From Neurons to Cognition: Technologies for Precise Recording of Neural Activity Underlying Behavior

Understanding how brain activity encodes information and controls behavior is a long-standing question in neuroscience. This complex problem requires converging efforts from neuroscience and engineering, including technological solutions to perform high-precision and large-scale recordings of neuronal activity in vivo as well as unbiased methods to reliably measure and quantify behavior. Thanks to advances in genetics, molecular biology, engineering, and neuroscience, in recent decades, a variety of optical imaging and electrophysiological approaches for recording neuronal activity in awake animals have been developed and widely applied in the field. Moreover, sophisticated computer vision and machine learning algorithms have been developed to analyze animal behavior. In this review, we provide an overview of the current state of technology for neuronal recordings with a focus on optical and electrophysiological methods in rodents. In addition, we discuss areas that future technological development will need to cover in order to further our understanding of the neural activity underlying behavior.