Demixed principal component analysis of neural population data

Between persistently active and activity-silent frameworks: novel vistas on the cellular basis of working memory

Recent work has revealed important new discoveries on the cellular mechanisms of working memory (WM). These findings have motivated several seemingly conflicting theories on the mechanisms of short‐term memory maintenance. Here, we summarize the key insights gained from these new experiments and critically evaluate them in light of three hypotheses: classical persistent activity, activity‐silent, and dynamic coding. The experiments discussed include the first direct demonstration of persistently active neurons in the human medial temporal lobe that form static attractors with relevance to WM, single‐neuron recordings in the macaque prefrontal cortex that show evidence for both persistent and more dynamic types of WM representations, and noninvasive neuroimaging in humans that argues for activity‐silent representations. A key insight that emerges from these new results is that there are several neural mechanisms that support the maintenance of information in WM. Finally, based on established cognitive theories of WM, we propose a coherent model that encompasses these seemingly contradictory results. We propose that the three neuronal mechanisms of persistent activity, activity‐silent, and dynamic coding map well onto the cognitive levels of information processing (within focus of attention, activated long‐term memory, and central executive) that Cowan's WM model proposes.

Flexible Sensorimotor Computations through Rapid Reconfiguration of Cortical Dynamics

Neural mechanisms that support flexible sensorimotor computations are not well understood. In a dynamical system whose state is determined by interactions among neurons, computations can be rapidly reconfigured by controlling the system's inputs and initial conditions. To investigate whether the brain employs such control mechanisms, we recorded from the dorsomedial frontal cortex of monkeys trained to measure and produce time intervals in two sensorimotor contexts. The geometry of neural trajectories during the production epoch was consistent with a mechanism wherein the measured interval and sensorimotor context exerted control over cortical dynamics by adjusting the system's initial condition and input, respectively. These adjustments, in turn, set the speed at which activity evolved in the production epoch, allowing the animal to flexibly produce different time intervals. These results provide evidence that the language of dynamical systems can be used to parsimoniously link brain activity to sensorimotor computations.

Comparing Open-Source Toolboxes for Processing and Analysis of Spike and Local Field Potentials Data

Analysis of spike and local field potential (LFP) data is an essential part of neuroscientific research. Today there exist many open-source toolboxes for spike and LFP data analysis implementing various functionality. Here we aim to provide a practical guidance for neuroscientists in the choice of an open-source toolbox best satisfying their needs. We overview major open-source toolboxes for spike and LFP data analysis as well as toolboxes with tools for connectivity analysis, dimensionality reduction and generalized linear modeling. We focus on comparing toolboxes functionality, statistical and visualization tools, documentation and support quality. To give a better insight, we compare and illustrate functionality of the toolboxes on open-access dataset or simulated data and make corresponding MATLAB scripts publicly available.

A New Nonlinear Sparse Component Analysis for a Biologically Plausible Model of Neurons

It is known that brain can create a sparse representation of the environment in both sensory and mnemonic forms (Olshausen & Field, 2004). Such sparse representation can be combined in downstream areas to create rich multisensory responses to support various cognitive and motor functions. Determining the components present in neuronal responses in a given region is key to deciphering its functional role and connection with upstream areas. One approach for parsing out various sources of information in a single neuron is by using linear blind source separation (BSS) techniques. However, applying linear techniques to neuronal spiking activity is likely to be suboptimal due to inherent and unknown nonlinearity of neuronal responses to inputs. This letter proposes a nonlinear sparse component analysis (SCA) method to separate jointly sparse inputs to neurons with post summation nonlinearity, or SCA for post-nonlinear neurons (SCAPL). Specifically, a linear clustering approach followed by principal curve regression (PCR) and a nonlinear curve fitting are used to separate sources. Analysis using simulated data shows that SCAPL accuracy outperforms ones obtained by linear SCA, as well as other separating methods, including linear independent and principal component analyses. In SCAPL, the number of derived sparse components is not limited by the number of neurons, unlike most BSS methods. Furthermore, this method allows for a broad range of post-summation nonlinearities that could differ among neurons. The sensitivity of our method to noise, joint sparseness, degree, and shape of nonlinearity and mixing ill conditions is discussed and compared to existing methods. Our results show that the proposed method can successfully separate input components in a population of neurons provided that they are temporally sparse to some degree. Application of SCAPL should facilitate comparison of functional roles across regions by parsing various elements present in a region.

Low Dimensionality, High Robustness in Neural Population Dynamics

Neuronal populations respond within a small number of relevant dimensions. New research by Trautmann et al. (2019) shows that spike sorting is not necessary to extract the important features of this low-dimensional population signal. Combined responses of multiple neurons (multiunit activity) only generate small changes in the extracted signals.

High-dimensional geometry of population responses in visual cortex

A neuronal population encodes information most efficiently when its stimulus responses are high-dimensional and uncorrelated, and most robustly when they are lower-dimensional and correlated. Here we analysed the dimensionality of the encoding of natural images by large populations of neurons in the visual cortex of awake mice. The evoked population activity was high-dimensional, and correlations obeyed an unexpected power law: the nth principal component variance scaled as 1/n. This scaling was not inherited from the power law spectrum of natural images, because it persisted after stimulus whitening. We proved mathematically that if the variance spectrum was to decay more slowly then the population code could not be smooth, allowing small changes in input to dominate population activity. The theory also predicts larger power-law exponents for lower-dimensional stimulus ensembles, which we validated experimentally. These results suggest that coding smoothness may represent a fundamental constraint that determines correlations in neural population codes.

Accurate Estimation of Neural Population Dynamics without Spike Sorting

A central goal of systems neuroscience is to relate an organism's neural activity to behavior. Neural population analyses often reduce the data dimensionality to focus on relevant activity patterns. A major hurdle to data analysis is spike sorting, and this problem is growing as the number of recorded neurons increases. Here, we investigate whether spike sorting is necessary to estimate neural population dynamics. The theory of random projections suggests that we can accurately estimate the geometry of low-dimensional manifolds from a small number of linear projections of the data. We recorded data using Neuropixels probes in motor cortex of nonhuman primates and reanalyzed data from three previous studies and found that neural dynamics and scientific conclusions are quite similar using multiunit threshold crossings rather than sorted neurons. This finding unlocks existing data for new analyses and informs the design and use of new electrode arrays for laboratory and clinical use.

Neural correlates of sparse coding and dimensionality reduction

Supported by recent computational studies, there is increasing evidence that a wide range of neuronal responses can be understood as an emergent property of nonnegative sparse coding (NSC), an efficient population coding scheme based on dimensionality reduction and sparsity constraints. We review evidence that NSC might be employed by sensory areas to efficiently encode external stimulus spaces, by some associative areas to conjunctively represent multiple behaviorally relevant variables, and possibly by the basal ganglia to coordinate movement. In addition, NSC might provide a useful theoretical framework under which to understand the often complex and nonintuitive response properties of neurons in other brain areas. Although NSC might not apply to all brain areas (for example, motor or executive function areas) the success of NSC-based models, especially in sensory areas, warrants further investigation for neural correlates in other regions.

Characterizing the nonlinear structure of shared variability in cortical neuron populations using latent variable models

Sensory neurons often have variable responses to repeated presentations of the same stimulus, which can significantly degrade the stimulus information contained in those responses. This information can in principle be preserved if variability is shared across many neurons, but depends on the structure of the shared variability and its relationship to sensory encoding at the population level. The structure of this shared variability in neural activity can be characterized by latent variable models, although they have thus far typically been used under restrictive mathematical assumptions, such as assuming linear transformations between the latent variables and neural activity. Here we introduce two nonlinear latent variable models for analyzing large-scale neural recordings. We first present a general nonlinear latent variable model that is agnostic to the stimulus tuning properties of the individual neurons, and is hence well suited for exploring neural populations whose tuning properties are not well characterized. This motivates a second class of model, the Generalized Affine Model, which simultaneously determines each neuron's stimulus selectivity and a set of latent variables that modulate these stimulus-driven responses both additively and multiplicatively. While these approaches can detect very general nonlinear relationships in shared neural variability, we find that neural activity recorded in anesthetized primary visual cortex (V1) is best described by a single additive and single multiplicative latent variable, i.e., an "affine model". In contrast, application of the same models to recordings in awake macaque prefrontal cortex discover more general nonlinearities to compactly describe the population response variability. These results thus demonstrate how nonlinear latent variable models can be used to describe population variability, and suggest that a range of methods is necessary to study different brain regions under different experimental conditions.

State-dependent geometry of population activity in rat auditory cortex

The accuracy of the neural code depends on the relative embedding of signal and noise in the activity of neural populations. Despite a wealth of theoretical work on population codes, there are few empirical characterizations of the high-dimensional signal and noise subspaces. We studied the geometry of population codes in the rat auditory cortex across brain states along the activation-inactivation continuum, using sounds varying in difference and mean level across the ears. As the cortex becomes more activated, single-hemisphere populations go from preferring contralateral loud sounds to a symmetric preference across lateralizations and intensities, gain-modulation effectively disappears, and the signal and noise subspaces become approximately orthogonal to each other and to the direction corresponding to global activity modulations. Level-invariant decoding of sound lateralization also becomes possible in the active state. Our results provide an empirical foundation for the geometry and state-dependence of cortical population codes.

The amplitude in periodic neural state trajectories underlies the tempo of rhythmic tapping

Our motor commands can be exquisitely timed according to the demands of the environment, and the ability to generate rhythms of different tempos is a hallmark of musical cognition. Yet, the neuronal underpinnings behind rhythmic tapping remain elusive. Here, we found that the activity of hundreds of primate medial premotor cortices (MPCs; pre-supplementary motor area [preSMA] and supplementary motor area [SMA]) neurons show a strong periodic pattern that becomes evident when their responses are projected into a state space using dimensionality reduction analysis. We show that different tapping tempos are encoded by circular trajectories that travelled at a constant speed but with different radii, and that this neuronal code is highly resilient to the number of participating neurons. Crucially, the changes in the amplitude of the oscillatory dynamics in neuronal state space are a signature of duration encoding during rhythmic timing, regardless of whether it is guided by an external metronome or is internally controlled and is not the result of repetitive motor commands. This dynamic state signal predicted the duration of the rhythmically produced intervals on a trial-by-trial basis. Furthermore, the increase in variability of the neural trajectories accounted for the scalar property, a hallmark feature of temporal processing across tasks and species. Finally, we found that the interval-dependent increments in the radius of periodic neural trajectories are the result of a larger number of neurons engaged in the production of longer intervals. Our results support the notion that rhythmic timing during tapping behaviors is encoded in the radial curvature of periodic MPC neural population trajectories.

Working memory capacity is enhanced by distributed prefrontal activation and invariant temporal dynamics

The amount of information that can be stored in working memory is limited but may be improved with practice. The basis of improved efficiency at the level of neural activity is unknown. To investigate this question, we trained monkeys to perform a working memory task that required memory for multiple stimuli. Performance decreased as a function of number of stimuli to be remembered, but improved as the animals practiced the task. Neuronal recordings acquired during this training revealed two hitherto unknown mechanisms of working memory capacity improvement. First, more prefrontal neurons became active as working memory improved, but their baseline activity decreased. Second, improved working memory capacity was characterized by less variable temporal dynamics, resulting in a more consistent firing rate at each time point during the course of a trial. Our results reveal that improved performance of working memory tasks is achieved through more distributed activation and invariant neuronal dynamics.

Temporal signals underlying a cognitive process in the dorsal premotor cortex

During discrimination between two sequential vibrotactile stimulus patterns, the primate dorsal premotor cortex (DPC) neurons exhibit a complex repertoire of coding dynamics associated with the working memory, comparison, and decision components of this task. In addition, these neurons and neurons with no coding responses show complex strong fluctuations in their firing rate associated with the temporal sequence of task events. Here, to make sense of this temporal complexity, we extracted the temporal signals that were latent in the population. We found a strong link between the individual and population response, suggesting a common neural substrate. Notably, in contrast to coding dynamics, these time-dependent responses were unaffected during error trials. However, in a nondemanding task in which monkeys did not require discrimination for reward, these time-dependent signals were largely reduced and changed. These results suggest that temporal dynamics in DPC reflect the underlying cognitive processes of this task.

Persistent hippocampal neural firing and hippocampal-cortical coupling predict verbal working memory load

The maintenance of items in working memory relies on persistent neural activity in a widespread network of brain areas. To investigate the influence of load on working memory, we asked human subjects to maintain sets of letters in memory while we recorded single neurons and intracranial encephalography (EEG) in the medial temporal lobe and scalp EEG. Along the periods of a trial, hippocampal neural firing differentiated between success and error trials during stimulus encoding, predicted workload during memory maintenance, and predicted the subjects' behavior during retrieval. During maintenance, neuronal firing was synchronized with intracranial hippocampal EEG. On the network level, synchronization between hippocampal and scalp EEG in the theta-alpha frequency range showed workload dependent oscillatory coupling between hippocampus and cortex. Thus, we found that persistent neural activity in the hippocampus participated in working memory processing that is specific to memory maintenance, load sensitive and synchronized to the cortex.

Analyzing biological and artificial neural networks: challenges with opportunities for synergy?

Deep neural networks (DNNs) transform stimuli across multiple processing stages to produce representations that can be used to solve complex tasks, such as object recognition in images. However, a full understanding of how they achieve this remains elusive. The complexity of biological neural networks substantially exceeds the complexity of DNNs, making it even more challenging to understand the representations they learn. Thus, both machine learning and computational neuroscience are faced with a shared challenge: how can we analyze their representations in order to understand how they solve complex tasks? We review how data-analysis concepts and techniques developed by computational neuroscientists can be useful for analyzing representations in DNNs, and in turn, how recently developed techniques for analysis of DNNs can be useful for understanding representations in biological neural networks. We explore opportunities for synergy between the two fields, such as the use of DNNs as in silico model systems for neuroscience, and how this synergy can lead to new hypotheses about the operating principles of biological neural networks.

Cortical Areas Interact through a Communication Subspace

Most brain functions involve interactions among multiple, distinct areas or nuclei. For instance, visual processing in primates requires the appropriate relaying of signals across many distinct cortical areas. Yet our understanding of how populations of neurons in interconnected brain areas communicate is in its infancy. Here we investigate how trial-to-trial fluctuations of population responses in primary visual cortex (V1) are related to simultaneously recorded population responses in area V2. Using dimensionality reduction methods, we find that V1-V2 interactions occur through a communication subspace: V2 fluctuations are related to a small subset of V1 population activity patterns, distinct from the largest fluctuations shared among neurons within V1. In contrast, interactions between subpopulations within V1 are less selective. We propose that the communication subspace may be a general, population-level mechanism by which activity can be selectively routed across brain areas.

Reduced neural representation of arm/hand actions in the medial posterior parietal cortex

Several investigations at a single-cell level demonstrated that the medial posterior parietal area V6A is involved in encoding reaching and grasping actions in different visual conditions. Here, we looked for a “low-dimensional” representation of these encoding processes by studying macaque V6A neurons tested in three different tasks with a dimensionality reduction technique, the demixed principal component analysis (dPCA), which is very suitable for neuroprosthetics readout. We compared neural activity in reaching and grasping tasks by highlighting the portions of population variance involved in the encoding of visual information, target position, wrist orientation and grip type. The weight of visual information and task parameters in the encoding process was dependent on the task. We found that the distribution of variance captured by visual information in the three tasks did not differ significantly among the tasks, whereas the variance captured by target position and grip type parameters were significantly higher with respect to that captured by wrist orientation regardless of the number of conditions considered in each task. These results suggest a different use of relevant information according to the type of planned and executed action. This study shows a simplified picture of encoding that describes how V6A processes relevant information for action planning and execution.

Bridging large-scale neuronal recordings and large-scale network models using dimensionality reduction

A long-standing goal in neuroscience has been to bring together neuronal recordings and neural network modeling to understand brain function. Neuronal recordings can inform the development of network models, and network models can in turn provide predictions for subsequent experiments. Traditionally, neuronal recordings and network models have been related using single-neuron and pairwise spike train statistics. We review here recent studies that have begun to relate neuronal recordings and network models based on the multi-dimensional structure of neuronal population activity, as identified using dimensionality reduction. This approach has been used to study working memory, decision making, motor control, and more. Dimensionality reduction has provided common ground for incisive comparisons and tight interplay between neuronal recordings and network models.

Understanding Sensory Information Processing Through Simultaneous Multi-area Population Recordings

The goal of sensory neuroscience is to understand how the brain creates its myriad of representations of the world, and uses these representations to produce perception and behavior. Circuits of neurons in spatially segregated regions of brain tissue have distinct functional specializations, and these regions are connected to form a functional processing hierarchy. Advances in technology for recording neuronal activity from multiple sites in multiple cortical areas mean that we are now able to collect data that reflects how information is transformed within and between connected members of this hierarchy. This advance is an important step in understanding the brain because, after the sensory organs have transduced a physical signal, every processing stage takes the activity of other neurons as its input, not measurements of the physical world. However, as we explore the potential of studying how populations of neurons in multiple areas respond in concert, we must also expand both the analytical tools that we use to make sense of these data and the scope of the theories that we attempt to define. In this article, we present an overview of some of the most promising analytical approaches for making inferences from population recordings in multiple brain areas, such as dimensionality reduction and measuring changes in correlated variability, and examine how they may be used to address longstanding questions in sensory neuroscience.

Activity of Prefrontal Neurons Predict Future Choices during Gambling

Neuronal signals in the prefrontal cortex have been reported to predict upcoming decisions. Such activity patterns are often coupled to perceptual cues indicating correct choices or values of different options. How does the prefrontal cortex signal future decisions when no cues are present but when decisions are made based on internal valuations of past experiences with stochastic outcomes? We trained rats to perform a two-arm bandit-task, successfully adjusting choices between certain-small or possible-big rewards with changing long-term advantages. We discovered specialized prefrontal neurons, whose firing during the encounter of no-reward predicted the subsequent choice of animals, even for unlikely or uncertain decisions and several seconds before choice execution. Optogenetic silencing of the prelimbic cortex exclusively timed to encounters of no reward, provoked animals to excessive gambling for large rewards. Firing of prefrontal neurons during outcome evaluation signals subsequent choices during gambling and is essential for dynamically adjusting decisions based on internal valuations.

Rethinking assumptions about how trial and nuisance variability impact neural task performance in a fast-processing regime

Task performance is determined not only by the amount of task-relevant signal present in our brains but also by the presence of noise, which can arise from multiple sources. Internal noise, or "trial variability," manifests as trial-by-trial variations in neural responses under seemingly identical conditions. External factors can also translate into noise, particularly when a task requires extraction of a particular type of information from our environment amid changes in other task-irrelevant "nuisance" parameters. To better understand how signal, trial variability, and nuisance variability combine to determine neural task performance, we explored their interactions, both in simulation and when applied to recorded neural data. This exploration revealed that trial variability is typically larger than a neuron's task-relevant signal for tasks with fast reaction times, where spike count integration windows are short. In this low signal-to-trial variability regime, nuisance variability has the counterintuitive property of having a negligible impact on single-neuron task performance, even when it dominates the task-relevant signal. The inconsequential impact of nuisance variability on individual neurons also extends to descriptions of population performance, under the assumption that both trial and nuisance variability are uncorrelated between neurons. These results demonstrate that some basic intuitions about neural coding are misguided in the context of a fast-processing, low-spike-count regime. NEW & NOTEWORTHY Many everyday tasks require us to extract specific information from our environment while ignoring other things. When the neurons in our brains that carry task-relevant signals are also modulated by task-irrelevant "nuisance" information, nuisance modulation is expected to act as performance-limiting noise. Using both simulated and recorded neural data, we demonstrate that these intuitions are misguided when the brain operates in a fast-processing, low-spike-count regime, where nuisance variability is largely inconsequential for performance.

Temporal Metacognition as the Decoding of Self-Generated Brain Dynamics

Metacognition, the ability to know about one’s thought process, is self-referential. Here, we combined psychophysics and time-resolved neuroimaging to explore metacognitive inference on the accuracy of a self-generated behavior. Human participants generated a time interval and evaluated the signed magnitude of their temporal production. We show that both self-generation and self-evaluation relied on the power of beta oscillations (β; 15–40 Hz) with increases in early β power predictive of increases in duration. We characterized the dynamics of β power in a low-dimensional space (β state-space trajectories) as a function of timing and found that the more distinct trajectories, the more accurate metacognitive inferences were. These results suggest that β states instantiate an internal variable determining the fate of the timing network’s trajectory, possibly as release from inhibition. Altogether, our study describes oscillatory mechanisms for timing, suggesting that temporal metacognition relies on inferential processes of self-generated dynamics.

Neural coding of intended and executed grasp force in macaque areas AIP, F5, and M1

Considerable progress has been made over the last decades in characterizing the neural coding of hand shape, but grasp force has been largely ignored. We trained two macaque monkeys (Macaca mulatta) on a delayed grasping task where grip type and grip force were instructed. Neural population activity was recorded from areas relevant for grasp planning and execution: the anterior intraparietal area (AIP), F5 of the ventral premotor cortex, and the hand area of the primary motor cortex (M1). Grasp force was strongly encoded by neural populations of all three areas, thereby demonstrating for the first time the coding of grasp force in single- and multi-units of AIP. Neural coding of intended grasp force was most strongly represented in area F5. In addition to tuning analysis, a dimensionality reduction method revealed low-dimensional responses to grip type and grip force. Additionally, this method revealed a high correlation between latent variables of the neural population representing grasp force and the corresponding latent variables of electromyographic forearm muscle activity. Our results therefore suggest an important role of the cortical areas AIP, F5, and M1 in coding grasp force during movement execution as well as of F5 for coding intended grasp force.

Closed-loop cortical control of virtual reach and posture using Cartesian and joint velocity commands

OBJECTIVE

Brain-computer interfaces (BCIs) are a promising technology for the restoration of function to people with paralysis, especially for controlling coordinated reaching. Typical BCI studies decode Cartesian endpoint velocities as commands, but human arm movements might be better controlled in a joint-based coordinate frame, which may match underlying movement encoding in the motor cortex. A better understanding of BCI controlled reaching by people with paralysis may lead to performance improvements in brain-controlled assistive devices.

APPROACH

Two intracortical BCI participants in the BrainGate2 pilot clinical trial performed a visual 3D endpoint virtual reality reaching task using two decoders: Cartesian and joint velocity. Task performance metrics (i.e. success rate and path efficiency) and single feature and population tuning were compared across the two decoder conditions. The participants also demonstrated the first BCI control of a fourth dimension of reaching, the arm's swivel angle, in a 4D posture matching task.

MAIN RESULTS

Both users achieved significantly higher success rates using Cartesian velocity control, and joint controlled trajectories were more variable and significantly more curved. Neural tuning analyses showed that most single feature activity was best described by a Cartesian kinematic encoding model, and population analyses revealed only slight differences in aggregate activity between the decoder conditions. Simulations of a BCI user reproduced trajectory features seen during closed-loop joint control when assuming only Cartesian-tuned features passed through a joint decoder. With minimal training, both participants controlled the virtual arm's swivel angle to complete a 4D posture matching task, and achieved significantly higher success using a Cartesian  +  swivel velocity decoder compared to a joint velocity decoder.

SIGNIFICANCE

These results suggest that Cartesian velocity command interfaces may provide better BCI control of arm movements than other kinematic variables, even in 4D posture tasks with swivel angle targets.

Model-based targeted dimensionality reduction for neuronal population data

Summarizing high-dimensional data using a small number of parameters is a ubiquitous first step in the analysis of neuronal population activity. Recently developed methods use "targeted" approaches that work by identifying multiple, distinct low-dimensional subspaces of activity that capture the population response to individual experimental task variables, such as the value of a presented stimulus or the behavior of the animal. These methods have gained attention because they decompose total neural activity into what are ostensibly different parts of a neuronal computation. However, existing targeted methods have been developed outside of the confines of probabilistic modeling, making some aspects of the procedures ad hoc, or limited in flexibility or interpretability. Here we propose a new model-based method for targeted dimensionality reduction based on a probabilistic generative model of the population response data. The low-dimensional structure of our model is expressed as a low-rank factorization of a linear regression model. We perform efficient inference using a combination of expectation maximization and direct maximization of the marginal likelihood. We also develop an efficient method for estimating the dimensionality of each subspace. We show that our approach outperforms alternative methods in both mean squared error of the parameter estimates, and in identifying the correct dimensionality of encoding using simulated data. We also show that our method provides more accurate inference of low-dimensional subspaces of activity than a competing algorithm, demixed PCA.

Cortical population activity within a preserved neural manifold underlies multiple motor behaviors

Populations of cortical neurons flexibly perform different functions; for the primary motor cortex (M1) this means a rich repertoire of motor behaviors. We investigate the flexibility of M1 movement control by analyzing neural population activity during a variety of skilled wrist and reach-to-grasp tasks. We compare across tasks the neural modes that capture dominant neural covariance patterns during each task. While each task requires different patterns of muscle and single unit activity, we find unexpected similarities at the neural population level: the structure and activity of the neural modes is largely preserved across tasks. Furthermore, we find two sets of neural modes with task-independent activity that capture, respectively, generic temporal features of the set of tasks and a task-independent mapping onto muscle activity. This system of flexibly combined, well-preserved neural modes may underlie the ability of M1 to learn and generate a wide-ranging behavioral repertoire.

Inferring single-trial neural population dynamics using sequential auto-encoders

Neuroscience is experiencing a revolution in which simultaneous recording of many thousands of neurons is revealing population dynamics that are not apparent from single-neuron responses. This structure is typically extracted from trial-averaged data, but deeper understanding requires studying single-trial phenomena, which is challenging due to incomplete sampling of the neural population, trial-to-trial variability, and fluctuations in action potential timing. We introduce Latent Factor Analysis via Dynamical Systems (LFADS), a deep learning method to infer latent dynamics from single-trial neural spiking data. LFADS uses a nonlinear dynamical system to infer the dynamics underlying observed spiking activity and to extract ‘de-noised’ single-trial firing rates. When applied to a variety of monkey and human motor cortical datasets, LFADS predicts observed behavioral variables with unprecedented accuracy, extracts precise estimates of neural dynamics on single trials, infers perturbations to those dynamics that correlate with behavioral choices, and combines data from non-overlapping recording sessions spanning months to improve inference of underlying dynamics.

Sensorimotor integration in Caenorhabditis elegans: a reappraisal towards dynamic and distributed computations

The nematode Caenorhabditis elegans is a tractable model system to study locomotion, sensory navigation and decision-making. In its natural habitat, it is thought to navigate complex multisensory environments in order to find food and mating partners, while avoiding threats like predators or toxic environments. While research in past decades has shed much light on the functions and mechanisms of selected sensory neurons, we are just at the brink of understanding how sensory information is integrated by interneuron circuits for action selection in the worm. Recent technological advances have enabled whole-brain Ca2+ imaging and Ca2+ imaging of neuronal activity in freely moving worms. A common principle emerging across multiple studies is that most interneuron activities are tightly coupled to the worm's instantaneous behaviour; notably, these observations encompass neurons receiving direct sensory neuron inputs. The new findings suggest that in the C. elegans brain, sensory and motor representations are integrated already at the uppermost sensory processing layers. Moreover, these results challenge a perhaps more intuitive view of sequential feed-forward sensory pathways that converge onto premotor interneurons and motor neurons. We propose that sensorimotor integration occurs rather in a distributed dynamical fashion. In this perspective article, we will explore this view, discuss the challenges and implications of these discoveries on the interpretation and design of neural activity experiments, and discuss possible functions. Furthermore, we will discuss the broader context of similar findings in fruit flies and rodents, which suggest generalizable principles that can be learnt from this amenable nematode model organism.This article is part of a discussion meeting issue 'Connectome to behaviour: modelling C. elegans at cellular resolution'.

Inferotemporal cortex multiplexes behaviorally-relevant target match signals and visual representations in a manner that minimizes their interference

Finding a sought visual target object requires combining visual information about a scene with a remembered representation of the target to create a "target match" signal that indicates when a target is in view. Target match signals have been reported to exist within high-level visual brain areas including inferotemporal cortex (IT), where they are mixed with representations of image and object identity. However, these signals are not well understood, particularly in the context of the real-world challenge that the objects we search for typically appear at different positions, sizes, and within different background contexts. To investigate these signals, we recorded neural responses in IT as two rhesus monkeys performed a delayed-match-to-sample object search task in which target objects could appear at a variety of identity-preserving transformations. Consistent with the existence of behaviorally-relevant target match signals in IT, we found that IT contained a linearly separable target match representation that reflected behavioral confusions on trials in which the monkeys made errors. Additionally, target match signals were highly distributed across the IT population, and while a small fraction of units reflected target match signals as target match suppression, most units reflected target match signals as target match enhancement. Finally, we found that the potentially detrimental impact of target match signals on visual representations was mitigated by target match modulation that was approximately (albeit imperfectly) multiplicative. Together, these results support the existence of a robust, behaviorally-relevant target match representation in IT that is configured to minimally interfere with IT visual representations.

Different population dynamics in the supplementary motor area and motor cortex during reaching

Neural populations perform computations through their collective activity. Different computations likely require different population-level dynamics. We leverage this assumption to examine neural responses recorded from the supplementary motor area (SMA) and motor cortex. During visually guided reaching, the respective roles of these areas remain unclear; neurons in both areas exhibit preparation-related activity and complex patterns of movement-related activity. To explore population dynamics, we employ a novel "hypothesis-guided" dimensionality reduction approach. This approach reveals commonalities but also stark differences: linear population dynamics, dominated by rotations, are prominent in motor cortex but largely absent in SMA. In motor cortex, the observed dynamics produce patterns resembling muscle activity. Conversely, the non-rotational patterns in SMA co-vary with cues regarding when movement should be initiated. Thus, while SMA and motor cortex display superficially similar single-neuron responses during visually guided reaching, their different population dynamics indicate they are likely performing quite different computations.

Attractor Dynamics in Networks with Learning Rules Inferred from In Vivo Data

The attractor neural network scenario is a popular scenario for memory storage in the association cortex, but there is still a large gap between models based on this scenario and experimental data. We study a recurrent network model in which both learning rules and distribution of stored patterns are inferred from distributions of visual responses for novel and familiar images in the inferior temporal cortex (ITC). Unlike classical attractor neural network models, our model exhibits graded activity in retrieval states, with distributions of firing rates that are close to lognormal. Inferred learning rules are close to maximizing the number of stored patterns within a family of unsupervised Hebbian learning rules, suggesting that learning rules in ITC are optimized to store a large number of attractor states. Finally, we show that there exist two types of retrieval states: one in which firing rates are constant in time and another in which firing rates fluctuate chaotically.

Unsupervised Discovery of Demixed, Low-Dimensional Neural Dynamics across Multiple Timescales through Tensor Component Analysis

Perceptions, thoughts, and actions unfold over millisecond timescales, while learned behaviors can require many days to mature. While recent experimental advances enable large-scale and long-term neural recordings with high temporal fidelity, it remains a formidable challenge to extract unbiased and interpretable descriptions of how rapid single-trial circuit dynamics change slowly over many trials to mediate learning. We demonstrate a simple tensor component analysis (TCA) can meet this challenge by extracting three interconnected, low-dimensional descriptions of neural data: neuron factors, reflecting cell assemblies; temporal factors, reflecting rapid circuit dynamics mediating perceptions, thoughts, and actions within each trial; and trial factors, describing both long-term learning and trial-to-trial changes in cognitive state. We demonstrate the broad applicability of TCA by revealing insights into diverse datasets derived from artificial neural networks, large-scale calcium imaging of rodent prefrontal cortex during maze navigation, and multielectrode recordings of macaque motor cortex during brain machine interface learning.

A Dynamic Connectome Supports the Emergence of Stable Computational Function of Neural Circuits through Reward-Based Learning

Synaptic connections between neurons in the brain are dynamic because of continuously ongoing spine dynamics, axonal sprouting, and other processes. In fact, it was recently shown that the spontaneous synapse-autonomous component of spine dynamics is at least as large as the component that depends on the history of pre- and postsynaptic neural activity. These data are inconsistent with common models for network plasticity and raise the following questions: how can neural circuits maintain a stable computational function in spite of these continuously ongoing processes, and what could be functional uses of these ongoing processes? Here, we present a rigorous theoretical framework for these seemingly stochastic spine dynamics and rewiring processes in the context of reward-based learning tasks. We show that spontaneous synapse-autonomous processes, in combination with reward signals such as dopamine, can explain the capability of networks of neurons in the brain to configure themselves for specific computational tasks, and to compensate automatically for later changes in the network or task. Furthermore, we show theoretically and through computer simulations that stable computational performance is compatible with continuously ongoing synapse-autonomous changes. After reaching good computational performance it causes primarily a slow drift of network architecture and dynamics in task-irrelevant dimensions, as observed for neural activity in motor cortex and other areas. On the more abstract level of reinforcement learning the resulting model gives rise to an understanding of reward-driven network plasticity as continuous sampling of network configurations.

Mixed selectivity encoding and action selection in the prefrontal cortex during threat assessment

The medial prefrontal cortex (mPFC) regulates expression of emotional behavior. The mPFC combines multivariate information from its inputs, and depending on the imminence of threat, activates downstream networks that either increase or decrease the expression of anxiety-related motor behavior and autonomic activation. Here, we selectively highlight how subcortical input to the mPFC from two example structures, the amygdala and ventral hippocampus, help shape mixed selectivity encoding and action selection during emotional processing. We outline a model where prefrontal subregions modulate behavior along orthogonal motor dimensions, and exhibit connectivity that selects for expression of one behavioral strategy while inhibiting the other.

Population coding of grasp and laterality-related information in the macaque fronto-parietal network

Preparing and executing grasping movements demands the coordination of sensory information across multiple scales. The position of an object, required hand shape, and which of our hands to extend must all be coordinated in parallel. The network formed by the macaque anterior intraparietal area (AIP) and hand area (F5) of the ventral premotor cortex is essential in the generation of grasping movements. Yet, the role of this circuit in hand selection is unclear. We recorded from 1342 single- and multi-units in AIP and F5 of two macaque monkeys (Macaca mulatta) during a delayed grasping task in which monkeys were instructed by a visual cue to perform power or precision grips on a handle presented in five different orientations with either the left or right hand, as instructed by an auditory tone. In AIP, intended hand use (left vs. right) was only weakly represented during preparation, while hand use was robustly present in F5 during preparation. Interestingly, visual-centric handle orientation information dominated AIP, while F5 contained an additional body-centric frame during preparation and movement. Together, our results implicate F5 as a site of visuo-motor transformation and advocate a strong transition between hand-independent and hand-dependent representations in this parieto-frontal circuit.

Restoration of Hindlimb Movements after Complete Spinal Cord Injury Using Brain-Controlled Functional Electrical Stimulation

Single neuron and local field potential signals recorded in the primary motor cortex have been repeatedly demonstrated as viable control signals for multi-degree-of-freedom actuators. Although the primary source of these signals has been fore/upper limb motor regions, recent evidence suggests that neural adaptation underlying neuroprosthetic control is generalizable across cortex, including hindlimb sensorimotor cortex. Here, adult rats underwent a longitudinal study that included a hindlimb pedal press task in response to cues for specific durations, followed by brain machine interface (BMI) tasks in healthy rats, after rats received a complete spinal transection and after the BMI signal controls epidural stimulation (BMI-FES). Over the course of the transition from learned behavior to BMI task, fewer neurons were responsive after the cue, the proportion of neurons selective for press duration increased and these neurons carried more information. After a complete, mid-thoracic spinal lesion that completely severed both ascending and descending connections to the lower limbs, there was a reduction in task-responsive neurons followed by a reacquisition of task selectivity in recorded populations. This occurred due to a change in pattern of neuronal responses not simple changes in firing rate. Finally, during BMI-FES, additional information about the intended press duration was produced. This information was not dependent on the stimulation, which was the same for short and long duration presses during the early phase of stimulation, but instead was likely due to sensory feedback to sensorimotor cortex in response to movement along the trunk during the restored pedal press. This post-cue signal could be used as an error signal in a continuous decoder providing information about the position of the limb to optimally control a neuroprosthetic device.

Decoding a Decision Process in the Neuronal Population of Dorsal Premotor Cortex

When trained monkeys discriminate the temporal structure of two sequential vibrotactile stimuli, dorsal premotor cortex (DPC) showed high heterogeneity among its neuronal responses. Notably, DPC neurons coded stimulus patterns as broader categories and signaled them during working memory, comparison, and postponed decision periods. Here, we show that such population activity can be condensed into two major coding components: one that persistently represented in working memory both the first stimulus identity and the postponed informed choice and another that transiently coded the initial sensory information and the result of the comparison between the two stimuli. Additionally, we identified relevant signals that coded the timing of task events. These temporal and task-parameter readouts were shown to be strongly linked to the monkeys' behavior when contrasted to those obtained in a non-demanding cognitive control task and during error trials. These signals, hidden in the heterogeneity, were prominently represented by the DPC population response.

Dynamical networks: Finding, measuring, and tracking neural population activity using network science

Systems neuroscience is in a headlong rush to record from as many neurons at the same time as possible. As the brain computes and codes using neuron populations, it is hoped these data will uncover the fundamentals of neural computation. But with hundreds, thousands, or more simultaneously recorded neurons come the inescapable problems of visualizing, describing, and quantifying their interactions. Here I argue that network science provides a set of scalable, analytical tools that already solve these problems. By treating neurons as nodes and their interactions as links, a single network can visualize and describe an arbitrarily large recording. I show that with this description we can quantify the effects of manipulating a neural circuit, track changes in population dynamics over time, and quantitatively define theoretical concepts of neural populations such as cell assemblies. Using network science as a core part of analyzing population recordings will thus provide both qualitative and quantitative advances to our understanding of neural computation.

Characterizing and interpreting the influence of internal variables on sensory activity

The concept of a tuning curve has been central for our understanding of how the responses of cortical neurons depend on external stimuli. Here, we describe how the influence of unobserved internal variables on sensory responses, in particular correlated neural variability, can be understood in a similar framework. We suggest that this will lead to deeper insights into the relationship between stimulus, sensory responses, and behavior. We review related recent work and discuss its implication for distinguishing feedforward from feedback influences on sensory responses, and for the information contained in those responses.

What can neuronal populations tell us about cognition?

Nowadays, it is possible to record the activity of hundreds of cells at the same time in behaving animals. However, these data are often treated and analyzed as if they consisted of many independently recorded neurons. How can neuronal populations be uniquely used to learn about cognition? We describe recent work that shows that populations of simultaneously recorded neurons are fundamental to understand the basis of decision-making, including processes such as ongoing deliberations and decision confidence, which generally fall outside the reach of single-cell analysis. Thus, neuronal population data allow addressing novel questions, but they also come with so far unsolved challenges.

Object comparison in the lateral intraparietal area

We can search for and locate specific objects in our environment by looking for objects with similar features. Object recognition involves stimulus similarity responses in ventral visual areas and task-related responses in prefrontal cortex. We tested whether neurons in the lateral intraparietal area (LIP) of posterior parietal cortex could form an intermediary representation, collating information from object-specific similarity map representations to allow general decisions about whether a stimulus matches the object being looked for. We hypothesized that responses to stimuli would correlate with how similar they are to a sample stimulus. When animals compared two peripheral stimuli to a sample at their fovea, the response to the matching stimulus was similar, independent of the sample identity, but the response to the nonmatch depended on how similar it was to the sample: the more similar, the greater the response to the nonmatch stimulus. These results could not be explained by task difficulty or confidence. We propose that LIP uses its known mechanistic properties to integrate incoming visual information, including that from the ventral stream about object identity, to create a dynamic representation that is concise, low dimensional, and task relevant and that signifies the choice priorities in mental matching behavior.NEW & NOTEWORTHY Studies in object recognition have focused on the ventral stream, in which neurons respond as a function of how similar a stimulus is to their preferred stimulus, and on prefrontal cortex, where neurons indicate which stimulus is being looked for. We found that parietal area LIP uses its known mechanistic properties to form an intermediary representation in this process. This creates a perceptual similarity map that can be used to guide decisions in prefrontal areas.

Structure in neural population recordings: an expected byproduct of simpler phenomena?

Neuroscientists increasingly analyze the joint activity of multi-neuron recordings to identify population-level structure that is believed to be significant and scientifically novel. Claims of significant population structure support hypotheses in many brain areas. However, these claims require first investigating the possibility that the population structure in question is an expected byproduct of simpler features known to exist in data. Classically, this critical examination can be either intuited or addressed with conventional controls. However, these approaches fail when considering population data, raising concerns about the scientific merit of population-level studies. Here we develop a framework to test the novelty of population-level findings against simpler features such as correlations across times, neurons and conditions. We apply this framework to test two recent population findings in prefrontal and motor cortices, providing essential context to those studies. More broadly, the methodologies we introduce provide a general neural population control for many population-level hypotheses.

A spiral attractor network drives rhythmic locomotion

The joint activity of neural populations is high dimensional and complex. One strategy for reaching a tractable understanding of circuit function is to seek the simplest dynamical system that can account for the population activity. By imaging Aplysia's pedal ganglion during fictive locomotion, here we show that its population-wide activity arises from a low-dimensional spiral attractor. Evoking locomotion moved the population into a low-dimensional, periodic, decaying orbit - a spiral - in which it behaved as a true attractor, converging to the same orbit when evoked, and returning to that orbit after transient perturbation. We found the same attractor in every preparation, and could predict motor output directly from its orbit, yet individual neurons' participation changed across consecutive locomotion bouts. From these results, we propose that only the low-dimensional dynamics for movement control, and not the high-dimensional population activity, are consistent within and between nervous systems.

Neural Manifolds for the Control of Movement

The analysis of neural dynamics in several brain cortices has consistently uncovered low-dimensional manifolds that capture a significant fraction of neural variability. These neural manifolds are spanned by specific patterns of correlated neural activity, the "neural modes." We discuss a model for neural control of movement in which the time-dependent activation of these neural modes is the generator of motor behavior. This manifold-based view of motor cortex may lead to a better understanding of how the brain controls movement.

Task-dependent recurrent dynamics in visual cortex

The capacity for flexible sensory-action association in animals has been related to context-dependent attractor dynamics outside the sensory cortices. Here, we report a line of evidence that flexibly modulated attractor dynamics during task switching are already present in the higher visual cortex in macaque monkeys. With a nonlinear decoding approach, we can extract the particular aspect of the neural population response that reflects the task-induced emergence of bistable attractor dynamics in a neural population, which could be obscured by standard unsupervised dimensionality reductions such as PCA. The dynamical modulation selectively increases the information relevant to task demands, indicating that such modulation is beneficial for perceptual decisions. A computational model that features nonlinear recurrent interaction among neurons with a task-dependent background input replicates the key properties observed in the experimental data. These results suggest that the context-dependent attractor dynamics involving the sensory cortex can underlie flexible perceptual abilities.

Global Representations of Goal-Directed Behavior in Distinct Cell Types of Mouse Neocortex

The successful planning and execution of adaptive behaviors in mammals may require long-range coordination of neural networks throughout cerebral cortex. The neuronal implementation of signals that could orchestrate cortex-wide activity remains unclear. Here, we develop and apply methods for cortex-wide Ca²⁺ imaging in mice performing decision-making behavior and identify a global cortical representation of task engagement encoded in the activity dynamics of both single cells and superficial neuropil distributed across the majority of dorsal cortex. The activity of multiple molecularly defined cell types was found to reflect this representation with type-specific dynamics. Focal optogenetic inhibition tiled across cortex revealed a crucial role for frontal cortex in triggering this cortex-wide phenomenon; local inhibition of this region blocked both the cortex-wide response to task-initiating cues and the voluntary behavior. These findings reveal cell-type-specific processes in cortex for globally representing goal-directed behavior and identify a major cortical node that gates the global broadcast of task-related information.

Distinct Sources of Deterministic and Stochastic Components of Action Timing Decisions in Rodent Frontal Cortex

The selection and timing of actions are subject to determinate influences such as sensory cues and internal state as well as to effectively stochastic variability. Although stochastic choice mechanisms are assumed by many theoretical models, their origin and mechanisms remain poorly understood. Here we investigated this issue by studying how neural circuits in the frontal cortex determine action timing in rats performing a waiting task. Electrophysiological recordings from two regions necessary for this behavior, medial prefrontal cortex (mPFC) and secondary motor cortex (M2), revealed an unexpected functional dissociation. Both areas encoded deterministic biases in action timing, but only M2 neurons reflected stochastic trial-by-trial fluctuations. This differential coding was reflected in distinct timescales of neural dynamics in the two frontal cortical areas. These results suggest a two-stage model in which stochastic components of action timing decisions are injected by circuits downstream of those carrying deterministic bias signals.