Single-unit activity, threshold crossings, and local field potentials in motor cortex differentially encode reach kinematics

Event detection and classification from multimodal time series with application to neural data

The detection of events in time-series data is a common signal-processing problem. When the data can be modeled as a known template signal with an unknown delay in Gaussian noise, detection of the template signal can be done with a traditional matched filter. However, in many applications, the event of interest is represented in multimodal data consisting of both Gaussian and point-process time series. Neuroscience experiments, for example, can simultaneously record multimodal neural signals such as local field potentials (LFPs), which can be modeled as Gaussian, and neuronal spikes, which can be modeled as point processes. Currently, no method exists for event detection from such multimodal data, and as such our objective in this work is to develop a method to meet this need. Here we address this challenge by developing the multimodal event detector (MED) algorithm which simultaneously estimates event times and classes. To do this, we write a multimodal likelihood function for Gaussian and point-process observations and derive the associated maximum likelihood estimator of simultaneous event times and classes. We additionally introduce a cross-modal scaling parameter to account for model mismatch in real datasets. We validate this method in extensive simulations as well as in a neural spike-LFP dataset recorded during an eye-movement task, where the events of interest are eye movements with unknown times and directions. We show that the MED can successfully detect eye movement onset and classify eye movement direction. Further, the MED successfully combines information across data modalities, with multimodal performance exceeding unimodal performance. This method can facilitate applications such as the discovery of latent events in multimodal neural population activity and the development of brain-computer interfaces for naturalistic settings without constrained tasks or prior knowledge of event times.

Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity

Objective.Learning dynamical latent state models for multimodal spiking and field potential activity can reveal their collective low-dimensional dynamics and enable better decoding of behavior through multimodal fusion. Toward this goal, developing unsupervised learning methods that are computationally efficient is important, especially for real-time learning applications such as brain-machine interfaces (BMIs). However, efficient learning remains elusive for multimodal spike-field data due to their heterogeneous discrete-continuous distributions and different timescales.Approach.Here, we develop a multiscale subspace identification (multiscale SID) algorithm that enables computationally efficient learning for modeling and dimensionality reduction for multimodal discrete-continuous spike-field data. We describe the spike-field activity as combined Poisson and Gaussian observations, for which we derive a new analytical SID method. Importantly, we also introduce a novel constrained optimization approach to learn valid noise statistics, which is critical for multimodal statistical inference of the latent state, neural activity, and behavior. We validate the method using numerical simulations and with spiking and local field potential population activity recorded during a naturalistic reach and grasp behavior.Main results.We find that multiscale SID accurately learned dynamical models of spike-field signals and extracted low-dimensional dynamics from these multimodal signals. Further, it fused multimodal information, thus better identifying the dynamical modes and predicting behavior compared to using a single modality. Finally, compared to existing multiscale expectation-maximization learning for Poisson-Gaussian observations, multiscale SID had a much lower training time while being better in identifying the dynamical modes and having a better or similar accuracy in predicting neural activity and behavior.Significance.Overall, multiscale SID is an accurate learning method that is particularly beneficial when efficient learning is of interest, such as for online adaptive BMIs to track non-stationary dynamics or for reducing offline training time in neuroscience investigations.

Machine learning-based high-frequency neuronal spike reconstruction from low-frequency and low-sampling-rate recordings

Recording neuronal activity using multiple electrodes has been widely used to understand the functional mechanisms of the brain. Increasing the number of electrodes allows us to decode more variety of functionalities. However, handling massive amounts of multichannel electrophysiological data is still challenging due to the limited hardware resources and unavoidable thermal tissue damage. Here, we present machine learning (ML)-based reconstruction of high-frequency neuronal spikes from subsampled low-frequency band signals. Inspired by the equivalence between high-frequency restoration and super-resolution in image processing, we applied a transformer ML model to neuronal data recorded from both in vitro cultures and in vivo male mouse brains. Even with the x8 downsampled datasets, our trained model reasonably estimated high-frequency information of spiking activity, including spike timing, waveform, and network connectivity. With our ML-based data reduction applicable to existing multichannel recording hardware while achieving neuronal signals of broad bandwidths, we expect to enable more comprehensive analysis and control of brain functions.

Dynamical flexible inference of nonlinear latent factors and structures in neural population activity

Modelling the spatiotemporal dynamics in the activity of neural populations while also enabling their flexible inference is hindered by the complexity and noisiness of neural observations. Here we show that the lower-dimensional nonlinear latent factors and latent structures can be computationally modelled in a manner that allows for flexible inference causally, non-causally and in the presence of missing neural observations. To enable flexible inference, we developed a neural network that separates the model into jointly trained manifold and dynamic latent factors such that nonlinearity is captured through the manifold factors and the dynamics can be modelled in tractable linear form on this nonlinear manifold. We show that the model, which we named 'DFINE' (for 'dynamical flexible inference for nonlinear embeddings') achieves flexible inference in simulations of nonlinear dynamics and across neural datasets representing a diversity of brain regions and behaviours. Compared with earlier neural-network models, DFINE enables flexible inference, better predicts neural activity and behaviour, and better captures the latent neural manifold structure. DFINE may advance the development of neurotechnology and investigations in neuroscience.

Spatio-temporal evolution of human neural activity during visually cued hand movements

Making hand movements in response to visual cues is common in daily life. It has been well known that this process activates multiple areas in the brain, but how these neural activations progress across space and time remains largely unknown. Taking advantage of intracranial electroencephalographic (iEEG) recordings using depth and subdural electrodes from 36 human subjects using the same task, we applied single-trial and cross-trial analyses to high-frequency iEEG activity. The results show that the neural activation was widely distributed across the human brain both within and on the surface of the brain, and focused specifically on certain areas in the parietal, frontal, and occipital lobes, where parietal lobes present significant left lateralization on the activation. We also demonstrate temporal differences across these brain regions. Finally, we evaluated the degree to which the timing of activity within these regions was related to sensory or motor function. The findings of this study promote the understanding of task-related neural processing of the human brain, and may provide important insights for translational applications.

Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity

Learning dynamical latent state models for multimodal spiking and field potential activity can reveal their collective low-dimensional dynamics and enable better decoding of behavior through multimodal fusion. Toward this goal, developing unsupervised learning methods that are computationally efficient is important, especially for real-time learning applications such as brain-machine interfaces (BMIs). However, efficient learning remains elusive for multimodal spike-field data due to their heterogeneous discrete-continuous distributions and different timescales. Here, we develop a multiscale subspace identification (multiscale SID) algorithm that enables computationally efficient modeling and dimensionality reduction for multimodal discrete-continuous spike-field data. We describe the spike-field activity as combined Poisson and Gaussian observations, for which we derive a new analytical subspace identification method. Importantly, we also introduce a novel constrained optimization approach to learn valid noise statistics, which is critical for multimodal statistical inference of the latent state, neural activity, and behavior. We validate the method using numerical simulations and spike-LFP population activity recorded during a naturalistic reach and grasp behavior. We find that multiscale SID accurately learned dynamical models of spike-field signals and extracted low-dimensional dynamics from these multimodal signals. Further, it fused multimodal information, thus better identifying the dynamical modes and predicting behavior compared to using a single modality. Finally, compared to existing multiscale expectation-maximization learning for Poisson-Gaussian observations, multiscale SID had a much lower computational cost while being better in identifying the dynamical modes and having a better or similar accuracy in predicting neural activity. Overall, multiscale SID is an accurate learning method that is particularly beneficial when efficient learning is of interest.

Dynamical flexible inference of nonlinear latent structures in neural population activity

Inferring complex spatiotemporal dynamics in neural population activity is critical for investigating neural mechanisms and developing neurotechnology. These activity patterns are noisy observations of lower-dimensional latent factors and their nonlinear dynamical structure. A major unaddressed challenge is to model this nonlinear structure, but in a manner that allows for flexible inference, whether causally, non-causally, or in the presence of missing neural observations. We address this challenge by developing DFINE, a new neural network that separates the model into dynamic and manifold latent factors, such that the dynamics can be modeled in tractable form. We show that DFINE achieves flexible nonlinear inference across diverse behaviors and brain regions. Further, despite enabling flexible inference unlike prior neural network models of population activity, DFINE also better predicts the behavior and neural activity, and better captures the latent neural manifold structure. DFINE can both enhance future neurotechnology and facilitate investigations across diverse domains of neuroscience.

Subdural CMOS optical probe (SCOPe) for bidirectional neural interfacing

Optical neurotechnologies use light to interface with neurons and can monitor and manipulate neural activity with high spatial-temporal precision over large cortical extents. While there has been significant progress in miniaturizing microscope for head-mounted configurations, these existing devices are still very bulky and could never be fully implanted. Any viable translation of these technologies to human use will require a much more noninvasive, fully implantable form factor. Here, we leverage advances in microelectronics and heterogeneous optoelectronic packaging to develop a transformative, ultrathin, miniaturized device for bidirectional optical stimulation and recording: the subdural CMOS Optical Probe (SCOPe). By being thin enough to lie entirely within the subdural space of the primate brain, SCOPe defines a path for the eventual human translation of a new generation of brain-machine interfaces based on light.

Modeling and inference methods for switching regime-dependent dynamical systems with multiscale neural observations

Objective.Realizing neurotechnologies that enable long-term neural recordings across multiple spatial-temporal scales during naturalistic behaviors requires new modeling and inference methods that can simultaneously address two challenges. First, the methods should aggregate information across all activity scales from multiple recording sources such as spiking and field potentials. Second, the methods should detect changes in the regimes of behavior and/or neural dynamics during naturalistic scenarios and long-term recordings. Prior regime detection methods are developed for a single scale of activity rather than multiscale activity, and prior multiscale methods have not considered regime switching and are for stationary cases.Approach.Here, we address both challenges by developing a switching multiscale dynamical system model and the associated filtering and smoothing methods. This model describes the encoding of an unobserved brain state in multiscale spike-field activity. It also allows for regime-switching dynamics using an unobserved regime state that dictates the dynamical and encoding parameters at every time-step. We also design the associated switching multiscale inference methods that estimate both the unobserved regime and brain states from simultaneous spike-field activity.Main results.We validate the methods in both extensive numerical simulations and prefrontal spike-field data recorded in a monkey performing saccades for fluid rewards. We show that these methods can successfully combine the spiking and field potential observations to simultaneously track the regime and brain states accurately. Doing so, these methods lead to better state estimation compared with single-scale switching methods or stationary multiscale methods. Also, for single-scale linear Gaussian observations, the new switching smoother can better generalize to diverse system settings compared to prior switching smoothers.Significance.These modeling and inference methods effectively incorporate both regime-detection and multiscale observations. As such, they could facilitate investigation of latent switching neural population dynamics and improve future brain-machine interfaces by enabling inference in naturalistic scenarios where regime-dependent multiscale activity and behavior arise.

Decoding the Time Course of Spatial Information from Spiking and Local Field Potential Activities in the Superior Colliculus

Place code representation is ubiquitous in circuits that encode spatial parameters. For visually guided eye movements, neurons in many brain regions emit spikes when a stimulus is presented in their receptive fields and/or when a movement is directed into their movement fields. Crucially, individual neurons respond for a broad range of directions or eccentricities away from the optimal vector, making it difficult to decode the stimulus location or the saccade vector from each cell's activity. We investigated whether it is possible to decode the spatial parameter with a population-level analysis, even when the optimal vectors are similar across neurons. Spiking activity and local field potentials (LFPs) in the superior colliculus (SC) were recorded with a laminar probe as monkeys performed a delayed saccade task to one of eight targets radially equidistant in direction. A classifier was applied offline to decode the spatial configuration as the trial progresses from sensation to action. For spiking activity, decoding performance across all eight directions was highest during the visual and motor epochs and lower but well above chance during the delay period. Classification performance followed a similar pattern for LFP activity too, except the performance during the delay period was limited mostly to the preferred direction. Increasing the number of neurons in the population consistently increased classifier performance for both modalities. Overall, this study demonstrates the power of population activity for decoding spatial information not possible from individual neurons.

Local field potentials reflect cortical population dynamics in a region-specific and frequency-dependent manner

The spiking activity of populations of cortical neurons is well described by the dynamics of a small number of population-wide covariance patterns, the 'latent dynamics'. These latent dynamics are largely driven by the same correlated synaptic currents across the circuit that determine the generation of local field potentials (LFP). Yet, the relationship between latent dynamics and LFPs remains largely unexplored. Here, we characterised this relationship for three different regions of primate sensorimotor cortex during reaching. The correlation between latent dynamics and LFPs was frequency-dependent and varied across regions. However, for any given region, this relationship remained stable throughout the behaviour: in each of primary motor and premotor cortices, the LFP-latent dynamics correlation profile was remarkably similar between movement planning and execution. These robust associations between LFPs and neural population latent dynamics help bridge the wealth of studies reporting neural correlates of behaviour using either type of recordings.

Modeling multiscale causal interactions between spiking and field potential signals during behavior

Objective. Brain recordings exhibit dynamics at multiple spatiotemporal scales, which are measured with spike trains and larger-scale field potential signals. To study neural processes, it is important to identify and model causal interactions not only at a single scale of activity, but also across multiple scales, i.e. between spike trains and field potential signals. Standard causality measures are not directly applicable here because spike trains are binary-valued but field potentials are continuous-valued. It is thus important to develop computational tools to recover multiscale neural causality during behavior, assess their performance on neural datasets, and study whether modeling multiscale causalities can improve the prediction of neural signals beyond what is possible with single-scale causality. Approach. We design a multiscale model-based Granger-like causality method based on directed information and evaluate its success both in realistic biophysical spike-field simulations and in motor cortical datasets from two non-human primates (NHP) performing a motor behavior. To compute multiscale causality, we learn point-process generalized linear models that predict the spike events at a given time based on the history of both spike trains and field potential signals. We also learn linear Gaussian models that predict the field potential signals at a given time based on their own history as well as either the history of binary spike events or that of latent firing rates. Main results. We find that our method reveals the true multiscale causality network structure in biophysical simulations despite the presence of model mismatch. Further, models with the identified multiscale causalities in the NHP neural datasets lead to better prediction of both spike trains and field potential signals compared to just modeling single-scale causalities. Finally, we find that latent firing rates are better predictors of field potential signals compared with the binary spike events in the NHP datasets. Significance. This multiscale causality method can reveal the directed functional interactions across spatiotemporal scales of brain activity to inform basic science investigations and neurotechnologies.

The significance of neural inter-frequency power correlations

It is of great interest in neuroscience to determine what frequency bands in the brain have covarying power. This would help us robustly identify the frequency signatures of neural processes. However to date, to the best of the author's knowledge, a comprehensive statistical approach to this question that accounts for intra-frequency autocorrelation, frequency-domain oversampling, and multiple testing under dependency has not been undertaken. As such, this work presents a novel statistical significance test for correlated power across frequency bands for a broad class of non-stationary time series. It is validated on synthetic data. It is then used to test all of the inter-frequency power correlations between 0.2 and 8500 Hz in continuous intracortical extracellular neural recordings in Macaque M1, using a very large, publicly available dataset. The recordings were Current Source Density referenced and were recorded with a Utah array. The results support previous results in the literature that show that neural processes in M1 have power signatures across a very broad range of frequency bands. In particular, the power in LFP frequency bands as low as 20 Hz was found to almost always be statistically significantly correlated to the power in kHz frequency ranges. It is proposed that this test can also be used to discover the superimposed frequency domain signatures of all the neural processes in a neural signal, allowing us to identify every interesting neural frequency band.

Chronic recordings from the marmoset motor cortex reveals modulation of neural firing and local field potentials overlap with macaques

Objective.The common marmoset has been increasingly used in neural interfacing studies due to its smaller size, easier handling, and faster breeding compared to Old World non-human primate (NHP) species. While assessment of cortical anatomy in marmosets has shown strikingly similar layout to macaques, comprehensive assessment of electrophysiological properties underlying forelimb reaching movements in this bridge species does not exist. The objective of this study is to characterize electrophysiological properties of signals recorded from the marmoset primary motor cortex (M1) during a reach task and compare with larger NHP models such that this smaller NHP model can be used in behavioral neural interfacing studies.Approach and main results.Neuronal firing rates and local field potentials (LFPs) were chronically recorded from M1 in three adult, male marmosets. Firing rates, mu + beta and high gamma frequency bands of LFPs were evaluated for modulation with respect to movement. Firing rate and regularity of neurons of the marmoset M1 were similar to that reported in macaques with a subset of neurons showing selectivity to movement direction. Movement phases (rest vs move) was classified from both neural spiking and LFPs. Microelectrode arrays provide the ability to sample small regions of the motor cortex to drive brain-machine interfaces (BMIs). The results demonstrate that marmosets are a robust bridge species for behavioral neuroscience studies with motor cortical electrophysiological signals recorded from microelectrode arrays that are similar to Old World NHPs.Significance. As marmosets represent an interesting step between rodent and macaque models, successful demonstration that neuron modulation in marmoset motor cortex is analogous to reports in macaques illustrates the utility of marmosets as a viable species for BMI studies.

Home Use of a Percutaneous Wireless Intracortical Brain-Computer Interface by Individuals With Tetraplegia

OBJECTIVE

Individuals with neurological disease or injury such as amyotrophic lateral sclerosis, spinal cord injury or stroke may become tetraplegic, unable to speak or even locked-in. For people with these conditions, current assistive technologies are often ineffective. Brain-computer interfaces are being developed to enhance independence and restore communication in the absence of physical movement. Over the past decade, individuals with tetraplegia have achieved rapid on-screen typing and point-and-click control of tablet apps using intracortical brain-computer interfaces (iBCIs) that decode intended arm and hand movements from neural signals recorded by implanted microelectrode arrays. However, cables used to convey neural signals from the brain tether participants to amplifiers and decoding computers and require expert oversight, severely limiting when and where iBCIs could be available for use. Here, we demonstrate the first human use of a wireless broadband iBCI.

METHODS

Based on a prototype system previously used in pre-clinical research, we replaced the external cables of a 192-electrode iBCI with wireless transmitters and achieved high-resolution recording and decoding of broadband field potentials and spiking activity from people with paralysis. Two participants in an ongoing pilot clinical trial completed on-screen item selection tasks to assess iBCI-enabled cursor control.

RESULTS

Communication bitrates were equivalent between cabled and wireless configurations. Participants also used the wireless iBCI to control a standard commercial tablet computer to browse the web and use several mobile applications. Within-day comparison of cabled and wireless interfaces evaluated bit error rate, packet loss, and the recovery of spike rates and spike waveforms from the recorded neural signals. In a representative use case, the wireless system recorded intracortical signals from two arrays in one participant continuously through a 24-hour period at home.

SIGNIFICANCE

Wireless multi-electrode recording of broadband neural signals over extended periods introduces a valuable tool for human neuroscience research and is an important step toward practical deployment of iBCI technology for independent use by individuals with paralysis. On-demand access to high-performance iBCI technology in the home promises to enhance independence and restore communication and mobility for individuals with severe motor impairment.

Single-neuronal predictions of others' beliefs in humans

Human social behaviour crucially depends on our ability to reason about others. This capacity for theory of mind has a vital role in social cognition because it enables us not only to form a detailed understanding of the hidden thoughts and beliefs of other individuals but also to understand that they may differ from our own^1-3. Although a number of areas in the human brain have been linked to social reasoning^4,5 and its disruption across a variety of psychosocial disorders^6-8, the basic cellular mechanisms that underlie human theory of mind remain undefined. Here, using recordings from single cells in the human dorsomedial prefrontal cortex, we identify neurons that reliably encode information about others' beliefs across richly varying scenarios and that distinguish self- from other-belief-related representations. By further following their encoding dynamics, we show how these cells represent the contents of the others' beliefs and accurately predict whether they are true or false. We also show how they track inferred beliefs from another's specific perspective and how their activities relate to behavioural performance. Together, these findings reveal a detailed cellular process in the human dorsomedial prefrontal cortex for representing another's beliefs and identify candidate neurons that could support theory of mind.

Information diversity in individual auditory cortical neurons is associated with functionally distinct coordinated neuronal ensembles

Neuronal activity in auditory cortex is often highly synchronous between neighboring neurons. Such coordinated activity is thought to be crucial for information processing. We determined the functional properties of coordinated neuronal ensembles (cNEs) within primary auditory cortical (AI) columns relative to the contributing neurons. Nearly half of AI cNEs showed robust spectro-temporal receptive fields whereas the remaining cNEs showed little or no acoustic feature selectivity. cNEs can therefore capture either specific, time-locked information of spectro-temporal stimulus features or reflect stimulus-unspecific, less-time specific processing aspects. By contrast, we show that individual neurons can represent both of those aspects through membership in multiple cNEs with either high or absent feature selectivity. These associations produce functionally heterogeneous spikes identifiable by instantaneous association with different cNEs. This demonstrates that single neuron spike trains can sequentially convey multiple aspects that contribute to cortical processing, including stimulus-specific and unspecific information.

Multiscale low-dimensional motor cortical state dynamics predict naturalistic reach-and-grasp behavior

Motor function depends on neural dynamics spanning multiple spatiotemporal scales of population activity, from spiking of neurons to larger-scale local field potentials (LFP). How multiple scales of low-dimensional population dynamics are related in control of movements remains unknown. Multiscale neural dynamics are especially important to study in naturalistic reach-and-grasp movements, which are relatively under-explored. We learn novel multiscale dynamical models for spike-LFP network activity in monkeys performing naturalistic reach-and-grasps. We show low-dimensional dynamics of spiking and LFP activity exhibited several principal modes, each with a unique decay-frequency characteristic. One principal mode dominantly predicted movements. Despite distinct principal modes existing at the two scales, this predictive mode was multiscale and shared between scales, and was shared across sessions and monkeys, yet did not simply replicate behavioral modes. Further, this multiscale mode's decay-frequency explained behavior. We propose that multiscale, low-dimensional motor cortical state dynamics reflect the neural control of naturalistic reach-and-grasp behaviors.

Robust and accurate decoding of hand kinematics from entire spiking activity using deep learning

OBJECTIVE

Brain-machine interfaces (BMIs) seek to restore lost motor functions in individuals with neurological disorders by enabling them to control external devices directly with their thoughts. This work aims to improve robustness and decoding accuracy that currently become major challenges in the clinical translation of intracortical BMIs.

APPROACH

We propose entire spiking activity (ESA) -an envelope of spiking activity that can be extracted by a simple, threshold-less, and automated technique- as the input signal. We couple ESA with deep learning-based decoding algorithm that uses quasi-recurrent neural network (QRNN) architecture. We evaluate comprehensively the performance of ESA-driven QRNN decoder for decoding hand kinematics from neural signals chronically recorded from the primary motor cortex area of three non-human primates performing different tasks.

MAIN RESULTS

Our proposed method yields consistently higher decoding performance than any other combinations of the input signal and decoding algorithm previously reported across long term recording sessions. It can sustain high decoding performance even when removing spikes from the raw signals, when using the different number of channels, and when using a smaller amount of training data.

SIGNIFICANCE

Overall results demonstrate exceptionally high decoding accuracy and chronic robustness, which is highly desirable given it is an unresolved challenge in BMIs.

Power-saving design opportunities for wireless intracortical brain-computer interfaces

The efficacy of wireless intracortical brain-computer interfaces (iBCIs) is limited in part by the number of recording channels, which is constrained by the power budget of the implantable system. Designing wireless iBCIs that provide the high-quality recordings of today's wired neural interfaces may lead to inadvertent over-design at the expense of power consumption and scalability. Here, we report analyses of neural signals collected from experimental iBCI measurements in rhesus macaques and from a clinical-trial participant with implanted 96-channel Utah multielectrode arrays to understand the trade-offs between signal quality and decoder performance. Moreover, we propose an efficient hardware design for clinically viable iBCIs, and suggest that the circuit design parameters of current recording iBCIs can be relaxed considerably without loss of performance. The proposed design may allow for an order-of-magnitude power savings and lead to clinically viable iBCIs with a higher channel count.

Decoding movement direction from cortical microelectrode recordings using an LSTM-based neural network

Brain-machine interfaces (BMIs) allow individuals to communicate with computers using neural signals, and Kalman Filter (KF) are prevailingly used to decode movement directions from these neural signals. In this paper, we implemented a multi-layer long short-term memory (LSTM)based artificial neural network (ANN) for decoding BMI neural signals. We collected motor cortical neural signals from a nonhuman primate (NHP), implanted with microelectrode array (MEA) while performing a directional joystick task. Next, we compared the LSTM model in decoding the joystick trajectories from the neural signals against the prevailing KF model. The results showed that the LSTM model yielded significantly improved decoding accuracy measured by mean correlation coefficient (0.84, p < 10^-7) than the KF model (0.72). In addition, using a principal component analysis (PCA)-based dimensionality reduction technique yielded slightly deteriorated accuracies for both the LSTM (0.80) and KF (0.70) models, but greatly reduced the computational complexity. The results showed that the LSTM decoding model holds promise to improve decoding in BMIs for paralyzed humans.

Principled BCI Decoder Design and Parameter Selection Using a Feedback Control Model

Decoders optimized offline to reconstruct intended movements from neural recordings sometimes fail to achieve optimal performance online when they are used in closed-loop as part of an intracortical brain-computer interface (iBCI). This is because typical decoder calibration routines do not model the emergent interactions between the decoder, the user, and the task parameters (e.g. target size). Here, we investigated the feasibility of simulating online performance to better guide decoder parameter selection and design. Three participants in the BrainGate2 pilot clinical trial controlled a computer cursor using a linear velocity decoder under different gain (speed scaling) and temporal smoothing parameters and acquired targets with different radii and distances. We show that a user-specific iBCI feedback control model can predict how performance changes under these different decoder and task parameters in held-out data. We also used the model to optimize a nonlinear speed scaling function for the decoder. When used online with two participants, it increased the dynamic range of decoded speeds and decreased the time taken to acquire targets (compared to an optimized standard decoder). These results suggest that it is feasible to simulate iBCI performance accurately enough to be useful for quantitative decoder optimization and design.

Accurate Estimation of Neural Population Dynamics without Spike Sorting

A central goal of systems neuroscience is to relate an organism's neural activity to behavior. Neural population analyses often reduce the data dimensionality to focus on relevant activity patterns. A major hurdle to data analysis is spike sorting, and this problem is growing as the number of recorded neurons increases. Here, we investigate whether spike sorting is necessary to estimate neural population dynamics. The theory of random projections suggests that we can accurately estimate the geometry of low-dimensional manifolds from a small number of linear projections of the data. We recorded data using Neuropixels probes in motor cortex of nonhuman primates and reanalyzed data from three previous studies and found that neural dynamics and scientific conclusions are quite similar using multiunit threshold crossings rather than sorted neurons. This finding unlocks existing data for new analyses and informs the design and use of new electrode arrays for laboratory and clinical use.

Reliability of motor and sensory neural decoding by threshold crossings for intracortical brain-machine interface

OBJECTIVE

For intracortical neurophysiological studies, spike sorting is an important procedure to isolate single units for analyzing specific functions. However, whether spike sorting is necessary or not for neural decoding applications is controversial. Several studies showed that using threshold crossings (TC) instead of spike sorting could also achieve a similar satisfactory performance. However, such studies were limited in similar behavioral tasks, and the neural signal source mainly focused on the motor-related cortical regions. It is not certain if this conclusion is applicable to other situations. Therefore, we compared the performance of TC and spike sorting in neural decoding with more comprehensive paradigms and parameters.

APPROACH

Two rhesus macaques implanted with Utah or floating microelectrode arrays (FMAs) in motor or sensory-related cortical regions were trained to perform a motor or a sensory task. Data from each monkey were preprocessed with three different schemes: TC, automatic sorting (AS), and manual sorting (MS). A support vector machine was used as the decoder, and the decoding accuracy was used for evaluating the performance of three preprocessing methods. Different neural signal sources, different decoders, and related parameters and decoding stability were further tested to systematically compare three preprocessing methods.

MAIN RESULTS

TC could achieve a similar (-4.5 RMS threshold) or better (-3.0 RMS threshold) decoding performance compared to the other two sorting methods in the motor or sensory tasks even if the neural signal sources or decoder-related parameters were changed. Moreover, TC was much more stable in neural decoding across sessions and robust to changes of threshold.

SIGNIFICANCE

Our results indicated that spike-firing patterns could be stably extracted through TC from multiple cortices in both motor and sensory neural decoding applications. Considering the stability of TC, it might be more suitable for neural decoding compared to sorting methods.

Watch, Imagine, Attempt: Motor Cortex Single-Unit Activity Reveals Context-Dependent Movement Encoding in Humans With Tetraplegia

Planning and performing volitional movement engages widespread networks in the human brain, with motor cortex considered critical to the performance of skilled limb actions. Motor cortex is also engaged when actions are observed or imagined, but the manner in which ensembles of neurons represent these volitional states (VoSs) is unknown. Here we provide direct demonstration that observing, imagining or attempting action activates shared neural ensembles in human motor cortex. Two individuals with tetraplegia (due to brainstem stroke or amyotrophic lateral sclerosis, ALS) were verbally instructed to watch, imagine, or attempt reaching actions displayed on a computer screen. Neural activity in the precentral gyrus incorporated information about both cognitive state and movement kinematics; the three conditions presented overlapping but unique, statistically distinct activity patterns. These findings demonstrate that individual neurons in human motor cortex reflect information related to sensory inputs and VoS in addition to movement features, and are a key part of a broader network linking perception and cognition to action.

The critical stability task: quantifying sensory-motor control during ongoing movement in nonhuman primates

Everyday behaviors require that we interact with the environment, using sensory information in an ongoing manner to guide our actions. Yet, by design, many of the tasks used in primate neurophysiology laboratories can be performed with limited sensory guidance. As a consequence, our knowledge about the neural mechanisms of motor control is largely limited to the feedforward aspects of the motor command. To study the feedback aspects of volitional motor control, we adapted the critical stability task (CST) from the human performance literature (Jex H, McDonnell J, Phatak A. IEEE Trans Hum Factors Electron 7: 138-145, 1966). In the CST, our monkey subjects interact with an inherently unstable (i.e., divergent) virtual system and must generate sensory-guided actions to stabilize it about an equilibrium point. The difficulty of the CST is determined by a single parameter, which allows us to quantitatively establish the limits of performance in the task for different sensory feedback conditions. Two monkeys learned to perform the CST with visual or vibrotactile feedback. Performance was better under visual feedback, as expected, but both monkeys were able to utilize vibrotactile feedback alone to successfully perform the CST. We also observed changes in behavioral strategy as the task became more challenging. The CST will have value for basic science investigations of the neural basis of sensory-motor integration during ongoing actions, and it may also provide value for the design and testing of bidirectional brain computer interface systems. NEW & NOTEWORTHY Currently, most behavioral tasks used in motor neurophysiology studies require primates to make short-duration, stereotyped movements that do not necessitate sensory feedback. To improve our understanding of sensorimotor integration, and to engineer meaningful artificial sensory feedback systems for brain-computer interfaces, it is crucial to have a task that requires sensory feedback for good control. The critical stability task demands that sensory information be used to guide long-duration movements.

Characterization of the Neuroinflammatory Response to Thiol-ene Shape Memory Polymer Coated Intracortical Microelectrodes

Thiol-ene based shape memory polymers (SMPs) have been developed for use as intracortical microelectrode substrates. The unique chemistry provides precise control over the mechanical and thermal glass-transition properties. As a result, SMP substrates are stiff at room temperature, allowing for insertion into the brain without buckling and subsequently soften in response to body temperatures, reducing the mechanical mismatch between device and tissue. Since the surface chemistry of the materials can contribute significantly to the ultimate biocompatibility, as a first step in the characterization of our SMPs, we sought to isolate the biological response to the implanted material surface without regards to the softening mechanics. To accomplish this, we tightly controlled for bulk stiffness by comparing bare silicon 'dummy' devices to thickness-matched silicon devices dip-coated with SMP. The neuroinflammatory response was evaluated after devices were implanted in the rat cortex for 2 or 16 weeks. We observed no differences in the markers tested at either time point, except that astrocytic scarring was significantly reduced for the dip-coated implants at 16 weeks. The surface properties of non-softening thiol-ene SMP substrates appeared to be equally-tolerated and just as suitable as silicon for neural implant substrates for applications such as intracortical microelectrodes, laying the groundwork for future softer devices to improve upon the prototype device performance presented here.

A High Definition Noninvasive Neuromuscular Electrical Stimulation System for Cortical Control of Combinatorial Rotary Hand Movements in a Human With Tetraplegia

OBJECTIVE

Paralysis resulting from spinal cord injury (SCI) can have a devastating effect on multiple arm and hand motor functions. Rotary hand movements, such as supination and pronation, are commonly impaired by upper extremity paralysis, and are essential for many activities of daily living. In this proof-of-concept study, we utilize a neural bypass system (NBS) to decode motor intention from motor cortex to control combinatorial rotary hand movements elicited through stimulation of the arm muscles, effectively bypassing the SCI of the study participant. We describe the NBS system architecture and design that enabled this functionality.

METHODS

The NBS consists of three main functional components: 1) implanted intracortical microelectrode array, 2) neural data processing using a computer, and, 3) a noninvasive neuromuscular electrical stimulation (NMES) system.

RESULTS

We address previous limitations of the NBS, and confirm the enhanced capability of the NBS to enable, in real-time, combinatorial hand rotary motor functions during a functionally relevant object manipulation task.

CONCLUSION

This enhanced capability was enabled by accurate decoding of multiple movement intentions from the participant's motor cortex, interleaving NMES patterns to combine hand movements, and dynamically switching between NMES patterns to adjust for hand position changes during movement.

SIGNIFICANCE

These results have implications for enabling complex rotary hand functions in sequence with other functionally relevant movements for patients suffering from SCI, stroke, and other sensorimotor dysfunctions.

Extracting wavelet based neural features from human intracortical recordings for neuroprosthetics applications

Background

Understanding the long-term behavior of intracortically-recorded signals is essential for improving the performance of Brain Computer Interfaces. However, few studies have systematically investigated chronic neural recordings from an implanted microelectrode array in the human brain.

Methods

In this study, we show the applicability of wavelet decomposition method to extract and demonstrate the utility of long-term stable features in neural signals obtained from a microelectrode array implanted in the motor cortex of a human with tetraplegia. Wavelet decomposition was applied to the raw voltage data to generate mean wavelet power (MWP) features, which were further divided into three sub-frequency bands, low-frequency MWP (lf-MWP, 0–234 Hz), mid-frequency MWP (mf-MWP, 234 Hz–3.75 kHz) and high-frequency MWP (hf-MWP, >3.75 kHz). We analyzed these features using data collected from two experiments that were repeated over the course of about 3 years and compared their signal stability and decoding performance with the more standard threshold crossings, local field potentials (LFP), multi-unit activity (MUA) features obtained from the raw voltage recordings.

Results

All neural features could stably track neural information for over 3 years post-implantation and were less prone to signal degradation compared to threshold crossings. Furthermore, when used as an input to support vector machine based decoding algorithms, the mf-MWP and MUA demonstrated significantly better performance, respectively, in classifying imagined motor tasks than using the lf-MWP, hf-MWP, LFP, or threshold crossings.

Conclusions

Our results suggest that using MWP features in the appropriate frequency bands can provide an effective neural feature for brain computer interface intended for chronic applications.

Trial registration

This study was approved by the U.S. Food and Drug Administration (Investigational Device Exemption) and the Ohio State University Medical Center Institutional Review Board (Columbus, Ohio). The study conformed to institutional requirements for the conduct of human subjects and was filed on ClinicalTrials.gov (Identifier NCT01997125).

Electronic supplementary material

The online version of this article (10.1186/s42234-018-0011-x) contains supplementary material, which is available to authorized users.

Evaluation of Decoding Algorithms for Estimating Bladder Pressure from Dorsal Root Ganglia Neural Recordings

A closed-loop device for bladder control may offer greater clinical benefit compared to current open-loop stimulation devices. Previous studies have demonstrated the feasibility of using single-unit recordings from sacral-level dorsal root ganglia (DRG) for decoding bladder pressure. Automatic online sorting, to differentiate single units, can be computationally heavy and unreliable, in contrast to simple multi-unit thresholded activity. In this study, the feasibility of using DRG multi-unit recordings to decode bladder pressure was examined. A broad range of feature selection methods and three algorithms (multivariate linear regression, basic Kalman filter, and a nonlinear autoregressive moving average model) were used to create training models and provide validation fits to bladder pressure for data collected in seven anesthetized feline experiments. A non-linear autoregressive moving average (NARMA) model with regularization provided the most accurate bladder pressure estimate, based on normalized root-mean-squared error, NRMSE, (17 ± 7%). A basic Kalman filter yielded the highest similarity to the bladder pressure with an average correlation coefficient, CC, of 0.81 ± 0.13. The best algorithm set (based on NRMSE) was further evaluated on data obtained from a chronic feline experiment. Testing results yielded a NRMSE and CC of 10.7% and 0.61, respectively from a model that was trained on data recorded 2 weeks prior. From offline analysis, implementation of NARMA in a closed-loop scheme for detecting bladder contractions would provide a robust control signal. Ultimate integration of closed-loop algorithms in bladder neuroprostheses will require evaluations of parameter and signal stability over time.

Idle state classification using spiking activity and local field potentials in a brain computer interface

Previous studies of intracortical brain-computer interfaces (BCIs) have often focused on or compared the use of spiking activity and local field potentials (LFPs) for decoding kinematic movement parameters. Conversely, using these signals to detect the initial intention to use a neuroprosthetic device or not has remained a relatively understudied problem. In this study, we examined the relative performance of spiking activity and LFP signals in detecting discrete state changes in attention regarding a user's desire to actively control a BCI device. Preliminary offline results suggest that the beta and high gamma frequency bands of LFP activity demonstrated a capacity for discriminating idle/active BCI control states equal to or greater than firing rate activity on the same channel. Population classifier models using either signal modality demonstrated an indistinguishably high degree of accuracy in decoding rest periods from active BCI reach periods as well as other portions of active BCI task trials. These results suggest that either signal modality may be used to reliably detect discrete state changes on a fine time scale for the purpose of gating neural prosthetic movements.

Sums of Spike Waveform Features for Motor Decoding

Traditionally, the key step before decoding motor intentions from cortical recordings is spike sorting, the process of identifying which neuron was responsible for an action potential. Recently, researchers have started investigating approaches to decoding which omit the spike sorting step, by directly using information about action potentials' waveform shapes in the decoder, though this approach is not yet widespread. Particularly, one recent approach involves computing the moments of waveform features and using these moment values as inputs to decoders. This computationally inexpensive approach was shown to be comparable in accuracy to traditional spike sorting. In this study, we use offline data recorded from two Rhesus monkeys to further validate this approach. We also modify this approach by using sums of exponentiated features of spikes, rather than moments. Our results show that using waveform feature sums facilitates significantly higher hand movement reconstruction accuracy than using waveform feature moments, though the magnitudes of differences are small. We find that using the sums of one simple feature, the spike amplitude, allows better offline decoding accuracy than traditional spike sorting by expert (correlation of 0.767, 0.785 vs. 0.744, 0.738, respectively, for two monkeys, average 16% reduction in mean-squared-error), as well as unsorted threshold crossings (0.746, 0.776; average 9% reduction in mean-squared-error). Our results suggest that the sums-of-features framework has potential as an alternative to both spike sorting and using unsorted threshold crossings, if developed further. Also, we present data comparing sorted vs. unsorted spike counts in terms of offline decoding accuracy. Traditional sorted spike counts do not include waveforms that do not match any template (“hash”), but threshold crossing counts do include this hash. On our data and in previous work, hash contributes to decoding accuracy. Thus, using the comparison between sorted spike counts and threshold crossing counts to evaluate the benefit of sorting is confounded by the presence of hash. We find that when the comparison is controlled for hash, performing sorting is better than not. These results offer a new perspective on the question of to sort or not to sort.

Signal-independent noise in intracortical brain-computer interfaces causes movement time properties inconsistent with Fitts' law

OBJECTIVE

Do movements made with an intracortical BCI (iBCI) have the same movement time properties as able-bodied movements? Able-bodied movement times typically obey Fitts' law: [Formula: see text] (where MT is movement time, D is target distance, R is target radius, and [Formula: see text] are parameters). Fitts' law expresses two properties of natural movement that would be ideal for iBCIs to restore: (1) that movement times are insensitive to the absolute scale of the task (since movement time depends only on the ratio [Formula: see text]) and (2) that movements have a large dynamic range of accuracy (since movement time is logarithmically proportional to [Formula: see text]).

APPROACH

Two participants in the BrainGate2 pilot clinical trial made cortically controlled cursor movements with a linear velocity decoder and acquired targets by dwelling on them. We investigated whether the movement times were well described by Fitts' law.

MAIN RESULTS

We found that movement times were better described by the equation [Formula: see text], which captures how movement time increases sharply as the target radius becomes smaller, independently of distance. In contrast to able-bodied movements, the iBCI movements we studied had a low dynamic range of accuracy (absence of logarithmic proportionality) and were sensitive to the absolute scale of the task (small targets had long movement times regardless of the [Formula: see text] ratio). We argue that this relationship emerges due to noise in the decoder output whose magnitude is largely independent of the user's motor command (signal-independent noise). Signal-independent noise creates a baseline level of variability that cannot be decreased by trying to move slowly or hold still, making targets below a certain size very hard to acquire with a standard decoder.

SIGNIFICANCE

The results give new insight into how iBCI movements currently differ from able-bodied movements and suggest that restoring a Fitts' law-like relationship to iBCI movements may require non-linear decoding strategies.

Making brain-machine interfaces robust to future neural variability

A major hurdle to clinical translation of brain–machine interfaces (BMIs) is that current decoders, which are trained from a small quantity of recent data, become ineffective when neural recording conditions subsequently change. We tested whether a decoder could be made more robust to future neural variability by training it to handle a variety of recording conditions sampled from months of previously collected data as well as synthetic training data perturbations. We developed a new multiplicative recurrent neural network BMI decoder that successfully learned a large variety of neural-to-kinematic mappings and became more robust with larger training data sets. Here we demonstrate that when tested with a non-human primate preclinical BMI model, this decoder is robust under conditions that disabled a state-of-the-art Kalman filter-based decoder. These results validate a new BMI strategy in which accumulated data history are effectively harnessed, and may facilitate reliable BMI use by reducing decoder retraining downtime.

Brain-machine interfaces (BMI) depend on algorithms to decode neural signals, but these decoders cope poorly with signal variability. Here, authors report a BMI decoder which circumvents these problems by using a large and perturbed training dataset to improve performance with variable neural signals.

Continuous decoding of human grasp kinematics using epidural and subdural signals

OBJECTIVE

Restoring or replacing function in paralyzed individuals will one day be achieved through the use of brain-machine interfaces. Regaining hand function is a major goal for paralyzed patients. Two competing prerequisites for the widespread adoption of any hand neuroprosthesis are accurate control over the fine details of movement, and minimized invasiveness. Here, we explore the interplay between these two goals by comparing our ability to decode hand movements with subdural and epidural field potentials (EFPs).

APPROACH

We measured the accuracy of decoding continuous hand and finger kinematics during naturalistic grasping motions in five human subjects. We recorded subdural surface potentials (electrocorticography; ECoG) as well as with EFPs, with both standard- and high-resolution electrode arrays.

MAIN RESULTS

In all five subjects, decoding of continuous kinematics significantly exceeded chance, using either EGoG or EFPs. ECoG decoding accuracy compared favorably with prior investigations of grasp kinematics (mean ± SD grasp aperture variance accounted for was 0.54 ± 0.05 across all subjects, 0.75 ± 0.09 for the best subject). In general, EFP decoding performed comparably to ECoG decoding. The 7-20 Hz and 70-115 Hz spectral bands contained the most information about grasp kinematics, with the 70-115 Hz band containing greater information about more subtle movements. Higher-resolution recording arrays provided clearly superior performance compared to standard-resolution arrays.

SIGNIFICANCE

To approach the fine motor control achieved by an intact brain-body system, it will be necessary to execute motor intent on a continuous basis with high accuracy. The current results demonstrate that this level of accuracy might be achievable not just with ECoG, but with EFPs as well. Epidural placement of electrodes is less invasive, and therefore may incur less risk of encephalitis or stroke than subdural placement of electrodes. Accurately decoding motor commands at the epidural level may be an important step towards a clinically viable brain-machine interface.

Extracellular voltage threshold settings can be tuned for optimal encoding of movement and stimulus parameters

OBJECTIVE

A traditional goal of neural recording with extracellular electrodes is to isolate action potential waveforms of an individual neuron. Recently, in brain-computer interfaces (BCIs), it has been recognized that threshold crossing events of the voltage waveform also convey rich information. To date, the threshold for detecting threshold crossings has been selected to preserve single-neuron isolation. However, the optimal threshold for single-neuron identification is not necessarily the optimal threshold for information extraction. Here we introduce a procedure to determine the best threshold for extracting information from extracellular recordings. We apply this procedure in two distinct contexts: the encoding of kinematic parameters from neural activity in primary motor cortex (M1), and visual stimulus parameters from neural activity in primary visual cortex (V1).

APPROACH

We record extracellularly from multi-electrode arrays implanted in M1 or V1 in monkeys. Then, we systematically sweep the voltage detection threshold and quantify the information conveyed by the corresponding threshold crossings.

MAIN RESULTS

The optimal threshold depends on the desired information. In M1, velocity is optimally encoded at higher thresholds than speed; in both cases the optimal thresholds are lower than are typically used in BCI applications. In V1, information about the orientation of a visual stimulus is optimally encoded at higher thresholds than is visual contrast. A conceptual model explains these results as a consequence of cortical topography.

SIGNIFICANCE

How neural signals are processed impacts the information that can be extracted from them. Both the type and quality of information contained in threshold crossings depend on the threshold setting. There is more information available in these signals than is typically extracted. Adjusting the detection threshold to the parameter of interest in a BCI context should improve our ability to decode motor intent, and thus enhance BCI control. Further, by sweeping the detection threshold, one can gain insights into the topographic organization of the nearby neural tissue.

Brain-state classification and a dual-state decoder dramatically improve the control of cursor movement through a brain-machine interface

OBJECTIVE

It is quite remarkable that brain machine interfaces (BMIs) can be used to control complex movements with fewer than 100 neurons. Success may be due in part to the limited range of dynamical conditions under which most BMIs are tested. Achieving high-quality control that spans these conditions with a single linear mapping will be more challenging. Even for simple reaching movements, existing BMIs must reduce the stochastic noise of neurons by averaging the control signals over time, instead of over the many neurons that normally control movement. This forces a compromise between a decoder with dynamics allowing rapid movement and one that allows postures to be maintained with little jitter. Our current work presents a method for addressing this compromise, which may also generalize to more highly varied dynamical situations, including movements with more greatly varying speed.

APPROACH

We have developed a system that uses two independent Wiener filters as individual components in a single decoder, one optimized for movement, and the other for postural control. We computed an LDA classifier using the same neural inputs. The decoder combined the outputs of the two filters in proportion to the likelihood assigned by the classifier to each state.

MAIN RESULTS

We have performed online experiments with two monkeys using this neural-classifier, dual-state decoder, comparing it to a standard, single-state decoder as well as to a dual-state decoder that switched states automatically based on the cursor's proximity to a target. The performance of both monkeys using the classifier decoder was markedly better than that of the single-state decoder and comparable to the proximity decoder.

SIGNIFICANCE

We have demonstrated a novel strategy for dealing with the need to make rapid movements while also maintaining precise cursor control when approaching and stabilizing within targets. Further gains can undoubtedly be realized by optimizing the performance of the individual movement and posture decoders.