Decoding arm speed during reaching

Degradation-aware neural imputation: Advancing decoding stability in brain machine interfaces

Neural signal degradation poses a significant challenge in maintaining stable performance when decoding motor tasks using multiunit activity (MUA) and local field potential (LFP) signals in the implantable brain machine interface (iBMI) applications. Effective methods for imputing degraded or missing signals are essential to restore neural signal integrity, thereby improving decoding accuracy and system robustness over long-term recordings with fluctuating signal quality. This study introduces a confidence-weighted Bayesian linear regression (CW-BLR) approach to impute neural signals affected by degradation, enhancing the robustness and consistency of decoding. The performance of CW-BLR was compared to traditional methods-mean imputation (Mean-imp) and Gaussian-mixture-model-based expectation-maximization (GMM-EM)-using a kernel-sliced inverse regression (kSIR) decoder to evaluate decoding outcomes. Four Wistar rats were trained to perform a forelimb-reaching task while neural activity (MUA and LFPs) was recorded over 27 days. CW-BLR imputed signals degraded during days 8-27. Decoding performance was evaluated using kSIR and compared with Mean-imp and GMM-EM. CW-BLR demonstrated superior performance by effectively preserving both temporal and spatial dependencies within the neural signals. CW-BLR-imputed data significantly improved decoding accuracy over traditional imputation methods, with the kSIR decoder showing consistently higher performance, particularly in maintaining signal quality from the degraded period. CW-BLR offers a robust and effective imputation framework for iBMI applications, addressing signal degradation challenges and maintaining accurate decoding over prolonged recordings. By utilizing confidence-based quality metrics, CW-BLR surpasses traditional methods, providing stable neural decoding across fluctuating signal quality scenarios.

Reveal the mechanism of brain function with fluorescence microscopy at single-cell resolution: from neural decoding to encoding

As a key pathway for understanding behavior, cognition, and emotion, neural decoding and encoding provide effective tools to bridge the gap between neural mechanisms and imaging recordings, especially at single-cell resolution. While neural decoding aims to establish an interpretable theory of how complex biological behaviors are represented in neural activities, neural encoding focuses on manipulating behaviors through the stimulation of specific neurons. We thoroughly analyze the application of fluorescence imaging techniques, particularly two-photon fluorescence imaging, in decoding neural activities, showcasing the theoretical analysis and technological advancements from imaging recording to behavioral manipulation. For decoding models, we compared linear and nonlinear methods, including independent component analysis, random forests, and support vector machines, highlighting their capabilities to reveal the intricate mapping between neural activity and behavior. By employing synthetic stimuli via optogenetics, fundamental principles of neural encoding are further explored. We elucidate various encoding types based on different stimulus paradigms-quantity encoding, spatial encoding, temporal encoding, and frequency encoding-enhancing our understanding of how the brain represents and processes information. We believe that fluorescence imaging-based neural decoding and encoding techniques have deepened our understanding of the brain, and hold great potential in paving the way for future neuroscience research and clinical applications.

Neural geometry from mixed sensorimotor selectivity for predictive sensorimotor control

Although recent studies suggest that activity in the motor cortex, in addition to generating motor outputs, receives substantial information regarding sensory inputs, it is still unclear how sensory context adjusts the motor commands. Here, we recorded population neural activity in the motor cortex via microelectrode arrays while monkeys performed flexible manual interceptions of moving targets. During this task, which requires predictive sensorimotor control, the activity of most neurons in the motor cortex encoding upcoming movements was influenced by ongoing target motion. Single-trial neural states at the movement onset formed staggered orbital geometries, suggesting that target motion modulates peri-movement activity in an orthogonal manner. This neural geometry was further evaluated with a representational model and recurrent neural networks (RNNs) with task-specific input-output mapping. We propose that the sensorimotor dynamics can be derived from neuronal mixed sensorimotor selectivity and dynamic interaction between modulations.

Reward influences movement vigor through multiple motor cortical mechanisms

The prospect of greater rewards often invigorates movements. What neural mechanisms support this increase of movement vigor for greater rewards? We had three rhesus monkeys perform reaching movements to targets worth different magnitudes of reward. We recorded neural population activity from primary motor and dorsal premotor cortex, brain areas at the output of cortical processing for voluntary movements, and asked how neural activity mediated the translation of reward into increased vigor. We identified features of neural activity during movement preparation, initiation, and execution that were both correlated with vigor and modulated by reward. We also found that the neural metrics that correlate with different aspects of movement vigor exhibit only limited correlation with one another, suggesting that there are multiple mechanisms through which reward modulates vigor. Finally, we note that the majority of reward's modulation of motor cortical activity cannot be accounted for by reward-mediated vigor differences in behavior, indicating that reward modulations within motor cortex may serve roles in addition to affecting vigor. Overall, our results provide insight into the neural mechanisms that link reward-driven motivation to the modulation of the details of movement.

Human motor cortex encodes complex handwriting through a sequence of stable neural states

How the human motor cortex (MC) orchestrates sophisticated sequences of fine movements such as handwriting remains a puzzle. Here we investigate this question through Utah array recordings from human MC during attempted handwriting of Chinese characters (n = 306, each consisting of 6.3 ± 2.0 strokes). We find that MC activity evolves through a sequence of states corresponding to the writing of stroke fragments during complicated handwriting. The directional tuning curve of MC neurons remains stable within states, but its gain or preferred direction strongly varies across states. By building models that can automatically infer the neural states and implement state-dependent directional tuning, we can significantly better explain the firing pattern of individual neurons and reconstruct recognizable handwriting trajectories with 69% improvement compared with baseline models. Our findings unveil that skilled and sophisticated movements are encoded through state-specific neural configurations.

Time Varying Encoding of Grasping Type and Force in the Primate Motor Cortex

The primary motor cortex (M1) is strongly engaged by movement planning and execution. However, the role of M1 activity in voluntary grasping is still not completely understood. Here we analyze recordings of M1 neurons during the execution of a delayed reach-to-grasp task, where monkeys had to actively grasp an object with either a side or a precision grip, and then pull it with a low or high amount of force. Single cell and neural populations analyses showed that grip type was robustly and specifically encoded by a large population of neurons, while force level was weakly and transiently encoded within mixed-selective neurons that also encoded grip type. Notably, the grip type was stably decoded from motor cortical populations during the preparation and execution epochs of the task. Our results are consistent with the idea that planning and performing specific grasping movements are high-level skills that strongly engage M1 neurons, while the execution of pulling force might be prominently encoded at lower stages of the motor system.

Explaining human motor coordination via the synergy expansion hypothesis

The search for an answer to Bernstein's degrees of freedom problem has propelled a large portion of research studies in human motor control over the past six decades. Different theories have been developed to explain how humans might use their incredibly complex neuro-musculo-skeletal system with astonishing ease. Among these theories, motor synergies appeared as one possible explanation. In this work, the authors investigate the nature and role of synergies and propose a theoretical framework, namely the "expansion hypothesis," to answer Bernstein's problem. The expansion hypothesis is articulated in three propositions: mechanical, developmental, and behavioral. Each proposition addresses a different question on the nature of synergies: i) How many synergies can humans have? ii) How do we learn and develop synergies? iii) How do we use synergies? An example numerical simulation is presented and analyzed to clarify the hypothesis propositions. The expansion hypothesis is contextualized with respect to the existing literature on motor synergies both in healthy and impaired individuals, as well as other prominent theories in human motor control and development. The expansion hypothesis provides a framework to better comprehend and explain the nature, use, and evolution of human motor skills.

Less is more: selection from a small set of options improves BCI velocity control

Objective.Decoding algorithms used in invasive brain-computer interfaces (iBCIs) typically convert neural activity into continuously varying velocity commands. We hypothesized that putting constraints on which decoded velocity commands are permissible could improve user performance. To test this hypothesis, we designed the discrete direction selection (DDS) decoder, which uses neural activity to select among a small menu of preset cursor velocities.Approach. We tested DDS in a closed-loop cursor control task against many common continuous velocity decoders in both a human-operated real-time iBCI simulator (the jaBCI) and in a monkey using an iBCI. In the jaBCI, we compared performance across four visits by each of 48 naïve, able-bodied human subjects using either DDS, direct regression with assist (an affine map from neural activity to cursor velocity, DR-A), ReFIT, or the velocity Kalman Filter (vKF). In a follow up study to verify the jaBCI results, we compared a monkey's performance using an iBCI with either DDS or the Wiener filter decoder (a direct regression decoder that includes time history, WF).Main Result. In the jaBCI, DDS substantially outperformed all other decoders with 93% mean targets hit per visit compared to DR-A, ReFIT, and vKF with 56%, 39%, and 26% mean targets hit, respectively. With the iBCI, the monkey achieved a 61% success rate with DDS and a 37% success rate with WF.Significance. Discretizing the decoded velocity with DDS effectively traded high resolution velocity commands for less tortuous and lower noise trajectories, highlighting the potential benefits of discretization in simplifying online BCI control.

Hybrid Neural Network Models Explain Cortical Neuronal Activity During Volitional Movement

Massive interconnectivity in large-scale neural networks is the key feature underlying their powerful and complex functionality. We have developed hybrid neural network (HNN) models that allow us to find statistical structure in this connectivity. Describing this structure is critical for understanding biological and artificial neural networks. The HNNs are composed of artificial neurons, a subset of which are trained to reproduce the responses of individual neurons recorded experimentally. The experimentally observed firing rates came from populations of neurons recorded in the motor cortices of monkeys performing a reaching task. After training, these networks (recurrent and spiking) underwent the same state transitions as those observed in the empirical data, a result that helps resolve a long-standing question of prescribed vs ongoing control of volitional movement. Because all aspects of the models are exposed, we were able to analyze the dynamic statistics of the connections between neurons. Our results show that the dynamics of extrinsic input to the network changed this connectivity to cause the state transitions. Two processes at the synaptic level were recognized: one in which many different neurons contributed to a buildup of membrane potential and another in which more specific neurons triggered an action potential. HNNs facilitate modeling of realistic neuron-neuron connectivity and provide foundational descriptions of large-scale network functionality.

Accurate neural control of a hand prosthesis by posture-related activity in the primate grasping circuit

Brain-computer interfaces (BCIs) have the potential to restore hand movement for people with paralysis, but current devices still lack the fine control required to interact with objects of daily living. Following our understanding of cortical activity during arm reaches, hand BCI studies have focused primarily on velocity control. However, mounting evidence suggests that posture, and not velocity, dominates in hand-related areas. To explore whether this signal can causally control a prosthesis, we developed a BCI training paradigm centered on the reproduction of posture transitions. Monkeys trained with this protocol were able to control a multidimensional hand prosthesis with high accuracy, including execution of the very intricate precision grip. Analysis revealed that the posture signal in the target grasping areas was the main contributor to control. We present, for the first time, neural posture control of a multidimensional hand prosthesis, opening the door for future interfaces to leverage this additional information channel.

Enhanced control of a brain-computer interface by tetraplegic participants via neural-network-mediated feature extraction

To infer intent, brain-computer interfaces must extract features that accurately estimate neural activity. However, the degradation of signal quality over time hinders the use of feature-engineering techniques to recover functional information. By using neural data recorded from electrode arrays implanted in the cortices of three human participants, here we show that a convolutional neural network can be used to map electrical signals to neural features by jointly optimizing feature extraction and decoding under the constraint that all the electrodes must use the same neural-network parameters. In all three participants, the neural network led to offline and online performance improvements in a cursor-control task across all metrics, outperforming the rate of threshold crossings and wavelet decomposition of the broadband neural data (among other feature-extraction techniques). We also show that the trained neural network can be used without modification for new datasets, brain areas and participants.

Multiscale effective connectivity analysis of brain activity using neural ordinary differential equations

Neural mechanisms and underlying directionality of signaling among brain regions depend on neural dynamics spanning multiple spatiotemporal scales of population activity. Despite recent advances in multimodal measurements of brain activity, there is no broadly accepted multiscale dynamical models for the collective activity represented in neural signals. Here we introduce a neurobiological-driven deep learning model, termed multiscale neural dynamics neural ordinary differential equation (msDyNODE), to describe multiscale brain communications governing cognition and behavior. We demonstrate that msDyNODE successfully captures multiscale activity using both simulations and electrophysiological experiments. The msDyNODE-derived causal interactions between recording locations and scales not only aligned well with the abstraction of the hierarchical neuroanatomy of the mammalian central nervous system but also exhibited behavioral dependences. This work offers a new approach for mechanistic multiscale studies of neural processes.

Decoding multi-limb movements from two-photon calcium imaging of neuronal activity using deep learning

Objective.Brain-machine interfaces (BMIs) aim to restore sensorimotor function to individuals suffering from neural injury and disease. A critical step in implementing a BMI is to decode movement intention from recorded neural activity patterns in sensorimotor areas. Optical imaging, including two-photon (2p) calcium imaging, is an attractive approach for recording large-scale neural activity with high spatial resolution using a minimally-invasive technique. However, relating slow two-photon calcium imaging data to fast behaviors is challenging due to the relatively low optical imaging sampling rates. Nevertheless, neural activity recorded with 2p calcium imaging has been used to decode information about stereotyped single-limb movements and to control BMIs. Here, we expand upon prior work by applying deep learning to decode multi-limb movements of running mice from 2p calcium imaging data.Approach.We developed a recurrent encoder-decoder network (LSTM-encdec) in which the output is longer than the input.Main results.LSTM-encdec could accurately decode information about all four limbs (contralateral and ipsilateral front and hind limbs) from calcium imaging data recorded in a single cortical hemisphere.Significance.Our approach provides interpretability measures to validate decoding accuracy and expands the utility of BMIs by establishing the groundwork for control of multiple limbs. Our work contributes to the advancement of neural decoding techniques and the development of next-generation optical BMIs.

Brain-computer interfaces: the innovative key to unlocking neurological conditions

Neurological disorders such as Parkinson's disease, stroke, and spinal cord injury can pose significant threats to human mortality, morbidity, and functional independence. Brain-Computer Interface (BCI) technology, which facilitates direct communication between the brain and external devices, emerges as an innovative key to unlocking neurological conditions, demonstrating significant promise in this context. This comprehensive review uniquely synthesizes the latest advancements in BCI research across multiple neurological disorders, offering an interdisciplinary perspective on both clinical applications and emerging technologies. We explore the progress in BCI research and its applications in addressing various neurological conditions, with a particular focus on recent clinical studies and prospective developments. Initially, the review provides an up-to-date overview of BCI technology, encompassing its classification, operational principles, and prevalent paradigms. It then critically examines specific BCI applications in movement disorders, disorders of consciousness, cognitive and mental disorders, as well as sensory disorders, highlighting novel approaches and their potential impact on patient care. This review reveals emerging trends in BCI applications, such as the integration of artificial intelligence and the development of closed-loop systems, which represent significant advancements over previous technologies. The review concludes by discussing the prospects and directions of BCI technology, underscoring the need for interdisciplinary collaboration and ethical considerations. It emphasizes the importance of prioritizing bidirectional and high-performance BCIs, areas that have been underexplored in previous reviews. Additionally, we identify crucial gaps in current research, particularly in long-term clinical efficacy and the need for standardized protocols. The role of neurosurgery in spearheading the clinical translation of BCI research is highlighted. Our comprehensive analysis presents BCI technology as an innovative key to unlocking neurological disorders, offering a transformative approach to diagnosing, treating, and rehabilitating neurological conditions, with substantial potential to enhance patients' quality of life and advance the field of neurotechnology.

Revealing unexpected complex encoding but simple decoding mechanisms in motor cortex via separating behaviorally relevant neural signals

In motor cortex, behaviorally relevant neural responses are entangled with irrelevant signals, which complicates the study of encoding and decoding mechanisms. It remains unclear whether behaviorally irrelevant signals could conceal some critical truth. One solution is to accurately separate behaviorally relevant and irrelevant signals at both single-neuron and single-trial levels, but this approach remains elusive due to the unknown ground truth of behaviorally relevant signals. Therefore, we propose a framework to define, extract, and validate behaviorally relevant signals. Analyzing separated signals in three monkeys performing different reaching tasks, we found neural responses previously considered to contain little information actually encode rich behavioral information in complex nonlinear ways. These responses are critical for neuronal redundancy and reveal movement behaviors occupy a higher-dimensional neural space than previously expected. Surprisingly, when incorporating often-ignored neural dimensions, behaviorally relevant signals can be decoded linearly with comparable performance to nonlinear decoding, suggesting linear readout may be performed in motor cortex. Our findings prompt that separating behaviorally relevant signals may help uncover more hidden cortical mechanisms.

Initial and corrective submovement encoding differences within primary motor cortex during precision reaching

Precision reaching often requires corrective submovements to obtain the desired goal. Most studies of reaching have focused on single initial movements, and implied the cortical encoding model was the same for all submovements. However, corrective submovements may show different encoding patterns from the initial submovement with distinct patterns of activation across the population. Two rhesus macaques performed a precision center-out-task with small targets. Neural activity from single units in the primary motor cortex and associated behavioral data were recorded to evaluate movement characteristics. Neural population data and individual neuronal firing rates identified with a peak finding algorithm to identify peaks in hand speed were examined for encoding differences between initial and corrective submovements. Individual neurons were fitted with a regression model that included the reach vector, position, and speed to predict firing rate. For both initial and corrective submovements, the largest effect remained movement direction. We observed a large subset changed their preferred direction greater than 45° between initial and corrective submovements. Neuronal depth of modulation also showed considerable variation when adjusted for movement speed. By using principal component analysis, neural trajectories of initial and corrective submovements progressed through different neural subspaces. These findings all suggest that different neural encoding patterns exist for initial and corrective submovements within the cortex. We hypothesize that this variation in how neurons change to encode small, corrective submovements might allow for a larger portion of the neural space being used to encode a greater range of movements with varying amplitudes and levels of precision.NEW & NOTEWORTHY Neuronal recordings matched with kinematic behavior were collected in a precision center-out task that often required corrective movements. We reveal large differences in preferred direction and depth of modulation between initial and corrective submovements across the neural population. We then present a model of the neural population describing how these shifts in tuning create different subspaces for signaling initial and corrective movements likely to improve motor precision.

Spike Neural Network of Motor Cortex Model for Arm Reaching Control

Motor cortex modeling is crucial for understanding movement planning and execution. While interconnected recurrent neural networks have successfully described the dynamics of neural population activity, most existing methods utilize continuous signal-based neural networks, which do not reflect the biological spike neural signal. To address this limitation, we propose a recurrent spike neural network to simulate motor cortical activity during an arm-reaching task. Specifically, our model is built upon integrate-and-fire spiking neurons with conductance-based synapses. We carefully designed the interconnections of neurons with two different firing time scales - "fast" and "slow" neurons. Experimental results demonstrate the effectiveness of our method, with the model's neuronal activity in good agreement with monkey's motor cortex data at both single-cell and population levels. Quantitative analysis reveals a correlation coefficient 0.89 between the model's and real data. These results suggest the possibility of multiple timescales in motor cortical control.

Less is more: selection from a small set of options improves BCI velocity control

We designed the discrete direction selection (DDS) decoder for intracortical brain computer interface (iBCI) cursor control and showed that it outperformed currently used decoders in a human-operated real-time iBCI simulator and in monkey iBCI use. Unlike virtually all existing decoders that map between neural activity and continuous velocity commands, DDS uses neural activity to select among a small menu of preset cursor velocities. We compared closed-loop cursor control across four visits by each of 48 naïve, able-bodied human subjects using either DDS or one of three common continuous velocity decoders: direct regression with assist (an affine map from neural activity to cursor velocity), ReFIT, and the velocity Kalman Filter. DDS outperformed all three by a substantial margin. Subsequently, a monkey using an iBCI also had substantially better performance with DDS than with the Wiener filter decoder (direct regression decoder that includes time history). Discretizing the decoded velocity with DDS effectively traded high resolution velocity commands for less tortuous and lower noise trajectories, highlighting the potential benefits of simplifying online iBCI control.

Motor cortex latent dynamics encode spatial and temporal arm movement parameters independently

The fluid movement of an arm requires multiple spatiotemporal parameters to be set independently. Recent studies have argued that arm movements are generated by the collective dynamics of neurons in motor cortex. An untested prediction of this hypothesis is that independent parameters of movement must map to independent components of the neural dynamics. Using a task where monkeys made a sequence of reaching movements to randomly placed targets, we show that the spatial and temporal parameters of arm movements are independently encoded in the low-dimensional trajectories of population activity in motor cortex: Each movement's direction corresponds to a fixed neural trajectory through neural state space and its speed to how quickly that trajectory is traversed. Recurrent neural network models show this coding allows independent control over the spatial and temporal parameters of movement by separate network parameters. Our results support a key prediction of the dynamical systems view of motor cortex, but also argue that not all parameters of movement are defined by different trajectories of population activity.

Machine learning decoding of single neurons in the thalamus for speech brain-machine interfaces

Objective. Our goal is to decode firing patterns of single neurons in the left ventralis intermediate nucleus (Vim) of the thalamus, related to speech production, perception, and imagery. For realistic speech brain-machine interfaces (BMIs), we aim to characterize the amount of thalamic neurons necessary for high accuracy decoding.Approach. We intraoperatively recorded single neuron activity in the left Vim of eight neurosurgical patients undergoing implantation of deep brain stimulator or RF lesioning during production, perception and imagery of the five monophthongal vowel sounds. We utilized the Spade decoder, a machine learning algorithm that dynamically learns specific features of firing patterns and is based on sparse decomposition of the high dimensional feature space.Main results. Spade outperformed all algorithms compared with, for all three aspects of speech: production, perception and imagery, and obtained accuracies of 100%, 96%, and 92%, respectively (chance level: 20%) based on pooling together neurons across all patients. The accuracy was logarithmic in the amount of neurons for all three aspects of speech. Regardless of the amount of units employed, production gained highest accuracies, whereas perception and imagery equated with each other.Significance. Our research renders single neuron activity in the left Vim a promising source of inputs to BMIs for restoration of speech faculties for locked-in patients or patients with anarthria or dysarthria to allow them to communicate again. Our characterization of how many neurons are necessary to achieve a certain decoding accuracy is of utmost importance for planning BMI implantation.

Calibrating Bayesian Decoders of Neural Spiking Activity

Accurately decoding external variables from observations of neural activity is a major challenge in systems neuroscience. Bayesian decoders, which provide probabilistic estimates, are some of the most widely used. Here we show how, in many common settings, the probabilistic predictions made by traditional Bayesian decoders are overconfident. That is, the estimates for the decoded stimulus or movement variables are more certain than they should be. We then show how Bayesian decoding with latent variables, taking account of low-dimensional shared variability in the observations, can improve calibration, although additional correction for overconfidence is still needed. Using data from males, we examine (1) decoding the direction of grating stimuli from spike recordings in the primary visual cortex in monkeys, (2) decoding movement direction from recordings in the primary motor cortex in monkeys, (3) decoding natural images from multiregion recordings in mice, and (4) decoding position from hippocampal recordings in rats. For each setting, we characterize the overconfidence, and we describe a possible method to correct miscalibration post hoc. Properly calibrated Bayesian decoders may alter theoretical results on probabilistic population coding and lead to brain-machine interfaces that more accurately reflect confidence levels when identifying external variables.

Multiplicative joint coding in preparatory activity for reaching sequence in macaque motor cortex

Although the motor cortex has been found to be modulated by sensory or cognitive sequences, the linkage between multiple movement elements and sequence-related responses is not yet understood. Here, we recorded neuronal activity from the motor cortex with implanted micro-electrode arrays and single electrodes while monkeys performed a double-reach task that was instructed by simultaneously presented memorized cues. We found that there existed a substantial multiplicative component jointly tuned to impending and subsequent reaches during preparation, then the coding mechanism transferred to an additive manner during execution. This multiplicative joint coding, which also spontaneously emerged in recurrent neural networks trained for double reach, enriches neural patterns for sequential movement, and might explain the linear readout of elemental movements.

Brownian processes in human motor control support descending neural velocity commands

The motor neuroscience literature suggests that the central nervous system may encode some motor commands in terms of velocity. In this work, we tackle the question: what consequences would velocity commands produce at the behavioral level? Considering the ubiquitous presence of noise in the neuromusculoskeletal system, we predict that velocity commands affected by stationary noise would produce "random walks", also known as Brownian processes, in position. Brownian motions are distinctively characterized by a linearly growing variance and a power spectral density that declines in inverse proportion to frequency. This work first shows that these Brownian processes are indeed observed in unbounded motion tasks e.g., rotating a crank. We further predict that such growing variance would still be present, but bounded, in tasks requiring a constant posture e.g., maintaining a static hand position or quietly standing. This hypothesis was also confirmed by experimental observations. A series of descriptive models are investigated to justify the observed behavior. Interestingly, one of the models capable of accounting for all the experimental results must feature forward-path velocity commands corrupted by stationary noise. The results of this work provide behavioral support for the hypothesis that humans plan the motion components of their actions in terms of velocity.

Putative cell-type-specific multiregional mode in posterior parietal cortex during coordinated visual behavior

In the reach and saccade regions of the posterior parietal cortex (PPC), multiregional communication depends on the timing of neuronal activity with respect to beta-frequency (10-30 Hz) local field potential (LFP) activity, termed dual coherence. Neural coherence is believed to reflect neural excitability, whereby spiking tends to occur at a particular phase of LFP activity, but the mechanisms of multiregional dual coherence remain unknown. Here, we investigate dual coherence in the PPC of non-human primates performing eye-hand movements. We computationally model dual coherence in terms of multiregional neural excitability and show that one latent component, a multiregional mode, reflects shared excitability across distributed PPC populations. Analyzing the power in the multiregional mode with respect to different putative cell types reveals significant modulations with the spiking of putative pyramidal neurons and not inhibitory interneurons. These results suggest a specific role for pyramidal neurons in dual coherence supporting multiregional communication in PPC.

Decoding and geometry of ten finger movements in human posterior parietal cortex and motor cortex

Objective. Enable neural control of individual prosthetic fingers for participants with upper-limb paralysis.Approach. Two tetraplegic participants were each implanted with a 96-channel array in the left posterior parietal cortex (PPC). One of the participants was additionally implanted with a 96-channel array near the hand knob of the left motor cortex (MC). Across tens of sessions, we recorded neural activity while the participants attempted to move individual fingers of the right hand. Offline, we classified attempted finger movements from neural firing rates using linear discriminant analysis with cross-validation. The participants then used the neural classifier online to control individual fingers of a brain-machine interface (BMI). Finally, we characterized the neural representational geometry during individual finger movements of both hands.Main Results. The two participants achieved 86% and 92% online accuracy during BMI control of the contralateral fingers (chance = 17%). Offline, a linear decoder achieved ten-finger decoding accuracies of 70% and 66% using respective PPC recordings and 75% using MC recordings (chance = 10%). In MC and in one PPC array, a factorized code linked corresponding finger movements of the contralateral and ipsilateral hands.Significance. This is the first study to decode both contralateral and ipsilateral finger movements from PPC. Online BMI control of contralateral fingers exceeded that of previous finger BMIs. PPC and MC signals can be used to control individual prosthetic fingers, which may contribute to a hand restoration strategy for people with tetraplegia.

The influence of non-stationarity of spike signals on decoding performance in intracortical brain-computer interface: a simulation study

Introduction

Intracortical Brain-Computer Interfaces (iBCI) establish a new pathway to restore motor functions in individuals with paralysis by interfacing directly with the brain to translate movement intention into action. However, the development of iBCI applications is hindered by the non-stationarity of neural signals induced by the recording degradation and neuronal property variance. Many iBCI decoders were developed to overcome this non-stationarity, but its effect on decoding performance remains largely unknown, posing a critical challenge for the practical application of iBCI.

Methods

To improve our understanding on the effect of non-stationarity, we conducted a 2D-cursor simulation study to examine the influence of various types of non-stationarities. Concentrating on spike signal changes in chronic intracortical recording, we used the following three metrics to simulate the non-stationarity: mean firing rate (MFR), number of isolated units (NIU), and neural preferred directions (PDs). MFR and NIU were decreased to simulate the recording degradation while PDs were changed to simulate the neuronal property variance. Performance evaluation based on simulation data was then conducted on three decoders and two different training schemes. Optimal Linear Estimation (OLE), Kalman Filter (KF), and Recurrent Neural Network (RNN) were implemented as decoders and trained using static and retrained schemes.

Results

In our evaluation, RNN decoder and retrained scheme showed consistent better performance under small recording degradation. However, the serious signal degradation would cause significant performance to drop eventually. On the other hand, RNN performs significantly better than the other two decoders in decoding simulated non-stationary spike signals, and the retrained scheme maintains the decoders' high performance when changes are limited to PDs.

Discussion

Our simulation work demonstrates the effects of neural signal non-stationarity on decoding performance and serves as a reference for selecting decoders and training schemes in chronic iBCI. Our result suggests that comparing to KF and OLE, RNN has better or equivalent performance using both training schemes. Performance of decoders under static scheme is influenced by recording degradation and neuronal property variation while decoders under retrained scheme are only influenced by the former one.

Interplay between external inputs and recurrent dynamics during movement preparation and execution in a network model of motor cortex

The primary motor cortex has been shown to coordinate movement preparation and execution through computations in approximately orthogonal subspaces. The underlying network mechanisms, and the roles played by external and recurrent connectivity, are central open questions that need to be answered to understand the neural substrates of motor control. We develop a recurrent neural network model that recapitulates the temporal evolution of neuronal activity recorded from the primary motor cortex of a macaque monkey during an instructed delayed-reach task. In particular, it reproduces the observed dynamic patterns of covariation between neural activity and the direction of motion. We explore the hypothesis that the observed dynamics emerges from a synaptic connectivity structure that depends on the preferred directions of neurons in both preparatory and movement-related epochs, and we constrain the strength of both synaptic connectivity and external input parameters from data. While the model can reproduce neural activity for multiple combinations of the feedforward and recurrent connections, the solution that requires minimum external inputs is one where the observed patterns of covariance are shaped by external inputs during movement preparation, while they are dominated by strong direction-specific recurrent connectivity during movement execution. Our model also demonstrates that the way in which single-neuron tuning properties change over time can explain the level of orthogonality of preparatory and movement-related subspaces.

From Parametric Representation to Dynamical System: Shifting Views of the Motor Cortex in Motor Control

In contrast to traditional representational perspectives in which the motor cortex is involved in motor control via neuronal preference for kinetics and kinematics, a dynamical system perspective emerging in the last decade views the motor cortex as a dynamical machine that generates motor commands by autonomous temporal evolution. In this review, we first look back at the history of the representational and dynamical perspectives and discuss their explanatory power and controversy from both empirical and computational points of view. Here, we aim to reconcile the above perspectives, and evaluate their theoretical impact, future direction, and potential applications in brain-machine interfaces.

Machine learning algorithm for decoding multiple subthalamic spike trains for speech brain-machine interfaces

Objective. The goal of this study is to decode the electrical activity of single neurons in the human subthalamic nucleus (STN) to infer the speech features that a person articulated, heard or imagined. We also aim to evaluate the amount of subthalamic neurons required for high accuracy decoding suitable for real-life speech brain-machine interfaces (BMI).Approach. We intraoperatively recorded single-neuron activity in the STN of 21 neurosurgical patients with Parkinson's disease undergoing implantation of deep brain stimulator while patients produced, perceived or imagined the five monophthongal vowel sounds. Our decoder is based on machine learning algorithms that dynamically learn specific features of the speech-related firing patterns.Main results. In an extensive comparison of algorithms, our sparse decoder ('SpaDe'), based on sparse decomposition of the high dimensional neuronal feature space, outperformed the other algorithms in all three conditions: production, perception and imagery. For speech production, our algorithm, Spade, predicted all vowels correctly (accuracy: 100%; chance level: 20%). For perception accuracy was 96%, and for imagery: 88%. The accuracy of Spade showed a linear behavior in the amount of neurons for the perception data, and even faster for production or imagery.Significance. Our study demonstrates that the information encoded by single neurons in the STN about the production, perception and imagery of speech is suitable for high-accuracy decoding. It is therefore an important step towards BMIs for restoration of speech faculties that bears an enormous potential to alleviate the suffering of completely paralyzed ('locked-in') patients and allow them to communicate again with their environment. Moreover, our research indicates how many subthalamic neurons may be necessary to achieve each level of decoding accuracy, which is of supreme importance for a neurosurgeon planning the implantation of a speech BMI.

Hierarchical Control of Visually-Guided Movements in a 3D-Printed Robot Arm

The control architecture guiding simple movements such as reaching toward a visual target remains an open problem. The nervous system needs to integrate different sensory modalities and coordinate multiple degrees of freedom in the human arm to achieve that goal. The challenge increases due to noise and transport delays in neural signals, non-linear and fatigable muscles as actuators, and unpredictable environmental disturbances. Here we examined the capabilities of hierarchical feedback control models proposed by W. T. Powers, so far only tested in silico. We built a robot arm system with four degrees of freedom, including a visual system for locating the planar position of the hand, joint angle proprioception, and pressure sensing in one point of contact. We subjected the robot to various human-inspired reaching and tracking tasks and found features of biological movement, such as isochrony and bell-shaped velocity profiles in straight-line movements, and the speed-curvature power law in curved movements. These behavioral properties emerge without trajectory planning or explicit optimization algorithms. We then applied static structural perturbations to the robot: we blocked the wrist joint, tilted the writing surface, extended the hand with a tool, and rotated the visual system. For all of them, we found that the arm in machina adapts its behavior without being reprogrammed. In sum, while limited in speed and precision (by the nature of the do-it-yourself inexpensive components we used to build the robot from scratch), when faced with the noise, delays, non-linearities, and unpredictable disturbances of the real world, the embodied control architecture shown here balances biological realism with design simplicity.

Improving reaching with functional electrical stimulation by incorporating stiffness modulation

Objective.Intracortical recordings have now been combined with functional electrical stimulation (FES) of arm/hand muscles to demonstrate restoration of upper-limb function after spinal cord injury. However, for each desired limb position decoded from the brain, there are multiple combinations of muscle stimulation levels that can produce that position. The objective of this simulation study is to explore how modulating the amount of coactivation of antagonist muscles during FES can impact reaching performance and energy usage. Stiffening the limb by cocontracting antagonist muscles makes the limb more resistant to perturbation. Minimizing cocontraction saves energy and reduces fatigue.Approach.Prior demonstrations of reaching via FES used a fixed empirically-derived lookup table for each joint that defined the muscle stimulation levels that would position the limb at the desired joint angle decoded from the brain at each timestep. This study expands on that previous work by using simulations to: (a) test the feasibility of controlling arm reaching using asuiteof lookup tables with varying levels of cocontraction instead of a single fixed lookup table for each joint, (b) optimize a simple function for automatically switching between these different cocontraction tables using only the desired kinematic information already being decoded from the brain, and (c) compare energy savings and movement performance when using the optimized function to automatically modulate cocontraction during reaching versus using the best fixed level of cocontraction.Main results.Our data suggests energy usage and/or movement performance can be significantly improved by dynamically modulating limb stiffness using our multi-table method and a simple function that determines cocontraction level based on decoded endpoint speed and its derivative.Significance.By demonstrating how modulating cocontraction can reduce energy usage while maintaining or even improving movement performance, this study makes brain-controlled FES a more viable option for restoration of reaching after paralysis.

Online asynchronous detection of error-related potentials in participants with a spinal cord injury using a generic classifier

For brain-computer interface (BCI) users, the awareness of an error is associated with a cortical signature known as an error-related potential (ErrP). The incorporation of ErrP detection into BCIs can improve their performance.

OBJECTIVE

This work has three main aims. First, we investigate whether an ErrP classifier is transferable from able-bodied participants to participants with a spinal cord injury (SCI). Second, we test this generic ErrP classifier with SCI and control participants, in an online experiment without offline calibration. Third, we investigate the morphology of ErrPs in both groups of participants.

APPROACH

We used previously recorded electroencephalographic data from able-bodied participants to train an ErrP classifier. We tested the classifier asynchronously, in an online experiment with 16 new participants: 8 participants with SCI and 8 able-bodied control participants. The experiment had no offline calibration and participants received feedback regarding the ErrP detections from the start. To increase the fluidity of the experiment, feedback regarding false positive ErrP detections was not presented to the participants, but these detections were taken into account in the evaluation of the classifier. The generic classifier was not trained with the user's brain signals. However, its performance was optimized during the online experiment by the use of personalized decision thresholds. The classifier's performance was evaluated using trial-based metrics, which considered the asynchronous detection of ErrPs during the entire trial's duration.

MAIN RESULTS

Participants with SCI presented a non-homogenous ErrP morphology, and four of them did not present clear ErrP signals. The generic classifier performed better than chance in participants with clear ErrP signals, independently of the SCI (11 out of 16 participants). Three out of the five participants that obtained chance level results with the generic classifier would have not benefitted from the use of a personalized classifier.

SIGNIFICANCE

This work shows the feasibility of transferring an ErrP classifier from able-bodied participants to participants with SCI, for asynchronous detection of ErrPs in an online experiment without offline calibration, which provided immediate feedback to the users.

Reconstruction of hand, elbow and shoulder actual and imagined trajectories in 3D space using EEG current source dipoles

OBJECTIVE

Growing evidence suggests that EEG electrode (sensor) potential time series (PTS) of slow cortical potentials (SCPs) hold motor neural correlates that can be used for motion trajectory prediction (MTP), commonly by multiple linear regression (mLR). It is not yet known whether arm-joint trajectories can be reliably decoded from current sources, computed from sensor data, from which brain areas they can be decoded and using which neural features.

APPROACH

In this study, the PTS of 44 sensors were fed into sLORETA source localization software to compute current source activity in 30 regions of interest (ROIs) found in a recent meta-analysis to be engaged in action execution, motor imagery and motor preparation. The current sources PTS and band-power time series (BTS) in several frequency bands and time lags were used to predict actual and imagined trajectories in 3D space of the three velocity components of the hand, elbow and shoulder of nine subjects using an mLR model.

MAIN RESULTS

For all arm joints and movement types, current source SCPs PTS contributed most to trajectory reconstruction with time lags 150ms, 116ms and 84ms providing the highest contribution, and current source BTS in any of the tested frequency bands was not informative. Person's correlation coefficient (r) averaged across movement types, arm joints and velocity components using source data was slightly lower than using sensor data (r=0.25 and r=0.28, respectively). For each ROI, the three current source dipoles had different contribution to the reconstruction of each of the three velocity components.

SIGNIFICANCE

Overall, our results demonstrate the feasibility of predicting of actual and imagined 3D trajectories of all arm joints from current sources, computed from scalp EEG. These findings may be used by developers of a future BCI as a validated set of contributing ROIs.

Distance- and speed-informed kinematics decoding improves M/EEG based upper-limb movement decoder accuracy

OBJECTIVE

One of the main goals in brain-computer interface (BCI) research is the replacement or restoration of lost function in individuals with paralysis. One line of research investigates the inference of movement kinematics from brain activity during different volitional states. A growing number of electroencephalography (EEG) and magnetoencephalography (MEG) studies suggest that information about directional (e.g. velocity) and nondirectional (e.g. speed) movement kinematics is accessible noninvasively. We sought to assess if the neural information associated with both types of kinematics can be combined to improve the decoding accuracy.

APPROACH

In an offline analysis, we reanalyzed the data of two previous experiments containing the recordings of 34 healthy participants (15 EEG, 19 MEG). We decoded 2D movement trajectories from low-frequency M/EEG signals in executed and observed tracking movements, and compared the accuracy of an unscented Kalman filter (UKF) that explicitly modeled the nonlinear relation between directional and nondirectional kinematics to the accuracies of linear Kalman (KF) and Wiener filters which did not combine both types of kinematics.

MAIN RESULTS

At the group level, posterior-parietal and parieto-occipital (executed and observed movements) and sensorimotor areas (executed movements) encoded kinematic information. Correlations between the recorded position and velocity trajectories and the UKF decoded ones were on average 0.49 during executed and 0.36 during observed movements. Compared to the other filters, the UKF could achieve the best trade-off between maximizing the signal to noise ratio and minimizing the amplitude mismatch between the recorded and decoded trajectories.

SIGNIFICANCE

We present direct evidence that directional and nondirectional kinematic information is simultaneously detectable in low-frequency M/EEG signals. Moreover, combining directional and nondirectional kinematic information significantly improves the decoding accuracy upon a linear KF.

A bioelectric neural interface towards intuitive prosthetic control for amputees

OBJECTIVE

While prosthetic hands with independently actuated digits have become commercially available, state-of-the-art human-machine interfaces (HMI) only permit control over a limited set of grasp patterns, which does not enable amputees to experience sufficient improvement in their daily activities to make an active prosthesis useful.

APPROACH

Here we present a technology platform combining fully-integrated bioelectronics, implantable intrafascicular microelectrodes and deep learning-based artificial intelligence (AI) to facilitate this missing bridge by tapping into the intricate motor control signals of peripheral nerves. The bioelectric neural interface includes an ultra-low-noise neural recording system to sense electroneurography (ENG) signals from microelectrode arrays implanted in the residual nerves, and AI models employing the recurrent neural network (RNN) architecture to decode the subject's motor intention.

MAIN RESULTS

A pilot human study has been carried out on a transradial amputee. We demonstrate that the information channel established by the proposed neural interface is sufficient to provide high accuracy control of a prosthetic hand up to 15 degrees of freedom (DOF). The interface is intuitive as it directly maps complex prosthesis movements to the patient's true intention.

SIGNIFICANCE

Our study layouts the foundation towards not only a robust and dexterous control strategy for modern neuroprostheses at a near-natural level approaching that of the able hand, but also an intuitive conduit for connecting human minds and machines through the peripheral neural pathways. (Clinical trial identifier: NCT02994160).

Activity in Primary Motor Cortex Related to Visual Feedback

Neural modulation in primate motor cortex exhibits complex patterns. We found that modulation during reaching could be separated into discrete temporal epochs. To determine if these epochs are driven by behavioral events, monkeys performed variations of a center-out reaching task. Monkeys viewed a computer cursor matched to hand position and a radial target at 1 of 16 locations. In some trials, they performed a visuomotor rotation (the cursor moved at an angle to the hand). After adaptation, encoding changes for single units are temporally structured: adaptation could affect one temporal component of a unit's response but not another. In half the normal and perturbed trials, we removed visual feedback before movement. Adaptation-sensitive firing components toward the end of movement are often weak or absent during reaches without feedback. These results show that temporal structure in motor cortical activity is driven by behavior, with a discrete component related to visual feedback.

Distributed processing of movement signaling

Basic neurophysiological research with monkeys has shown how neurons in the motor cortex have firing rates tuned to movement direction. This original finding would have been difficult to uncover without the use of a behaving primate paradigm in which subjects grasped a handle and moved purposefully to targets in different directions. Subsequent research, again using behaving primate models, extended these findings to continuous drawing and to arm and hand movements encompassing action across multiple joints. This research also led to robust extraction algorithms in which information from neuronal populations is used to decode movement intent. The ability to decode intended movement provided the foundation for neural prosthetics in which brain-controlled interfaces are used by paralyzed human subjects to control computer cursors or high-performance motorized prosthetic arms and hands. This translation of neurophysiological laboratory findings to therapy is a clear example of why using nonhuman primates for basic research is valuable for advancing treatment of neurological disorders. Recent research emphasizes the distribution of intention signaling through neuronal populations and shows how many movement parameters are encoded simultaneously. In addition to direction and velocity, the arm's impedance has now been found to be encoded as well. The ability to decode motion and force from neural populations will make it possible to extend neural prosthetic paradigms to precise interaction with objects, enabling paralyzed individuals to perform many tasks of daily living.

Principled BCI Decoder Design and Parameter Selection Using a Feedback Control Model

Decoders optimized offline to reconstruct intended movements from neural recordings sometimes fail to achieve optimal performance online when they are used in closed-loop as part of an intracortical brain-computer interface (iBCI). This is because typical decoder calibration routines do not model the emergent interactions between the decoder, the user, and the task parameters (e.g. target size). Here, we investigated the feasibility of simulating online performance to better guide decoder parameter selection and design. Three participants in the BrainGate2 pilot clinical trial controlled a computer cursor using a linear velocity decoder under different gain (speed scaling) and temporal smoothing parameters and acquired targets with different radii and distances. We show that a user-specific iBCI feedback control model can predict how performance changes under these different decoder and task parameters in held-out data. We also used the model to optimize a nonlinear speed scaling function for the decoder. When used online with two participants, it increased the dynamic range of decoded speeds and decreased the time taken to acquire targets (compared to an optimized standard decoder). These results suggest that it is feasible to simulate iBCI performance accurately enough to be useful for quantitative decoder optimization and design.