Motor cortical prediction of EMG: evidence that a kinetic brain-machine interface may be robust across altered movement dynamics

Nonlinear manifolds underlie neural population activity during behaviour

There is rich variety in the activity of single neurons recorded during behaviour. Yet, these diverse single neuron responses can be well described by relatively few patterns of neural co-modulation. The study of such low-dimensional structure of neural population activity has provided important insights into how the brain generates behaviour. Virtually all of these studies have used linear dimensionality reduction techniques to estimate these population-wide co-modulation patterns, constraining them to a flat "neural manifold". Here, we hypothesised that since neurons have nonlinear responses and make thousands of distributed and recurrent connections that likely amplify such nonlinearities, neural manifolds should be intrinsically nonlinear. Combining neural population recordings from monkey, mouse, and human motor cortex, and mouse striatum, we show that: 1) neural manifolds are intrinsically nonlinear; 2) their nonlinearity becomes more evident during complex tasks that require more varied activity patterns; and 3) manifold nonlinearity varies across architecturally distinct brain regions. Simulations using recurrent neural network models confirmed the proposed relationship between circuit connectivity and manifold nonlinearity, including the differences across architecturally distinct regions. Thus, neural manifolds underlying the generation of behaviour are inherently nonlinear, and properly accounting for such nonlinearities will be critical as neuroscientists move towards studying numerous brain regions involved in increasingly complex and naturalistic behaviours.

Using adversarial networks to extend brain computer interface decoding accuracy over time

Existing intracortical brain computer interfaces (iBCIs) transform neural activity into control signals capable of restoring movement to persons with paralysis. However, the accuracy of the 'decoder' at the heart of the iBCI typically degrades over time due to turnover of recorded neurons. To compensate, decoders can be recalibrated, but this requires the user to spend extra time and effort to provide the necessary data, then learn the new dynamics. As the recorded neurons change, one can think of the underlying movement intent signal being expressed in changing coordinates. If a mapping can be computed between the different coordinate systems, it may be possible to stabilize the original decoder's mapping from brain to behavior without recalibration. We previously proposed a method based on Generalized Adversarial Networks (GANs), called 'Adversarial Domain Adaptation Network' (ADAN), which aligns the distributions of latent signals within underlying low-dimensional neural manifolds. However, we tested ADAN on only a very limited dataset. Here we propose a method based on Cycle-Consistent Adversarial Networks (Cycle-GAN), which aligns the distributions of the full-dimensional neural recordings. We tested both Cycle-GAN and ADAN on data from multiple monkeys and behaviors and compared them to a third, quite different method based on Procrustes alignment of axes provided by Factor Analysis. All three methods are unsupervised and require little data, making them practical in real life. Overall, Cycle-GAN had the best performance and was easier to train and more robust than ADAN, making it ideal for stabilizing iBCI systems over time.

Neural Plasticity in Sensorimotor Brain-Machine Interfaces

Brain-machine interfaces (BMIs) aim to treat sensorimotor neurological disorders by creating artificial motor and/or sensory pathways. Introducing artificial pathways creates new relationships between sensory input and motor output, which the brain must learn to gain dexterous control. This review highlights the role of learning in BMIs to restore movement and sensation, and discusses how BMI design may influence neural plasticity and performance. The close integration of plasticity in sensory and motor function influences the design of both artificial pathways and will be an essential consideration for bidirectional devices that restore both sensory and motor function.

Linking cortex and contraction-Integrating models along the corticomuscular pathway

Computational models of the neuromusculoskeletal system provide a deterministic approach to investigate input-output relationships in the human motor system. Neuromusculoskeletal models are typically used to estimate muscle activations and forces that are consistent with observed motion under healthy and pathological conditions. However, many movement pathologies originate in the brain, including stroke, cerebral palsy, and Parkinson's disease, while most neuromusculoskeletal models deal exclusively with the peripheral nervous system and do not incorporate models of the motor cortex, cerebellum, or spinal cord. An integrated understanding of motor control is necessary to reveal underlying neural-input and motor-output relationships. To facilitate the development of integrated corticomuscular motor pathway models, we provide an overview of the neuromusculoskeletal modelling landscape with a focus on integrating computational models of the motor cortex, spinal cord circuitry, α-motoneurons  and skeletal muscle in regard to their role in generating voluntary muscle contraction. Further, we highlight the challenges and opportunities associated with an integrated corticomuscular pathway model, such as challenges in defining neuron connectivities, modelling standardisation, and opportunities in applying models to study emergent behaviour. Integrated corticomuscular pathway models have applications in brain-machine-interaction, education, and our understanding of neurological disease.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

Understanding Variable Motor Responses to Direct Electrical Stimulation of the Human Motor Cortex During Brain Surgery

Direct electrical stimulation of the brain is the gold standard technique used to define functional-anatomical relationships during neurosurgical procedures. Areas that respond to stimulation are considered “critical nodes” of circuits that must remain intact for the subject to maintain the ability to perform certain functions, like moving and speaking. Despite its routine use, the neurophysiology underlying downstream motor responses to electrical stimulation of the brain, such as muscle contraction or movement arrest, is poorly understood. Furthermore, varying and sometimes counterintuitive responses can be seen depending on how and where the stimulation is applied, even within the human primary motor cortex. Therefore, here we review relevant neuroanatomy of the human motor system, provide a brief historical perspective on electrical brain stimulation, explore mechanistic variations in stimulation applications, examine neurophysiological properties of different parts of the motor system, and suggest areas of future research that can promote a better understanding of the interaction between electrical stimulation of the brain and its function.

Deep learning multimodal fNIRS and EEG signals for bimanual grip force decoding

Objective.Non-invasive brain-machine interfaces (BMIs) offer an alternative, safe and accessible way to interact with the environment. To enable meaningful and stable physical interactions, BMIs need to decode forces. Although previously addressed in the unimanual case, controlling forces from both hands would enable BMI-users to perform a greater range of interactions. We here investigate the decoding of hand-specific forces.Approach.We maximise cortical information by using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) and developing a deep-learning architecture with attention and residual layers (cnnatt) to improve their fusion. Our task required participants to generate hand-specific force profiles on which we trained and tested our deep-learning and linear decoders.Main results.The use of EEG and fNIRS improved the decoding of bimanual force and the deep-learning models outperformed the linear model. In both cases, the greatest gain in performance was due to the detection of force generation. In particular, the detection of forces was hand-specific and better for the right dominant hand andcnnattwas better at fusing EEG and fNIRS. Consequently, the study ofcnnattrevealed that forces from each hand were differently encoded at the cortical level.Cnnattalso revealed traces of the cortical activity being modulated by the level of force which was not previously found using linear models.Significance.Our results can be applied to avoid hand-cross talk during hand force decoding to improve the robustness of BMI robotic devices. In particular, we improve the fusion of EEG and fNIRS signals and offer hand-specific interpretability of the encoded forces which are valuable during motor rehabilitation assessment.

Noise-assisted Multivariate Empirical Mode Decomposition based Causal Decomposition for brain-physiological network in bivariate and multiscale time series

Objective.Noise-assisted Multivariate Empirical Mode Decomposition (NA-MEMD) based Causal Decomposition depicts a cause and effect relationship that is not based on the term of prediction, but rather on the phase dependence of time series. Here, we present the NA-MEMD based Causal Decomposition approach according to the covariation and power views traced to Hume and Kant: a priori cause-effect interaction is first acquired, and the presence of a candidate cause and of the effect is then computed from the sensory input somehow.Approach.Based on the definition of NA-MEMD based Causal Decomposition, we show such causal relation is a phase relation where the candidate causes are not merely followed by effects, but rather produce effects.Main results.The predominant methods used in neuroscience (Granger causality, EMD-based Causal Decomposition) are validated, showing the applicability of NA-MEMD based Causal Decomposition, particular to brain physiological processes in bivariate and multiscale time series.Significance.We point to the potential use in the causality inference analysis in a complex dynamic process.

Prediction of Forelimb EMGs and Movement Phases from Corticospinal Signals in the Rat During the Reach-to-Pull Task

Brain-computer interfaces access the volitional command signals from various brain areas in order to substitute for the motor functions lost due to spinal cord injury or disease. As the final common pathway of the central nervous system (CNS) outputs, the descending tracts of the spinal cord offer an alternative site to extract movement-related command signals. Using flexible 2D microelectrode arrays, we have recorded the corticospinal tract (CST) signals in rats during a reach-to-pull task. The CST activity was then classified by the forelimb movement phases into two or three classes in a training dataset and cross validated in a test set. The average classification accuracies were [Formula: see text] (min: [Formula: see text] to max: [Formula: see text]) and [Formula: see text] (min: 43% to max: 71%) for two-class and three-class cases, respectively. The forelimb flexor and extensor EMG envelopes were also predicted from the CST signals using linear regression. The average correlation coefficient between the actual and predicted EMG signals was [Formula: see text] [Formula: see text], whereas the highest correlation was 0.81 for the biceps EMG. Although the forelimb motor function cannot be explained completely by the CST activity alone, the success rates obtained in reconstructing the EMG signals support the feasibility of a spinal-cord-computer interface as a concept.

Decoding of muscle activity from the sensorimotor cortex in freely behaving monkeys

Remarkable advances have recently been made in the development of Brain-Machine Interface (BMI) technologies for restoring or enhancing motor function. However, the application of these technologies may be limited to patients in static conditions, as these developments have been largely based on studies of animals (e.g., non-human primates) in constrained movement conditions. The ultimate goal of BMI technology is to enable individuals to move their bodies naturally or control external devices without physical constraints. Here, we demonstrate accurate decoding of muscle activity from electrocorticogram (ECoG) signals in unrestrained, freely behaving monkeys. We recorded ECoG signals from the sensorimotor cortex as well as electromyogram signals from multiple muscles in the upper arm while monkeys performed two types of movements with no physical restraints, as follows: forced forelimb movement (lever-pull task) and natural whole-body movement (free movement within the cage). As in previous reports using restrained monkeys, we confirmed that muscle activity during forced forelimb movement was accurately predicted from simultaneously recorded ECoG data. More importantly, we demonstrated that accurate prediction of muscle activity from ECoG data was possible in monkeys performing natural whole-body movement. We found that high-gamma activity in the primary motor cortex primarily contributed to the prediction of muscle activity during natural whole-body movement as well as forced forelimb movement. In contrast, the contribution of high-gamma activity in the premotor and primary somatosensory cortices was significantly larger during natural whole-body movement. Thus, activity in a larger area of the sensorimotor cortex was needed to predict muscle activity during natural whole-body movement. Furthermore, decoding models obtained from forced forelimb movement could not be generalized to natural whole-body movement, which suggests that decoders should be built individually and according to different behavior types. These results contribute to the future application of BMI systems in unrestrained individuals.

A muscle-activity-dependent gain between motor cortex and EMG

Whether one is delicately placing a contact lens on the surface of the eye or lifting a heavy weight from the floor, the motor system must produce a wide range of forces under different dynamical loads. How does the motor cortex, with neurons that have a limited activity range, function effectively under these widely varying conditions? In this study, we explored the interaction of activity in primary motor cortex (M1) and muscles (electromyograms, EMGs) of two male rhesus monkeys for wrist movements made during three tasks requiring different dynamical loads and forces. Despite traditionally providing adequate predictions in single tasks, in our experiments, a single linear model failed to account for the relation between M1 activity and EMG across conditions. However, a model with a gain parameter that increased with the target force remained accurate across forces and dynamical loads. Surprisingly, this model showed that a greater proportion of EMG changes were explained by the nonlinear gain than the linear mapping from M1. In addition to its theoretical implications, the strength of this nonlinearity has important implications for brain-computer interfaces (BCIs). If BCI decoders are to be used to control movement dynamics (including interaction forces) directly, they will need to be nonlinear and include training data from broad data sets to function effectively across tasks. Our study reinforces the need to investigate neural control of movement across a wide range of conditions to understand its basic characteristics as well as translational implications. NEW & NOTEWORTHY We explored the motor cortex-to-electromyogram (EMG) mapping across a wide range of forces and loading conditions, which we found to be highly nonlinear. A greater proportion of EMG was explained by a nonlinear gain than a linear mapping. This nonlinearity allows motor cortex to control the wide range of forces encountered in the real world. These results unify earlier observations and inform the next-generation brain-computer interfaces that will control movement dynamics and interaction forces.

Correlations Between Primary Motor Cortex Activity with Recent Past and Future Limb Motion During Unperturbed Reaching

Many studies highlight that human movements are highly successful yet display a surprising amount of variability from trial to trial. There is a consistent pattern of variability throughout movement: initial motor errors are corrected by the end of movement, suggesting the presence of a powerful online control process. Here, we analyze the trial-by-trial variability of goal-directed reaching in nonhuman primates (five male Rhesus monkeys) and demonstrate that they display a similar pattern of variability during reaching, including a strong negative correlation between initial and late hand motion. We then demonstrate that trial-to-trial neural variability of primary motor cortex (M1) is positively correlated with variability of future hand motion (τ = ∼160 ms) during reaching. Furthermore, the variability of M1 activity is also correlated with variability of past hand motion (τ = ∼90 ms), but in the opposite polarity (i.e., negative correlation). Partial correlation analysis demonstrated that M1 activity independently reflects the variability of both past and future hand motions. These findings provide support for the hypothesis that M1 activity is involved in online feedback control of motor actions.SIGNIFICANCE STATEMENT Previous studies highlight that primary motor cortex (M1) rapidly responds to either visual or mechanical disturbances, suggesting its involvement in online feedback control. However, these studies required external disturbances to the motor system and it is not clear whether a similar feedback process addresses internal noise/errors generated by the motor system itself. Here, we introduce a novel analysis that evaluates how variations in the activity of M1 neurons covary with variations in hand motion on a trial-to-trial basis. The analyses demonstrate that M1 activity is correlated with hand motion in both the near future and the recent past, but with opposite polarity. These results suggest that M1 is involved in online feedback motor control to address errors/noise within the motor system.

From the motor cortex to the movement and back again

The motor cortex controls motor behaviors by generating movement-specific signals and transmitting them through spinal cord circuits and motoneurons to the muscles. Precise and well-coordinated muscle activation patterns are necessary for accurate movement execution. Therefore, the activity of cortical neurons should correlate with movement parameters. To investigate the specifics of such correlations among activities of the motor cortex, spinal cord network and muscles, we developed a model for neural control of goal-directed reaching movements that simulates the entire pathway from the motor cortex through spinal cord circuits to the muscles controlling arm movements. In this model, the arm consists of two joints (shoulder and elbow), whose movements are actuated by six muscles (4 single-joint and 2 double-joint flexors and extensors). The muscles provide afferent feedback to the spinal cord circuits. Cortical neurons are defined as cortical "controllers" that solve an inverse problem based on a proposed straight-line trajectory to a target position and a predefined bell-shaped velocity profile. Thus, the controller generates a motor program that produces a task-specific activation of low-level spinal circuits that in turn induce the muscle activation realizing the intended reaching movement. Using the model, we describe the mechanisms of correlation between cortical and motoneuronal activities and movement direction and other movement parameters. We show that the directional modulation of neuronal activity in the motor cortex and the spinal cord may result from direction-specific dynamics of muscle lengths. Our model suggests that directional modulation first emerges at the level of muscle forces, augments at the motoneuron level, and further increases at the level of the motor cortex due to the dependence of frictional forces in the joints, contractility of the muscles and afferent feedback on muscle lengths and/or velocities.

Musculoskeletal geometry accounts for apparent extrinsic representation of paw position in dorsal spinocerebellar tract

Proprioception, the sense of limb position and motion, arises from individual muscle receptors. An important question is how and where in the neuroaxis our high level "extrinsic" sense of limb movement originates. In the 1990s, a series of papers detailed the properties of neurons in the dorsal spinocerebellar tract (DSCT) of the cat. Despite their direct projections from sensory receptors, it appeared that half of these neurons had consistent, high-level tuning to paw position rather than to joint angles (or muscle lengths). These results suggested that many DSCT neurons compute paw position from lower level sensory information. We examined the contribution of musculoskeletal geometry to this apparent extrinsic representation by simulating a three-joint hindlimb with mono- and biarticular muscles, each providing a muscle spindlelike signal, modulated by the muscle length. We simulated neurons driven by randomly weighted combinations of these signals and moved the paw to different positions under two joint-covariance conditions similar to the original experiments. Our results paralleled those experiments in a number of respects: 1) Many neurons were tuned to paw position relative to the hip under both conditions. 2) The distribution of tuning was strongly bimodal, with most neurons driven by whole-leg flexion or extension. 3) The change in tuning between conditions clustered around zero (median absolute change ~20°). These results indicate that, at least for these constraint conditions, extrinsic-like representation can be achieved simply through musculoskeletal geometry and convergent muscle length inputs. Consequently, they suggest a reinterpretation of the earlier results may be required.NEW & NOTEWORTHY A classic experiment concluding that many dorsal spinocerebellar tract neurons encode paw position rather than joint angles has been cited by many studies as evidence for high-level computation occurring within a single synapse of the sensors. However, our study provides evidence that such a computation is not required to explain the results. Using simulation, we replicated many of the original results with purely random connectivity, suggesting that a reinterpretation of the classic experiment is needed.

Enhancing Nervous System Recovery through Neurobiologics, Neural Interface Training, and Neurorehabilitation

After an initial period of recovery, human neurological injury has long been thought to be static. In order to improve quality of life for those suffering from stroke, spinal cord injury, or traumatic brain injury, researchers have been working to restore the nervous system and reduce neurological deficits through a number of mechanisms. For example, neurobiologists have been identifying and manipulating components of the intra- and extracellular milieu to alter the regenerative potential of neurons, neuro-engineers have been producing brain-machine and neural interfaces that circumvent lesions to restore functionality, and neurorehabilitation experts have been developing new ways to revitalize the nervous system even in chronic disease. While each of these areas holds promise, their individual paths to clinical relevance remain difficult. Nonetheless, these methods are now able to synergistically enhance recovery of native motor function to levels which were previously believed to be impossible. Furthermore, such recovery can even persist after training, and for the first time there is evidence of functional axonal regrowth and rewiring in the central nervous system of animal models. To attain this type of regeneration, rehabilitation paradigms that pair cortically-based intent with activation of affected circuits and positive neurofeedback appear to be required-a phenomenon which raises new and far reaching questions about the underlying relationship between conscious action and neural repair. For this reason, we argue that multi-modal therapy will be necessary to facilitate a truly robust recovery, and that the success of investigational microscopic techniques may depend on their integration into macroscopic frameworks that include task-based neurorehabilitation. We further identify critical components of future neural repair strategies and explore the most updated knowledge, progress, and challenges in the fields of cellular neuronal repair, neural interfacing, and neurorehabilitation, all with the goal of better understanding neurological injury and how to improve recovery.

Generalizability of EMG decoding using local field potentials

Motor cortical local field potentials (LFPs) have been successfully used to decode both kinematics and kinetics of arm movement. For future clinically viable prostheses, however, brain activity decoders will have to generalize well under a wide spectrum of behavioral conditions. This property has not yet been demonstrated clearly. Here, we provide evidence for the first time, that an LFP-based electromyogram (EMG) decoder can generalize reasonably well across two different types of behavior. We implanted intracortical microelectrode arrays in the primary motor (M1) and ventral pre-motor (PMv) cortices of a rhesus macaque, and recorded LFP and EMG activity from arm and hand muscles of the contralateral forelimb during a two-dimensional (2-D) centre-out isometric wrist torque task (TT), and during free reach and grasp behavior (FB). Selected temporal and spectral features of the LFP signals were used to train EMG decoders using data from both types of behavior separately. We assessed the decoding performance for both within- and across-task cases. The average achieved generalization score was 65 ± 20%, while in many cases individual scores reached 100%.

Adaptive neuron-to-EMG decoder training for FES neuroprostheses

OBJECTIVE

We have previously demonstrated a brain-machine interface neuroprosthetic system that provided continuous control of functional electrical stimulation (FES) and restoration of grasp in a primate model of spinal cord injury (SCI). Predicting intended EMG directly from cortical recordings provides a flexible high-dimensional control signal for FES. However, no peripheral signal such as force or EMG is available for training EMG decoders in paralyzed individuals.

APPROACH

Here we present a method for training an EMG decoder in the absence of muscle activity recordings; the decoder relies on mapping behaviorally relevant cortical activity to the inferred EMG activity underlying an intended action. Monkeys were trained at a 2D isometric wrist force task to control a computer cursor by applying force in the flexion, extension, ulnar, and radial directions and execute a center-out task. We used a generic muscle force-to-endpoint force model based on muscle pulling directions to relate each target force to an optimal EMG pattern that attained the target force while minimizing overall muscle activity. We trained EMG decoders during the target hold periods using a gradient descent algorithm that compared EMG predictions to optimal EMG patterns.

MAIN RESULTS

We tested this method both offline and online. We quantified both the accuracy of offline force predictions and the ability of a monkey to use these real-time force predictions for closed-loop cursor control. We compared both offline and online results to those obtained with several other direct force decoders, including an optimal decoder computed from concurrently measured neural and force signals.

SIGNIFICANCE

This novel approach to training an adaptive EMG decoder could make a brain-control FES neuroprosthesis an effective tool to restore the hand function of paralyzed individuals. Clinical implementation would make use of individualized EMG-to-force models. Broad generalization could be achieved by including data from multiple grasping tasks in the training of the neuron-to-EMG decoder. Our approach would make it possible for persons with SCI to grasp objects with their own hands, using near-normal motor intent.

Tuning Curves for Arm Posture Control in Motor Cortex Are Consistent with Random Connectivity

Neuronal responses characterized by regular tuning curves are typically assumed to arise from structured synaptic connectivity. However, many responses exhibit both regular and irregular components. To address the relationship between tuning curve properties and underlying circuitry, we analyzed neuronal activity recorded from primary motor cortex (M1) of monkeys performing a 3D arm posture control task and compared the results with a neural network model. Posture control is well suited for examining M1 neuronal tuning because it avoids the dynamic complexity of time-varying movements. As a function of hand position, the neuronal responses have a linear component, as has previously been described, as well as heterogeneous and highly irregular nonlinearities. These nonlinear components involve high spatial frequencies and therefore do not support explicit encoding of movement parameters. Yet both the linear and nonlinear components contribute to the decoding of EMG of major muscles used in the task. Remarkably, despite the presence of a strong linear component, a feedforward neural network model with entirely random connectivity can replicate the data, including both the mean and distributions of the linear and nonlinear components as well as several other features of the neuronal responses. This result shows that smoothness provided by the regularity in the inputs to M1 can impose apparent structure on neural responses, in this case a strong linear (also known as cosine) tuning component, even in the absence of ordered synaptic connectivity.

Brain-state classification and a dual-state decoder dramatically improve the control of cursor movement through a brain-machine interface

OBJECTIVE

It is quite remarkable that brain machine interfaces (BMIs) can be used to control complex movements with fewer than 100 neurons. Success may be due in part to the limited range of dynamical conditions under which most BMIs are tested. Achieving high-quality control that spans these conditions with a single linear mapping will be more challenging. Even for simple reaching movements, existing BMIs must reduce the stochastic noise of neurons by averaging the control signals over time, instead of over the many neurons that normally control movement. This forces a compromise between a decoder with dynamics allowing rapid movement and one that allows postures to be maintained with little jitter. Our current work presents a method for addressing this compromise, which may also generalize to more highly varied dynamical situations, including movements with more greatly varying speed.

APPROACH

We have developed a system that uses two independent Wiener filters as individual components in a single decoder, one optimized for movement, and the other for postural control. We computed an LDA classifier using the same neural inputs. The decoder combined the outputs of the two filters in proportion to the likelihood assigned by the classifier to each state.

MAIN RESULTS

We have performed online experiments with two monkeys using this neural-classifier, dual-state decoder, comparing it to a standard, single-state decoder as well as to a dual-state decoder that switched states automatically based on the cursor's proximity to a target. The performance of both monkeys using the classifier decoder was markedly better than that of the single-state decoder and comparable to the proximity decoder.

SIGNIFICANCE

We have demonstrated a novel strategy for dealing with the need to make rapid movements while also maintaining precise cursor control when approaching and stabilizing within targets. Further gains can undoubtedly be realized by optimizing the performance of the individual movement and posture decoders.

Brain-controlled muscle stimulation for the restoration of motor function

Loss of the ability to move, as a consequence of spinal cord injury or neuromuscular disorder, has devastating consequences for the paralyzed individual, and great economic consequences for society. Functional electrical stimulation (FES) offers one means to restore some mobility to these individuals, improving not only their autonomy, but potentially their general health and well-being as well. FES uses electrical stimulation to cause the paralyzed muscles to contract. Existing clinical systems require the stimulation to be preprogrammed, with the patient typically using residual voluntary movement of another body part to trigger and control the patterned stimulation. The rapid development of neural interfacing in the past decade offers the promise of dramatically improved control for these patients, potentially allowing continuous control of FES through signals recorded from motor cortex, as the patient attempts to control the paralyzed body part. While application of these 'brain-machine interfaces' (BMIs) has undergone dramatic development for control of computer cursors and even robotic limbs, their use as an interface for FES has been much more limited. In this review, we consider both FES and BMI technologies and discuss the prospect for combining the two to provide important new options for paralyzed individuals.

Monte Carlo point process estimation of electromyographic envelopes from motor cortical spikes for brain-machine interfaces

OBJECTIVE

Representation of movement in the motor cortex (M1) has been widely studied in brain-machine interfaces (BMIs). The electromyogram (EMG) has greater bandwidth than the conventional kinematic variables (such as position, velocity), and is functionally related to the discharge of cortical neurons. As the stochastic information of EMG is derived from the explicit spike time structure, point process (PP) methods will be a good solution for decoding EMG directly from neural spike trains. Previous studies usually assume linear or exponential tuning curves between neural firing and EMG, which may not be true.

APPROACH

In our analysis, we estimate the tuning curves in a data-driven way and find both the traditional functional-excitatory and functional-inhibitory neurons, which are widely found across a rat's motor cortex. To accurately decode EMG envelopes from M1 neural spike trains, the Monte Carlo point process (MCPP) method is implemented based on such nonlinear tuning properties.

MAIN RESULTS

Better reconstruction of EMG signals is shown on baseline and extreme high peaks, as our method can better preserve the nonlinearity of the neural tuning during decoding. The MCPP improves the prediction accuracy (the normalized mean squared error) 57% and 66% on average compared with the adaptive point process filter using linear and exponential tuning curves respectively, for all 112 data segments across six rats. Compared to a Wiener filter using spike rates with an optimal window size of 50 ms, MCPP decoding EMG from a point process improves the normalized mean square error (NMSE) by 59% on average.

SIGNIFICANCE

These results suggest that neural tuning is constantly changing during task execution and therefore, the use of spike timing methodologies and estimation of appropriate tuning curves needs to be undertaken for better EMG decoding in motor BMIs.

A training platform for many-dimensional prosthetic devices using a virtual reality environment

Brain machine interfaces (BMIs) have the potential to assist in the rehabilitation of millions of patients worldwide. Despite recent advancements in BMI technology for the restoration of lost motor function, a training environment to restore full control of the anatomical segments of an upper limb extremity has not yet been presented. Here, we develop a virtual upper limb prosthesis with 27 independent dimensions, the anatomical dimensions of the human arm and hand, and deploy the virtual prosthesis as an avatar in a virtual reality environment (VRE) that can be controlled in real-time. The prosthesis avatar accepts kinematic control inputs that can be captured from movements of the arm and hand as well as neural control inputs derived from processed neural signals. We characterize the system performance under kinematic control using a commercially available motion capture system. We also present the performance under kinematic control achieved by two non-human primates (Macaca Mulatta) trained to use the prosthetic avatar to perform reaching and grasping tasks. This is the first virtual prosthetic device that is capable of emulating all the anatomical movements of a healthy upper limb in real-time. Since the system accepts both neural and kinematic inputs for a variety of many-dimensional skeletons, we propose it provides a customizable training platform for the acquisition of many-dimensional neural prosthetic control.

Motor cortex single-neuron and population contributions to compensation for multiple dynamic force fields

To elucidate how primary motor cortex (M1) neurons contribute to the performance of a broad range of different and even incompatible motor skills, we trained two monkeys to perform single-degree-of-freedom elbow flexion/extension movements that could be perturbed by a variety of externally generated force fields. Fields were presented in a pseudorandom sequence of trial blocks. Different computer monitor background colors signaled the nature of the force field throughout each block. There were five different force fields: null field without perturbing torque, assistive and resistive viscous fields proportional to velocity, a resistive elastic force field proportional to position and a resistive viscoelastic field that was the linear combination of the resistive viscous and elastic force fields. After the monkeys were extensively trained in the five field conditions, neural recordings were subsequently made in M1 contralateral to the trained arm. Many caudal M1 neurons altered their activity systematically across most or all of the force fields in a manner that was appropriate to contribute to the compensation for each of the fields. The net activity of the entire sample population likewise provided a predictive signal about the differences in the time course of the external forces encountered during the movements across all force conditions. The neurons showed a broad range of sensitivities to the different fields, and there was little evidence of a modular structure by which subsets of M1 neurons were preferentially activated during movements in specific fields or combinations of fields.

Intentional signal in prefrontal cortex generalizes across different sensory modalities

Biofeedback-EEG training to learn the mental control of an external device (e.g., a cursor on the screen) has been an important paradigm to attempt to understand the involvements of various areas of the brain in the volitional control and the modulation of intentional thought processes. Often the areas to adapt and to monitor progress are selected a priori. Less explored, however, has been the notion of automatically emerging activation in a particular area or subregions within that area recruited above and beyond the rest of the brain. Likewise, the notion of evoking such a signal as an amodal, abstract one remaining robust across different sensory modalities could afford some exploration. Here we develop a simple binary control task in the context of brain-computer interface (BCI) and use a Bayesian sparse probit classification algorithm to automatically uncover brain regional activity that maximizes task performance. We trained and tested 19 participants using the visual modality for instructions and feedback. Across training blocks we quantified coupling of the frontoparietal nodes and selective involvement of visual and auditory regions as a function of the real-time sensory feedback. The testing phase under both forms of sensory feedback revealed automatic recruitment of the prefrontal cortex with a parcellation of higher strength levels in Brodmann's areas 9, 10, and 11 significantly above those in other brain areas. We propose that the prefrontal signal may be a neural correlate of externally driven intended direction and discuss our results in the context of various aspects involved in the cognitive control of our thoughts.

Motor cortical correlates of arm resting in the context of a reaching task and implications for prosthetic control

Prosthetic devices are being developed to restore movement for motor-impaired individuals. A robotic arm can be controlled based on models that relate motor-cortical ensemble activity to kinematic parameters. The models are typically built and validated on data from structured trial periods during which a subject actively performs specific movements, but real-world prosthetic devices will need to operate correctly during rest periods as well. To develop a model of motor cortical modulation during rest, we trained monkeys (Macaca mulatta) to perform a reaching task with their own arm while recording motor-cortical single-unit activity. When a monkey spontaneously put its arm down to rest between trials, our traditional movement decoder produced a nonzero velocity prediction, which would cause undesired motion when applied to a prosthetic arm. During these rest periods, a marked shift was found in individual units' tuning functions. The activity pattern of the whole population during rest (Idle state) was highly distinct from that during reaching movements (Active state), allowing us to predict arm resting from instantaneous firing rates with 98% accuracy using a simple classifier. By cascading this state classifier and the movement decoder, we were able to predict zero velocity correctly, which would avoid undesired motion in a prosthetic application. Interestingly, firing rates during hold periods followed the Active pattern even though hold kinematics were similar to those during rest with near-zero velocity. These findings expand our concept of motor-cortical function by showing that population activity reflects behavioral context in addition to the direct parameters of the movement itself.

Restoring sensorimotor function through intracortical interfaces: progress and looming challenges

The loss of a limb or paralysis resulting from spinal cord injury has devastating consequences on quality of life. One approach to restoring lost sensory and motor abilities in amputees and patients with tetraplegia is to supply them with implants that provide a direct interface with the CNS. Such brain-machine interfaces might enable a patient to exert voluntary control over a prosthetic or robotic limb or over the electrically induced contractions of paralysed muscles. A parallel interface could convey sensory information about the consequences of these movements back to the patient. Recent developments in the algorithms that decode motor intention from neuronal activity and in approaches to convey sensory feedback by electrically stimulating neurons, using biomimetic and adaptation-based approaches, have shown the promise of invasive interfaces with sensorimotor cortices, although substantial challenges remain.

Primary motor cortical discharge during force field adaptation reflects muscle-like dynamics

We often make reaching movements having similar trajectories within very different mechanical environments, for example, with and without an added load in the hand. Under these varying conditions, our kinematic intentions must be transformed into muscle commands that move the limbs. Primary motor cortex (M1) has been implicated in the neural mechanism that mediates this adaptation to new movement dynamics, but our recent experiments suggest otherwise. We have recorded from electrode arrays that were chronically implanted in M1 as monkeys made reaching movements under two different dynamic conditions: the movements were opposed by either a clockwise or counterclockwise velocity-dependent force field acting at the hand. Under these conditions, the preferred direction (PD) of neural discharge for nearly all neurons rotated in the direction of the applied field, as did those of proximal limb electromyograms (EMGs), although the median neural rotation was significantly smaller than that of muscles. For a given neuron, the rotation angle was very consistent, even across multiple sessions. Within the limits of measurement uncertainty, both the neural and EMG changes occurred nearly instantaneously, reaching a steady state despite ongoing behavioral adaptation. Our results suggest that M1 is not directly involved in the adaptive changes that occurred within an experimental session. Rather, most M1 neurons are directly related to the dynamics of muscle activation that themselves reflect the external load. It appears as though gain modulation, the differential recruitment of M1 neurons by higher motor areas, can account for the load and behavioral adaptation-related changes in M1 discharge.

Predictions not commands: active inference in the motor system

The descending projections from motor cortex share many features with top-down or backward connections in visual cortex; for example, corticospinal projections originate in infragranular layers, are highly divergent and (along with descending cortico-cortical projections) target cells expressing NMDA receptors. This is somewhat paradoxical because backward modulatory characteristics would not be expected of driving motor command signals. We resolve this apparent paradox using a functional characterisation of the motor system based on Helmholtz's ideas about perception; namely, that perception is inference on the causes of visual sensations. We explain behaviour in terms of inference on the causes of proprioceptive sensations. This explanation appeals to active inference, in which higher cortical levels send descending proprioceptive predictions, rather than motor commands. This process mirrors perceptual inference in sensory cortex, where descending connections convey predictions, while ascending connections convey prediction errors. The anatomical substrate of this recurrent message passing is a hierarchical system consisting of functionally asymmetric driving (ascending) and modulatory (descending) connections: an arrangement that we show is almost exactly recapitulated in the motor system, in terms of its laminar, topographic and physiological characteristics. This perspective casts classical motor reflexes as minimising prediction errors and may provide a principled explanation for why motor cortex is agranular.

Volitional control of neural activity relies on the natural motor repertoire

BACKGROUND

The results from recent brain-machine interface (BMI) studies suggest that it may be more efficient to use simple arbitrary relationships between individual neuron activity and BMI movements than the complex relationship observed between neuron activity and natural movements. This idea is based on the assumption that individual neurons can be conditioned independently regardless of their natural movement association.

RESULTS

We tested this assumption in the parietal reach region (PRR), an important candidate area for BMIs in which neurons encode the target location for reaching movements. Monkeys could learn to elicit arbitrarily assigned activity patterns, but the seemingly arbitrary patterns always belonged to the response set for natural reaching movements. Moreover, neurons that are free from conditioning showed correlated responses with the conditioned neurons as if they encoded common reach targets. Thus, learning was accomplished by finding reach targets (intrinsic variable of PRR neurons) for which the natural response of reach planning could approximate the arbitrary patterns.

CONCLUSIONS

Our results suggest that animals learn to volitionally control single-neuron activity in PRR by preferentially exploring and exploiting their natural movement repertoire. Thus, for optimal performance, BMIs utilizing neural signals in PRR should harness, not disregard, the activity patterns in the natural sensorimotor repertoire.

Restoration of grasp following paralysis through brain-controlled stimulation of muscles

Patients with spinal cord injury lack the connections between brain and spinal cord circuits essential for voluntary movement. Clinical systems that achieve muscle contraction through functional electrical stimulation (FES) have proven to be effective in allowing patients with tetraplegia to regain control of hand movement and to achieve a greater measure of independence in activities of daily living ^1,2. In typical systems, the patient uses residual proximal limb movements to trigger pre-programmed stimulation that causes the paralyzed muscles to contract, allowing use of one or two basic grasps. Instead, we have developed, in primates, an FES system that is controlled by recordings made from microelectrodes permanently implanted in the brain. We simulated some of the effects of the paralysis caused by C5-C6 spinal cord injury ³ by injecting a local anesthetic to block the median and ulnar nerves at the elbow. Then, using recordings from approximately 100 neurons in the motor cortex, we predicted the intended activity of several of the paralyzed muscles, and used these predictions to control the intensity of stimulation of the same muscles. This process essentially bypassed the spinal cord, restoring to the monkeys voluntary control of their paralyzed muscles. This achievement represents a major advance toward similar restoration of hand function in human patients through brain-controlled FES. We anticipate that in human patients, this neuroprosthesis would allow much more flexible and dexterous use of the hand than is possible with existing FES systems.