Cortical decoding of individual finger and wrist kinematics for an upper-limb neuroprosthesis

Data-Driven Transducer Design and Identification for Internally-Paced Motor Brain Computer Interfaces: A Review

Brain-Computer Interfaces (BCIs) are systems that establish a direct communication pathway between the users' brain activity and external effectors. They offer the potential to improve the quality of life of motor-impaired patients. Motor BCIs aim to permit severely motor-impaired users to regain limb mobility by controlling orthoses or prostheses. In particular, motor BCI systems benefit patients if the decoded actions reflect the users' intentions with an accuracy that enables them to efficiently interact with their environment. One of the main challenges of BCI systems is to adapt the BCI's signal translation blocks to the user to reach a high decoding accuracy. This paper will review the literature of data-driven and user-specific transducer design and identification approaches and it focuses on internally-paced motor BCIs. In particular, continuous kinematic biomimetic and mental-task decoders are reviewed. Furthermore, static and dynamic decoding approaches, linear and non-linear decoding, offline and real-time identification algorithms are considered. The current progress and challenges related to the design of clinical-compatible motor BCI transducers are additionally discussed.

Robust tactile sensory responses in finger area of primate motor cortex relevant to prosthetic control

OBJECTIVE

Challenges in improving the performance of dexterous upper-limb brain-machine interfaces (BMIs) have prompted renewed interest in quantifying the amount and type of sensory information naturally encoded in the primary motor cortex (M1). Previous single unit studies in monkeys showed M1 is responsive to tactile stimulation, as well as passive and active movement of the limbs. However, recent work in this area has focused primarily on proprioception. Here we examined instead how tactile somatosensation of the hand and fingers is represented in M1.

APPROACH

We recorded multi- and single units and thresholded neural activity from macaque M1 while gently brushing individual finger pads at 2 Hz. We also recorded broadband neural activity from electrocorticogram (ECoG) grids placed on human motor cortex, while applying the same tactile stimulus.

MAIN RESULTS

Units displaying significant differences in firing rates between individual fingers (p  <  0.05) represented up to 76.7% of sorted multiunits across four monkeys. After normalizing by the number of channels with significant motor finger responses, the percentage of electrodes with significant tactile responses was 74.9%  ±  24.7%. No somatotopic organization of finger preference was obvious across cortex, but many units exhibited cosine-like tuning across multiple digits. Sufficient sensory information was present in M1 to correctly decode stimulus position from multiunit activity above chance levels in all monkeys, and also from ECoG gamma power in two human subjects.

SIGNIFICANCE

These results provide some explanation for difficulties experienced by motor decoders in clinical trials of cortically controlled prosthetic hands, as well as the general problem of disentangling motor and sensory signals in primate motor cortex during dextrous tasks. Additionally, examination of unit tuning during tactile and proprioceptive inputs indicates cells are often tuned differently in different contexts, reinforcing the need for continued refinement of BMI training and decoding approaches to closed-loop BMI systems for dexterous grasping.

Neural control of finger movement via intracortical brain-machine interface

OBJECTIVE

Intracortical brain-machine interfaces (BMIs) are a promising source of prosthesis control signals for individuals with severe motor disabilities. Previous BMI studies have primarily focused on predicting and controlling whole-arm movements; precise control of hand kinematics, however, has not been fully demonstrated. Here, we investigate the continuous decoding of precise finger movements in rhesus macaques.

APPROACH

In order to elicit precise and repeatable finger movements, we have developed a novel behavioral task paradigm which requires the subject to acquire virtual fingertip position targets. In the physical control condition, four rhesus macaques performed this task by moving all four fingers together in order to acquire a single target. This movement was equivalent to controlling the aperture of a power grasp. During this task performance, we recorded neural spikes from intracortical electrode arrays in primary motor cortex.

MAIN RESULTS

Using a standard Kalman filter, we could reconstruct continuous finger movement offline with an average correlation of ρ  =  0.78 between actual and predicted position across four rhesus macaques. For two of the monkeys, this movement prediction was performed in real-time to enable direct brain control of the virtual hand. Compared to physical control, neural control performance was slightly degraded; however, the monkeys were still able to successfully perform the task with an average target acquisition rate of 83.1%. The monkeys' ability to arbitrarily specify fingertip position was also quantified using an information throughput metric. During brain control task performance, the monkeys achieved an average 1.01 bits s-1 throughput, similar to that achieved in previous studies which decoded upper-arm movements to control computer cursors using a standard Kalman filter.

SIGNIFICANCE

This is, to our knowledge, the first demonstration of brain control of finger-level fine motor skills. We believe that these results represent an important step towards full and dexterous control of neural prosthetic devices.

Decoding Information for Grasping from the Macaque Dorsomedial Visual Stream

Neurodecoders have been developed by researchers mostly to control neuroprosthetic devices, but also to shed new light on neural functions. In this study, we show that signals representing grip configurations can be reliably decoded from neural data acquired from area V6A of the monkey medial posterior parietal cortex. Two Macaca fascicularis monkeys were trained to perform an instructed-delay reach-to-grasp task in the dark and in the light toward objects of different shapes. Population neural activity was extracted at various time intervals on vision of the objects, the delay before movement, and grasp execution. This activity was used to train and validate a Bayes classifier used for decoding objects and grip types. Recognition rates were well over chance level for all the epochs analyzed in this study. Furthermore, we detected slightly different decoding accuracies, depending on the task's visual condition. Generalization analysis was performed by training and testing the system during different time intervals. This analysis demonstrated that a change of code occurred during the course of the task. Our classifier was able to discriminate grasp types fairly well in advance with respect to grasping onset. This feature might be important when the timing is critical to send signals to external devices before the movement start. Our results suggest that the neural signals from the dorsomedial visual pathway can be a good substrate to feed neural prostheses for prehensile actions.SIGNIFICANCE STATEMENT Recordings of neural activity from nonhuman primate frontal and parietal cortex have led to the development of methods of decoding movement information to restore coordinated arm actions in paralyzed human beings. Our results show that the signals measured from the monkey medial posterior parietal cortex are valid for correctly decoding information relevant for grasping. Together with previous studies on decoding reach trajectories from the medial posterior parietal cortex, this highlights the medial parietal cortex as a target site for transforming neural activity into control signals to command prostheses to allow human patients to dexterously perform grasping actions.

Estimating Fast Neural Input Using Anatomical and Functional Connectivity

In the last 20 years there has been an increased interest in estimating signals that are sent between neurons and brain areas. During this time many new methods have appeared for measuring those signals. Here we review a wide range of methods for which connected neurons can be identified anatomically, by tracing axons that run between the cells, or functionally, by detecting if the activity of two neurons are correlated with a short lag. The signals that are sent between the neurons are represented by the activity in the neurons that are connected to the target population or by the activity at the corresponding synapses. The different methods not only differ in the accuracy of the signal measurement but they also differ in the type of signal being measured. For example, unselective recording of all neurons in the source population encompasses more indirect pathways to the target population than if one selectively record from the neurons that project to the target population. Infact, this degree of selectivity is similar to that of optogenetic perturbations; one can perturb selectively or unselectively. Thus it becomes possible to match a given signal measurement method with a signal perturbation method, something that allows for an exact input control to any neuronal population.

Neostriatal Neuronal Activity Correlates Better with Movement Kinematics under Certain Rewards

This study investigated how the activity of neostriatal neurons is related to the kinematics of movement when monkeys performed visually and vibratory cued wrist extensions and flexions. Single-unit recordings of 142/236 neostriatal neurons showed pre-movement activity (PMA) in a reaction time task with unpredictable reward. Monkeys were pseudo-randomly (75%) rewarded for correct performance. A regression model was used to determine whether the correlation between neostriatal neuronal activity and the kinematic variables (position, velocity, and acceleration) of wrist movement changes as a function of reward contingency, sensory cues, and movement direction. The coefficients of determination (CoD) representing the proportion of the variance in neuronal activity explained by the regression model on a trial by trial basis, together with their temporal occurrences (time of best regression/correlation, ToC) were compared across sensory modality, movement direction, and reward contingency. The best relationship (correlation) between neuronal activity and movement kinematic variables, given by the average coefficient of determination (CoD), was: (a) greater during trials in which rewards were certain, called "A" trials, as compared with those in which reward was uncertain called ("R") trials, (b) greater during flexion (Flex) trials as compared with extension (Ext) trials, and (c) greater during visual (VIS) cued trials than during vibratory (VIB) cued trials, for the same type of trial and the same movement direction. These results are consistent with the hypothesis that predictability of reward for correct performance is accompanied by faster linkage between neostriatal PMA and the vigor of wrist movement kinematics. Furthermore, the results provide valuable insights for building an upper-limb neuroprosthesis.

Decoding repetitive finger movements with brain activity acquired via non-invasive electroencephalography

We investigated how well repetitive finger tapping movements can be decoded from scalp electroencephalography (EEG) signals. A linear decoder with memory was used to infer continuous index finger angular velocities from the low-pass filtered fluctuations of the amplitude of a plurality of EEG signals distributed across the scalp. To evaluate the accuracy of the decoder, the Pearson's correlation coefficient (r) between the observed and predicted trajectories was calculated in a 10-fold cross-validation scheme. We also assessed attempts to decode finger kinematics from EEG data that was cleaned with independent component analysis (ICA), EEG data from peripheral sensors, and EEG data from rest periods. A genetic algorithm (GA) was used to select combinations of EEG channels that maximized decoding accuracies. Our results (lower quartile r = 0.18, median r = 0.36, upper quartile r = 0.50) show that delta-band EEG signals contain useful information that can be used to infer finger kinematics. Further, the highest decoding accuracies were characterized by highly correlated delta band EEG activity mostly localized to the contralateral central areas of the scalp. Spectral analysis of EEG also showed bilateral alpha band (8–13 Hz) event related desynchronizations (ERDs) and contralateral beta band (20–30 Hz) event related synchronizations (ERSs) localized over central scalp areas. Overall, this study demonstrates the feasibility of decoding finger kinematics from scalp EEG signals.

Electrocorticographic decoding of ipsilateral reach in the setting of contralateral arm weakness from a cortical lesion

Brain machine interfaces have the potential for restoring motor function not only in patients with amputations or lesions of efferent pathways in the spinal cord and peripheral nerves, but also patients with acquired brain lesions such as strokes and tumors. In these patients the most efficient components of cortical motor systems are not available for BMI control. Here we had the opportunity to investigate the possibility of utilizing subdural electrocorticographic (ECoG) signals to control natural reaching movements under these circumstances. In a subject with a left arm monoparesis following resection of a recurrent glioma, we found that ECoG signals recorded in remaining cortex were sufficient for decoding kinematics of natural reach movements of the nonparetic arm, ipsilateral to the ECoG recordings. The relationship between the subject's ECoG signals and reach trajectory in three dimensions, two of which were highly correlated, was captured with a computationally simple linear model (mean Pearson's r in depth dimension= 0.68, in height= 0.73, in lateral= 0.24). These results were attained with only a small subset of 7 temporal/spectral neural signal features. The small subset of neural features necessary to attain high decoding results show promise for a restorative BMI controlled solely by ipsilateral ECoG signals.

State-based decoding of hand and finger kinematics using neuronal ensemble and LFP activity during dexterous reach-to-grasp movements

The performance of brain-machine interfaces (BMIs) that continuously control upper limb neuroprostheses may benefit from distinguishing periods of posture and movement so as to prevent inappropriate movement of the prosthesis. Few studies, however, have investigated how decoding behavioral states and detecting the transitions between posture and movement could be used autonomously to trigger a kinematic decoder. We recorded simultaneous neuronal ensemble and local field potential (LFP) activity from microelectrode arrays in primary motor cortex (M1) and dorsal (PMd) and ventral (PMv) premotor areas of two male rhesus monkeys performing a center-out reach-and-grasp task, while upper limb kinematics were tracked with a motion capture system with markers on the dorsal aspect of the forearm, hand, and fingers. A state decoder was trained to distinguish four behavioral states (baseline, reaction, movement, hold), while a kinematic decoder was trained to continuously decode hand end point position and 18 joint angles of the wrist and fingers. LFP amplitude most accurately predicted transition into the reaction (62%) and movement (73%) states, while spikes most accurately decoded arm, hand, and finger kinematics during movement. Using an LFP-based state decoder to trigger a spike-based kinematic decoder [r = 0.72, root mean squared error (RMSE) = 0.15] significantly improved decoding of reach-to-grasp movements from baseline to final hold, compared with either a spike-based state decoder combined with a spike-based kinematic decoder (r = 0.70, RMSE = 0.17) or a spike-based kinematic decoder alone (r = 0.67, RMSE = 0.17). Combining LFP-based state decoding with spike-based kinematic decoding may be a valuable step toward the realization of BMI control of a multifingered neuroprosthesis performing dexterous manipulation.

Decoding dexterous finger movements in a neural prosthesis model approaching real-world conditions

Dexterous finger movements can be decoded from neuronal action potentials acquired from a nonhuman primate using a chronically implanted Utah Electrode Array. We have developed an algorithm that can, after training, detect and classify individual and combined finger movements without any a priori knowledge of the data, task, or behavior. The algorithm is based on changes in the firing rates of individual neurons that are tuned for one or more finger movement types. Nine different movement types, which consisted of individual flexions, individual extensions, and combined flexions of the thumb, index finger, and middle finger, were decoded. The algorithm performed reliably on data recorded continuously during movement tasks, including a no-movement state, with an overall average sensitivity and specificity that were both > 92%. These results demonstrate a viable algorithm for decoding dexterous finger movements under conditions similar to those required for a real-world neural prosthetic application.

Closing the loop for memory prosthesis: detecting the role of hippocampal neural ensembles using nonlinear models

A major factor involved in providing closed loop feedback for control of neural function is to understand how neural ensembles encode online information critical to the final behavioral endpoint. This issue was directly assessed in rats performing a short-term delay memory task in which successful encoding of task information is dependent upon specific spatio-temporal firing patterns recorded from ensembles of CA3 and CA1 hippocampal neurons. Such patterns, extracted by a specially designed nonlinear multi-input multi-output (MIMO) nonlinear mathematical model, were used to predict successful performance online via a closed loop paradigm which regulated trial difficulty (time of retention) as a function of the "strength" of stimulus encoding. The significance of the MIMO model as a neural prosthesis has been demonstrated by substituting trains of electrical stimulation pulses to mimic these same ensemble firing patterns. This feature was used repeatedly to vary "normal" encoding as a means of understanding how neural ensembles can be "tuned" to mimic the inherent process of selecting codes of different strength and functional specificity. The capacity to enhance and tune hippocampal encoding via MIMO model detection and insertion of critical ensemble firing patterns shown here provides the basis for possible extension to other disrupted brain circuitry.

Optimizing the decoding of movement goals from local field potentials in macaque cortex

The successful development of motor neuroprosthetic devices hinges on the ability to accurately and reliably decode signals from the brain. Motor neuroprostheses are widely investigated in behaving non-human primates, but technical constraints have limited progress in optimizing performance. In particular, the organization of movement-related neuronal activity across cortical layers remains poorly understood due, in part, to the widespread use of fixed-geometry multielectrode arrays. In this study, we use chronically implanted multielectrode arrays with individually movable electrodes to examine how the encoding of movement goals depends on cortical depth. In a series of recordings spanning several months, we varied the depth of each electrode in the prearcuate gyrus of frontal cortex in two monkeys as they performed memory-guided eye movements. We decode eye movement goals from local field potentials (LFPs) and multiunit spiking activity recorded across a range of depths up to 3 mm from the cortical surface. We show that both LFP and multiunit signals yield the highest decoding performance at superficial sites, within 0.5 mm of the cortical surface, while performance degrades substantially at sites deeper than 1 mm. We also analyze performance by varying bandpass filtering characteristics and simulating changes in microelectrode array channel count and density. The results indicate that the performance of LFP-based neuroprostheses strongly depends on recording configuration and that recording depth is a critical parameter limiting system performance.

The functional consequences of chronic, physiologically effective intracortical microstimulation

Many studies have demonstrated the ability of chronically implanted multielectrode arrays (MEAs) to extract information from the motor cortex of both humans and nonhuman primates. Similarly, many studies have shown the ability of intracortical microstimulation to impart information to the brain via a single or a few electrodes acutely implanted in sensory cortex of nonhuman primates, but relatively few microstimulation studies characterizing chronically implanted MEAs have been performed. Additionally, device and tissue damage have been reported at the levels of microstimulation used in these studies. Whether the damage resulting from microstimulation impairs the ability of MEAs to chronically produce physiological effects, however, has not been directly tested. In this study, we examined the functional consequences of multiple months of periodic microstimulation via chronically implanted MEAs at levels capable of evoking physiological responses, that is, electromyogram (EMG) activity. The functionality of the MEA and neural tissue was determined by measuring impedances, the ability of microstimulation to evoke EMG responses, and the recording of action potentials. We found that impedances and the number of recorded action potentials followed the previously reported trend of decreasing over time in both animals that received microstimulation and those which did not receive microstimulation. Despite these trends, the ability to evoke EMG responses and record action potentials was retained throughout the study. The results of this study suggest that intracortical microstimulation via MEAs did not cause functional failure, suggesting that MEA-based microstimulation is ready to transition into subchronic (< 30 days) human trials to determine whether complex spatiotemporal sensory percepts can be evoked by patterned microstimulation.

Single trial discrimination of individual finger movements on one hand: a combined MEG and EEG study

It is crucial to understand what brain signals can be decoded from single trials with different recording techniques for the development of Brain-Machine Interfaces. A specific challenge for non-invasive recording methods are activations confined to small spatial areas on the cortex such as the finger representation of one hand. Here we study the information content of single trial brain activity in non-invasive MEG and EEG recordings elicited by finger movements of one hand. We investigate the feasibility of decoding which of four fingers of one hand performed a slight button press. With MEG we demonstrate reliable discrimination of single button presses performed with the thumb, the index, the middle or the little finger (average over all subjects and fingers 57%, best subject 70%, empirical guessing level: 25.1%). EEG decoding performance was less robust (average over all subjects and fingers 43%, best subject 54%, empirical guessing level 25.1%). Spatiotemporal patterns of amplitude variations in the time series provided best information for discriminating finger movements. Non-phase-locked changes of mu and beta oscillations were less predictive. Movement related high gamma oscillations were observed in average induced oscillation amplitudes in the MEG but did not provide sufficient information about the finger's identity in single trials. Importantly, pre-movement neuronal activity provided information about the preparation of the movement of a specific finger. Our study demonstrates the potential of non-invasive MEG to provide informative features for individual finger control in a Brain-Machine Interface neuroprosthesis.

Grasp movement decoding from premotor and parietal cortex

Despite recent advances in harnessing cortical motor-related activity to control computer cursors and robotic devices, the ability to decode and execute different grasping patterns remains a major obstacle. Here we demonstrate a simple Bayesian decoder for real-time classification of grip type and wrist orientation in macaque monkeys that uses higher-order planning signals from anterior intraparietal cortex (AIP) and ventral premotor cortex (area F5). Real-time decoding was based on multiunit signals, which had similar tuning properties to cells in previous single-unit recording studies. Maximum decoding accuracy for two grasp types (power and precision grip) and five wrist orientations was 63% (chance level, 10%). Analysis of decoder performance showed that grip type decoding was highly accurate (90.6%), with most errors occurring during orientation classification. In a subsequent off-line analysis, we found small but significant performance improvements (mean, 6.25 percentage points) when using an optimized spike-sorting method (superparamagnetic clustering). Furthermore, we observed significant differences in the contributions of F5 and AIP for grasp decoding, with F5 being better suited for classification of the grip type and AIP contributing more toward decoding of object orientation. However, optimum decoding performance was maximal when using neural activity simultaneously from both areas. Overall, these results highlight quantitative differences in the functional representation of grasp movements in AIP and F5 and represent a first step toward using these signals for developing functional neural interfaces for hand grasping.

Encoding of coordinated grasp trajectories in primary motor cortex

Few studies have investigated how the cortex encodes the preshaping of the hand as an object is grasped, an ethological movement referred to as prehension. We developed an encoding model of hand kinematics to test whether primary motor cortex (MI) neurons encode temporally extensive combinations of joint motions that characterize a prehensile movement. Two female rhesus macaque monkeys were trained to grasp 4 different objects presented by a robot while their arm was held in place by a thermoplastic brace. We used multielectrode arrays to record MI neurons and an infrared camera motion tracking system to record the 3-D positions of 14 markers placed on the monkeys' wrist and digits. A generalized linear model framework was used to predict the firing rate of each neuron in a 4 ms time interval, based on its own spiking history and the spatiotemporal kinematics of the joint angles of the hand. Our results show that the variability of the firing rate of MI neurons is better described by temporally extensive combinations of finger and wrist joint angle kinematics rather than any individual joint motion or any combination of static kinematic parameters at their optimal lag. Moreover, a higher percentage of neurons encoded joint angular velocities than joint angular positions. These results suggest that neurons encode the covarying trajectories of the hand's joints during a prehensile movement.