Watch, Imagine, Attempt: Motor Cortex Single-Unit Activity Reveals Context-Dependent Movement Encoding in Humans With Tetraplegia

Machine learning decoding of single neurons in the thalamus for speech brain-machine interfaces

Objective. Our goal is to decode firing patterns of single neurons in the left ventralis intermediate nucleus (Vim) of the thalamus, related to speech production, perception, and imagery. For realistic speech brain-machine interfaces (BMIs), we aim to characterize the amount of thalamic neurons necessary for high accuracy decoding.Approach. We intraoperatively recorded single neuron activity in the left Vim of eight neurosurgical patients undergoing implantation of deep brain stimulator or RF lesioning during production, perception and imagery of the five monophthongal vowel sounds. We utilized the Spade decoder, a machine learning algorithm that dynamically learns specific features of firing patterns and is based on sparse decomposition of the high dimensional feature space.Main results. Spade outperformed all algorithms compared with, for all three aspects of speech: production, perception and imagery, and obtained accuracies of 100%, 96%, and 92%, respectively (chance level: 20%) based on pooling together neurons across all patients. The accuracy was logarithmic in the amount of neurons for all three aspects of speech. Regardless of the amount of units employed, production gained highest accuracies, whereas perception and imagery equated with each other.Significance. Our research renders single neuron activity in the left Vim a promising source of inputs to BMIs for restoration of speech faculties for locked-in patients or patients with anarthria or dysarthria to allow them to communicate again. Our characterization of how many neurons are necessary to achieve a certain decoding accuracy is of utmost importance for planning BMI implantation.

Motor cortex retains and reorients neural dynamics during motor imagery

The most prominent characteristic of motor cortex is its activation during movement execution, but it is also active when we simply imagine movements in the absence of actual motor output. Despite decades of behavioural and imaging studies, it is unknown how the specific activity patterns and temporal dynamics in motor cortex during covert motor imagery relate to those during motor execution. Here we recorded intracortical activity from the motor cortex of two people who retain some residual wrist function following incomplete spinal cord injury as they performed both actual and imagined isometric wrist extensions. We found that we could decompose the population activity into three orthogonal subspaces, where one was similarly active during both action and imagery, and the others were active only during a single task type-action or imagery. Although they inhabited orthogonal neural dimensions, the action-unique and imagery-unique subspaces contained a strikingly similar set of dynamic features. Our results suggest that during motor imagery, motor cortex maintains the same overall population dynamics as during execution by reorienting the components related to motor output and/or feedback into a unique, output-null imagery subspace.

An Investigation of Manifold-Based Direct Control for a Brain-to-Body Neural Bypass

Objective: Brain-body interfaces (BBIs) have emerged as a very promising solution for restoring voluntary hand control in people with upper-limb paralysis. The BBI module decoding motor commands from brain signals should provide the user with intuitive, accurate, and stable control. Here, we present a preliminary investigation in a monkey of a brain decoding strategy based on the direct coupling between the activity of intrinsic neural ensembles and output variables, aiming at achieving ease of learning and long-term robustness. Results: We identified an intrinsic low-dimensional space (called manifold) capturing the co-variation patterns of the monkey's neural activity associated to reach-to-grasp movements. We then tested the animal's ability to directly control a computer cursor using cortical activation along the manifold axes. By daily recalibrating only scaling factors, we achieved rapid learning and stable high performance in simple, incremental 2D tasks over more than 12 weeks of experiments. Finally, we showed that this brain decoding strategy can be effectively coupled to peripheral nerve stimulation to trigger voluntary hand movements. Conclusions: These results represent a proof of concept of manifold-based direct control for BBI applications.

Motor cortex retains and reorients neural dynamics during motor imagery

The most prominent role of motor cortex is generating patterns of neural activity that lead to movement, but it is also active when we simply imagine movements in the absence of actual motor output. Despite decades of behavioral and imaging studies, it is unknown how the specific activity patterns and temporal dynamics within motor cortex during covert motor imagery relate to those during motor execution. Here we recorded intracortical activity from the motor cortex of two people with residual wrist function following incomplete spinal cord injury as they performed both actual and imagined isometric wrist extensions. We found that we could decompose the population-level activity into orthogonal subspaces such that one set of components was similarly active during both action and imagery, and others were only active during a single task typeâ€"action or imagery. Although they inhabited orthogonal neural dimensions, the action-unique and imagery-unique subspaces contained a strikingly similar set of dynamical features. Our results suggest that during motor imagery, motor cortex maintains the same overall population dynamics as during execution by recreating the missing components related to motor output and/or feedback within a unique imagery-only subspace.

Neuromotor prosthetic to treat stroke-related paresis: N-of-1 trial

Background

Functional recovery of arm movement typically plateaus following a stroke, leaving chronic motor deficits. Brain-computer interfaces (BCI) may be a potential treatment for post-stroke deficits

Methods

In this n-of-1 trial (NCT03913286), a person with chronic subcortical stroke with upper-limb motor impairment used a powered elbow-wrist-hand orthosis that opened and closed the affected hand using cortical activity, recorded from a percutaneous BCI comprised of four microelectrode arrays implanted in the ipsilesional precentral gyrus, based on decoding of spiking patterns and high frequency field potentials generated by imagined hand movements. The system was evaluated in a home setting for 12 weeks

Results

Robust single unit activity, modulating with attempted or imagined movement, was present throughout the precentral gyrus. The participant acquired voluntary control over a hand-orthosis, achieving 10 points on the Action Research Arm Test using the BCI, compared to 0 without any device, and 5 using myoelectric control. Strength, spasticity, the Fugl-Meyer scores improved.

Conclusions

We demonstrate in a human being that ensembles of individual neurons in the cortex overlying a chronic supratentorial, subcortical stroke remain active and engaged in motor representation and planning and can be used to electrically bypass the stroke and promote limb function. The participant’s ability to rapidly acquire control over otherwise paralyzed hand opening, more than 18 months after a stroke, may justify development of a fully implanted movement restoration system to expand the utility of fully implantable BCI to a clinical population that numbers in the tens of millions worldwide.

Stroke is a restriction of blood flow to part of the brain and can lead to chronic issues with a person’s ability to control the limbs. The aim of this study was to see if a new type of device could restore movement in a person with arm weakness due to a stroke that occurred a year earlier. In our trial, a sensor was implanted into the surface of the brain, near the site of the stroke, and was connected to a computer that generated a command to open and close the hand with a motorized brace worn on the hand. This person was able to use their own brain activity to trigger the brace and pick up and move objects. This research could support the development of similar medical devices to restore movement in people who have had strokes.

Serruya et al. test in an N-of-1 trial whether a wearable, powered exoskeletal orthosis, driven by a percutaneous, implanted brain–computer interface can restore voluntary upper extremity function following chronic hemiparesis subsequent to a cerebral subcortical stroke. Using this approach, voluntary opening of the paralyzed hand is restored.

Spelling interface using intracortical signals in a completely locked-in patient enabled via auditory neurofeedback training

Patients with amyotrophic lateral sclerosis (ALS) can lose all muscle-based routes of communication as motor neuron degeneration progresses, and ultimately, they may be left without any means of communication. While others have evaluated communication in people with remaining muscle control, to the best of our knowledge, it is not known whether neural-based communication remains possible in a completely locked-in state. Here, we implanted two 64 microelectrode arrays in the supplementary and primary motor cortex of a patient in a completely locked-in state with ALS. The patient modulated neural firing rates based on auditory feedback and he used this strategy to select letters one at a time to form words and phrases to communicate his needs and experiences. This case study provides evidence that brain-based volitional communication is possible even in a completely locked-in state.

Feel Your Reach: An EEG-Based Framework to Continuously Detect Goal-Directed Movements and Error Processing to Gate Kinesthetic Feedback Informed Artificial Arm Control

Establishing the basic knowledge, methodology, and technology for a framework for the continuous decoding of hand/arm movement intention was the aim of the ERC-funded project “Feel Your Reach”. In this work, we review the studies and methods we performed and implemented in the last 6 years, which build the basis for enabling severely paralyzed people to non-invasively control a robotic arm in real-time from electroencephalogram (EEG). In detail, we investigated goal-directed movement detection, decoding of executed and attempted movement trajectories, grasping correlates, error processing, and kinesthetic feedback. Although we have tested some of our approaches already with the target populations, we still need to transfer the “Feel Your Reach” framework to people with cervical spinal cord injury and evaluate the decoders’ performance while participants attempt to perform upper-limb movements. While on the one hand, we made major progress towards this ambitious goal, we also critically discuss current limitations.

Human brain mapping with multithousand-channel PtNRGrids resolves spatiotemporal dynamics

Electrophysiological devices are critical for mapping eloquent and diseased brain regions and for therapeutic neuromodulation in clinical settings and are extensively used for research in brain-machine interfaces. However, the existing clinical and experimental devices are often limited in either spatial resolution or cortical coverage. Here, we developed scalable manufacturing processes with a dense electrical connection scheme to achieve reconfigurable thin-film, multithousand-channel neurophysiological recording grids using platinum nanorods (PtNRGrids). With PtNRGrids, we have achieved a multithousand-channel array of small (30 μm) contacts with low impedance, providing high spatial and temporal resolution over a large cortical area. We demonstrated that PtNRGrids can resolve submillimeter functional organization of the barrel cortex in anesthetized rats that captured the tissue structure. In the clinical setting, PtNRGrids resolved fine, complex temporal dynamics from the cortical surface in an awake human patient performing grasping tasks. In addition, the PtNRGrids identified the spatial spread and dynamics of epileptic discharges in a patient undergoing epilepsy surgery at 1-mm spatial resolution, including activity induced by direct electrical stimulation. Collectively, these findings demonstrated the power of the PtNRGrids to transform clinical mapping and research with brain-machine interfaces.

Functional Characterization of Human Pluripotent Stem Cell-Derived Models of the Brain with Microelectrode Arrays

Human pluripotent stem cell (hPSC)-derived neuron cultures have emerged as models of electrical activity in the human brain. Microelectrode arrays (MEAs) measure changes in the extracellular electric potential of cell cultures or tissues and enable the recording of neuronal network activity. MEAs have been applied to both human subjects and hPSC-derived brain models. Here, we review the literature on the functional characterization of hPSC-derived two- and three-dimensional brain models with MEAs and examine their network function in physiological and pathological contexts. We also summarize MEA results from the human brain and compare them to the literature on MEA recordings of hPSC-derived brain models. MEA recordings have shown network activity in two-dimensional hPSC-derived brain models that is comparable to the human brain and revealed pathology-associated changes in disease models. Three-dimensional hPSC-derived models such as brain organoids possess a more relevant microenvironment, tissue architecture and potential for modeling the network activity with more complexity than two-dimensional models. hPSC-derived brain models recapitulate many aspects of network function in the human brain and provide valid disease models, but certain advancements in differentiation methods, bioengineering and available MEA technology are needed for these approaches to reach their full potential.

Machine learning algorithm for decoding multiple subthalamic spike trains for speech brain-machine interfaces

Objective. The goal of this study is to decode the electrical activity of single neurons in the human subthalamic nucleus (STN) to infer the speech features that a person articulated, heard or imagined. We also aim to evaluate the amount of subthalamic neurons required for high accuracy decoding suitable for real-life speech brain-machine interfaces (BMI).Approach. We intraoperatively recorded single-neuron activity in the STN of 21 neurosurgical patients with Parkinson's disease undergoing implantation of deep brain stimulator while patients produced, perceived or imagined the five monophthongal vowel sounds. Our decoder is based on machine learning algorithms that dynamically learn specific features of the speech-related firing patterns.Main results. In an extensive comparison of algorithms, our sparse decoder ('SpaDe'), based on sparse decomposition of the high dimensional neuronal feature space, outperformed the other algorithms in all three conditions: production, perception and imagery. For speech production, our algorithm, Spade, predicted all vowels correctly (accuracy: 100%; chance level: 20%). For perception accuracy was 96%, and for imagery: 88%. The accuracy of Spade showed a linear behavior in the amount of neurons for the perception data, and even faster for production or imagery.Significance. Our study demonstrates that the information encoded by single neurons in the STN about the production, perception and imagery of speech is suitable for high-accuracy decoding. It is therefore an important step towards BMIs for restoration of speech faculties that bears an enormous potential to alleviate the suffering of completely paralyzed ('locked-in') patients and allow them to communicate again with their environment. Moreover, our research indicates how many subthalamic neurons may be necessary to achieve each level of decoding accuracy, which is of supreme importance for a neurosurgeon planning the implantation of a speech BMI.

Intracortical Microelectrode Array Unit Yield under Chronic Conditions: A Comparative Evaluation

While microelectrode arrays (MEAs) offer the promise of elucidating functional neural circuitry and serve as the basis for a cortical neuroprosthesis, the challenge of designing and demonstrating chronically reliable technology remains. Numerous studies report “chronic” data but the actual time spans and performance measures corresponding to the experimental work vary. In this study, we reviewed the experimental durations that constitute chronic studies across a range of MEA types and animal species to gain an understanding of the widespread variability in reported study duration. For rodents, which are the most commonly used animal model in chronic studies, we examined active electrode yield (AEY) for different array types as a means to contextualize the study duration variance, as well as investigate and interpret the performance of custom devices in comparison to conventional MEAs. We observed wide-spread variance within species for the chronic implantation period and an AEY that decayed linearly in rodent models that implanted commercially-available devices. These observations provide a benchmark for comparing the performance of new technologies and highlight the need for consistency in chronic MEA studies. Additionally, to fully derive performance under chronic conditions, the duration of abiotic failure modes, biological processes induced by indwelling probes, and intended application of the device are key determinants.

A novel paired associative stimulation protocol with a high-frequency peripheral component: A review on results in spinal cord injury rehabilitation

In recent decades, a multitude of therapeutic approaches has been developed for spinal cord injury (SCI), but few have progressed to regular clinical practice. Novel non-invasive, cost-effective, and feasible approaches to treat this challenging condition are needed. A novel variant of paired associative stimulation (PAS), high-PAS, consists of non-invasive high-intensity transcranial magnetic stimulation (TMS) and non-invasive high-frequency electrical peripheral nerve stimulation (PNS). We observed a therapeutic effect of high-PAS in 20 patients with incomplete SCI with wide range of injury severity, age, and time since injury. Tetraplegic and paraplegic, traumatic, and neurological SCI patients benefited from upper- or lower-limb high-PAS. We observed increases in manual motor scores (MMT) of upper and lower limbs, functional hand tests, walking tests, and measures of functional independence. We also optimized PAS settings in several studies in healthy subjects and began elucidating the mechanisms of therapeutic action. The scope of this review is to describe the clinical experience gained with this novel PAS approach. This review is focused on the summary of our results and observations and the methodological considerations for researchers and clinicians interested in adopting and further developing this new method.

The Neural Representation of Force across Grasp Types in Motor Cortex of Humans with Tetraplegia

Intracortical brain-computer interfaces (iBCIs) have the potential to restore hand grasping and object interaction to individuals with tetraplegia. Optimal grasping and object interaction require simultaneous production of both force and grasp outputs. However, since overlapping neural populations are modulated by both parameters, grasp type could affect how well forces are decoded from motor cortex in a closed-loop force iBCI. Therefore, this work quantified the neural representation and offline decoding performance of discrete hand grasps and force levels in two human participants with tetraplegia. Participants attempted to produce three discrete forces (light, medium, hard) using up to five hand grasp configurations. A two-way Welch ANOVA was implemented on multiunit neural features to assess their modulation to force and grasp Demixed principal component analysis (dPCA) was used to assess for population-level tuning to force and grasp and to predict these parameters from neural activity. Three major findings emerged from this work: (1) force information was neurally represented and could be decoded across multiple hand grasps (and, in one participant, across attempted elbow extension as well); (2) grasp type affected force representation within multiunit neural features and offline force classification accuracy; and (3) grasp was classified more accurately and had greater population-level representation than force. These findings suggest that force and grasp have both independent and interacting representations within cortex, and that incorporating force control into real-time iBCI systems is feasible across multiple hand grasps if the decoder also accounts for grasp type.

Auditory cues reveal intended movement information in middle frontal gyrus neuronal ensemble activity of a person with tetraplegia

Intracortical brain-computer interfaces (iBCIs) allow people with paralysis to directly control assistive devices using neural activity associated with the intent to move. Realizing the full potential of iBCIs critically depends on continued progress in understanding how different cortical areas contribute to movement control. Here we present the first comparison between neuronal ensemble recordings from the left middle frontal gyrus (MFG) and precentral gyrus (PCG) of a person with tetraplegia using an iBCI. As expected, PCG was more engaged in selecting and generating intended movements than in earlier perceptual stages of action planning. By contrast, MFG displayed movement-related information during the sensorimotor processing steps preceding the appearance of the action plan in PCG, but only when the actions were instructed using auditory cues. These results describe a previously unreported function for neurons in the human left MFG in auditory processing contributing to motor control.

Distance- and speed-informed kinematics decoding improves M/EEG based upper-limb movement decoder accuracy

OBJECTIVE

One of the main goals in brain-computer interface (BCI) research is the replacement or restoration of lost function in individuals with paralysis. One line of research investigates the inference of movement kinematics from brain activity during different volitional states. A growing number of electroencephalography (EEG) and magnetoencephalography (MEG) studies suggest that information about directional (e.g. velocity) and nondirectional (e.g. speed) movement kinematics is accessible noninvasively. We sought to assess if the neural information associated with both types of kinematics can be combined to improve the decoding accuracy.

APPROACH

In an offline analysis, we reanalyzed the data of two previous experiments containing the recordings of 34 healthy participants (15 EEG, 19 MEG). We decoded 2D movement trajectories from low-frequency M/EEG signals in executed and observed tracking movements, and compared the accuracy of an unscented Kalman filter (UKF) that explicitly modeled the nonlinear relation between directional and nondirectional kinematics to the accuracies of linear Kalman (KF) and Wiener filters which did not combine both types of kinematics.

MAIN RESULTS

At the group level, posterior-parietal and parieto-occipital (executed and observed movements) and sensorimotor areas (executed movements) encoded kinematic information. Correlations between the recorded position and velocity trajectories and the UKF decoded ones were on average 0.49 during executed and 0.36 during observed movements. Compared to the other filters, the UKF could achieve the best trade-off between maximizing the signal to noise ratio and minimizing the amplitude mismatch between the recorded and decoded trajectories.

SIGNIFICANCE

We present direct evidence that directional and nondirectional kinematic information is simultaneously detectable in low-frequency M/EEG signals. Moreover, combining directional and nondirectional kinematic information significantly improves the decoding accuracy upon a linear KF.

Spatial-temporal aspects of continuous EEG-based neurorobotic control

OBJECTIVE

The goal of this work is to identify the spatio-temporal facets of state-of-the-art electroencephalography (EEG)-based continuous neurorobotics that need to be addressed, prior to deployment in practical applications at home and in the clinic.

APPROACH

Nine healthy human subjects participated in five sessions of one-dimensional (1D) horizontal (LR), 1D vertical (UD) and two-dimensional (2D) neural tracking from EEG. Users controlled a robotic arm and virtual cursor to continuously track a Gaussian random motion target using EEG sensorimotor rhythm modulation via motor imagery (MI) commands. Continuous control quality was analyzed in the temporal and spatial domains separately.

MAIN RESULTS

Axis-specific errors during 2D tasks were significantly larger than during 1D counterparts. Fatigue rates were larger for control tasks with higher cognitive demand (LR, left- and right-hand MI) compared to those with lower cognitive demand (UD, both hands MI and rest). Additionally, robotic arm and virtual cursor control exhibited equal tracking error during all tasks. However, further spatial error analysis of 2D control revealed a significant reduction in tracking quality that was dependent on the visual interference of the physical device. In fact, robotic arm performance was significantly greater than that of virtual cursor control when the users' sightlines were not obstructed.

SIGNIFICANCE

This work emphasizes the need for practical interfaces to be designed around real-world tasks of increased complexity. Here, the dependence of control quality on cognitive task demand emphasizes the need for decoders that facilitate the translation of 1D task mastery to 2D control. When device footprint was accounted for, the introduction of a physical robotic arm improved control quality, likely due to increased user engagement. In general, this work demonstrates the need to consider both the physical footprint of devices, the complexity of training tasks, and the synergy of control strategies during the development of neurorobotic control.

The Representation of Finger Movement and Force in Human Motor and Premotor Cortices

The ability to grasp and manipulate objects requires controlling both finger movement kinematics and isometric force in rapid succession. Previous work suggests that these behavioral modes are controlled separately, but it is unknown whether the cerebral cortex represents them differently. Here, we asked the question of how movement and force were represented cortically, when executed sequentially with the same finger. We recorded high-density electrocorticography (ECoG) from the motor and premotor cortices of seven human subjects performing a movement-force motor task. We decoded finger movement [0.7 ± 0.3 fractional variance accounted for (FVAF)] and force (0.7 ± 0.2 FVAF) with high accuracy, yet found different spatial representations. In addition, we used a state-of-the-art deep learning method to uncover smooth, repeatable trajectories through ECoG state space during the movement-force task. We also summarized ECoG across trials and participants by developing a new metric, the neural vector angle (NVA). Thus, state-space techniques can help to investigate broad cortical networks. Finally, we were able to classify the behavioral mode from neural signals with high accuracy (90 ± 6%). Thus, finger movement and force appear to have distinct representations in motor/premotor cortices. These results inform our understanding of the neural control of movement, as well as the design of grasp brain-machine interfaces (BMIs).

Neural Representation of Observed, Imagined, and Attempted Grasping Force in Motor Cortex of Individuals with Chronic Tetraplegia

Hybrid kinetic and kinematic intracortical brain-computer interfaces (iBCIs) have the potential to restore functional grasping and object interaction capabilities in individuals with tetraplegia. This requires an understanding of how kinetic information is represented in neural activity, and how this representation is affected by non-motor parameters such as volitional state (VoS), namely, whether one observes, imagines, or attempts an action. To this end, this work investigates how motor cortical neural activity changes when three human participants with tetraplegia observe, imagine, and attempt to produce three discrete hand grasping forces with the dominant hand. We show that force representation follows the same VoS-related trends as previously shown for directional arm movements; namely, that attempted force production recruits more neural activity compared to observed or imagined force production. Additionally, VoS-modulated neural activity to a greater extent than grasping force. Neural representation of forces was lower than expected, possibly due to compromised somatosensory pathways in individuals with tetraplegia, which have been shown to influence motor cortical activity. Nevertheless, attempted forces (but not always observed or imagined forces) could be decoded significantly above chance, thereby potentially providing relevant information towards the development of a hybrid kinetic and kinematic iBCI.

Causes and consequences of representational drift

The nervous system learns new associations while maintaining memories over long periods, exhibiting a balance between flexibility and stability. Recent experiments reveal that neuronal representations of learned sensorimotor tasks continually change over days and weeks, even after animals have achieved expert behavioral performance. How is learned information stored to allow consistent behavior despite ongoing changes in neuronal activity? What functions could ongoing reconfiguration serve? We highlight recent experimental evidence for such representational drift in sensorimotor systems, and discuss how this fits into a framework of distributed population codes. We identify recent theoretical work that suggests computational roles for drift and argue that the recurrent and distributed nature of sensorimotor representations permits drift while limiting disruptive effects. We propose that representational drift may create error signals between interconnected brain regions that can be used to keep neural codes consistent in the presence of continual change. These concepts suggest experimental and theoretical approaches to studying both learning and maintenance of distributed and adaptive population codes.