Decoding of finger trajectory from ECoG using deep learning

5-year follow-up of a fully implanted brain-computer interface in a spinal cord injury patient

Spinal cord injury (SCI) affects over 250 000 individuals in the US. Brain-computer interfaces (BCIs) may improve quality of life by controlling external devices. Invasive intracortical BCIs have shown promise in clinical trials but degrade in the chronic period and tether patients to acquisition hardware. Alternatively, electrocorticography (ECoG) records data from electrodes on the cortex,and studies evaluating fully implanted BCI-ECoG systems are scarce. Objective. We seek to address this need using a fully implanted ECoG-based BCI that allows for home use in SCI.Approach.The patient used a long-term BCI system, initially controlling an functional electrical stimulation orthosis in the lab and later using an external mechanical orthosis at home. To evaluate its long-term viability, electrode contact impedance, signal quality, and decoder performance were measured. Signal quality was assessed using signal-to-noise ratio and maximum bandwidth of the signal. Decoder performance was monitored using the area under the receiver operator characteristic curve (AUROC).Main results.The study analyzed data from the patient's home environment over 54 months, revealing that the device was used at home for 38 ± 24 min on average daily. After six months, we observed stable event-related desynchronization that aided in determining the onset of motor intention. The decoder's average AUROC across months was 0.959. Importantly, 40 months of the data collected was gather from the subject's home or community environment. The results indicate long-term ECoG recordings were stable for motor-imagery classification and motor control in the community environment in a case of an individual with SCI.Significance.This study presents the long-term feasibility and viability of an ECoG-based BCI system that persists in the home environment in a case of SCI. Future research should explore larger electrode counts with more participants to confirm this stability. Understanding these trends is crucial for clinical utility and chronic viability in broader patient populations.

A protocol for trustworthy EEG decoding with neural networks

Deep learning solutions have rapidly emerged for EEG decoding, achieving state-of-the-art performance on a variety of decoding tasks. Despite their high performance, existing solutions do not fully address the challenge posed by the introduction of many hyperparameters, defining data pre-processing, network architecture, network training, and data augmentation. Automatic hyperparameter search is rarely performed and limited to network-related hyperparameters. Moreover, pipelines are highly sensitive to performance fluctuations due to random initialization, hindering their reliability. Here, we design a comprehensive protocol for EEG decoding that explores the hyperparameters characterizing the entire pipeline and that includes multi-seed initialization for providing robust performance estimates. Our protocol is validated on 9 datasets about motor imagery, P300, SSVEP, including 204 participants and 26 recording sessions, and on different deep learning models. We accompany our protocol with extensive experiments on the main aspects influencing it, such as the number of participants used for hyperparameter search, the split into sequential simpler searches (multi-step search), the use of informed vs. non-informed search algorithms, and the number of random seeds for obtaining stable performance. The best protocol included 2-step hyperparameter search via an informed search algorithm, with the final training and evaluation performed using 10 random initializations. The optimal trade-off between performance and computational time was achieved by using a subset of 3-5 participants for hyperparameter search. Our protocol consistently outperformed baseline state-of-the-art pipelines, widely across datasets and models, and could represent a standard approach for neuroscientists for decoding EEG in a trustworthy and reliable way.

Decoding micro-electrocorticographic signals by using explainable 3D convolutional neural network to predict finger movements

BACKGROUND

Electroencephalography (EEG) and electrocorticography (ECoG) recordings have been used to decode finger movements by analyzing brain activity. Traditional methods focused on single bandpass power changes for movement decoding, utilizing machine learning models requiring manual feature extraction.

NEW METHOD

This study introduces a 3D convolutional neural network (3D-CNN) model to decode finger movements using ECoG data. The model employs adaptive, explainable AI (xAI) techniques to interpret the physiological relevance of brain signals. ECoG signals from epilepsy patients during awake craniotomy were processed to extract power spectral density across multiple frequency bands. These data formed a 3D matrix used to train the 3D-CNN to predict finger trajectories.

RESULTS

The 3D-CNN model showed significant accuracy in predicting finger movements, with root-mean-square error (RMSE) values of 0.26-0.38 for single finger movements and 0.20-0.24 for combined movements. Explainable AI techniques, Grad-CAM and SHAP, identified the high gamma (HG) band as crucial for movement prediction, showing specific cortical regions involved in different finger movements. These findings highlighted the physiological significance of the HG band in motor control.

COMPARISON WITH EXISTING METHODS

The 3D-CNN model outperformed traditional machine learning approaches by effectively capturing spatial and temporal patterns in ECoG data. The use of xAI techniques provided clearer insights into the model's decision-making process, unlike the "black box" nature of standard deep learning models.

CONCLUSIONS

The proposed 3D-CNN model, combined with xAI methods, enhances the decoding accuracy of finger movements from ECoG data. This approach offers a more efficient and interpretable solution for brain-computer interface (BCI) applications, emphasizing the HG band's role in motor control.

Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks

Understanding the dynamical transformation of neural activity to behavior requires new capabilities to nonlinearly model, dissociate and prioritize behaviorally relevant neural dynamics and test hypotheses about the origin of nonlinearity. We present dissociative prioritized analysis of dynamics (DPAD), a nonlinear dynamical modeling approach that enables these capabilities with a multisection neural network architecture and training approach. Analyzing cortical spiking and local field potential activity across four movement tasks, we demonstrate five use-cases. DPAD enabled more accurate neural-behavioral prediction. It identified nonlinear dynamical transformations of local field potentials that were more behavior predictive than traditional power features. Further, DPAD achieved behavior-predictive nonlinear neural dimensionality reduction. It enabled hypothesis testing regarding nonlinearities in neural-behavioral transformation, revealing that, in our datasets, nonlinearities could largely be isolated to the mapping from latent cortical dynamics to behavior. Finally, DPAD extended across continuous, intermittently sampled and categorical behaviors. DPAD provides a powerful tool for nonlinear dynamical modeling and investigation of neural-behavioral data.

Single Finger Trajectory Prediction From Intracranial Brain Activity Using Block-Term Tensor Regression With Fast and Automatic Component Extraction

Multiway- or tensor-based decoding techniques for brain-computer interfaces (BCIs) are believed to better account for the multilinear structure of brain signals than conventional vector- or matrix-based ones. However, despite their outlook on significant performance gains, the used parameter optimization approach is often too computationally demanding so that conventional techniques are still preferred. We propose two novel tensor factorizations which we integrate into our block-term tensor regression (BTTR) algorithm and further introduce a marginalization procedure that guarantees robust predictions while reducing the risk of overfitting (generalized regression). BTTR accounts for the underlying (hidden) data structure in a fully automatic and computationally efficient manner, leading to a significant performance gain over conventional vector- or matrix-based techniques in a challenging real-world application. As a challenging real-world application, we apply BTTR to accurately predict single finger movement trajectories from intracranial recordings in human subjects. We compare the obtained performance with that of the state-of-the-art.

Fast, accurate, and interpretable decoding of electrocorticographic signals using dynamic mode decomposition

Dynamic mode (DM) decomposition decomposes spatiotemporal signals into basic oscillatory components (DMs). DMs can improve the accuracy of neural decoding when used with the nonlinear Grassmann kernel, compared to conventional power features. However, such kernel-based machine learning algorithms have three limitations: large computational time preventing real-time application, incompatibility with non-kernel algorithms, and low interpretability. Here, we propose a mapping function corresponding to the Grassmann kernel that explicitly transforms DMs into spatial DM (sDM) features, which can be used in any machine learning algorithm. Using electrocorticographic signals recorded during various movement and visual perception tasks, the sDM features were shown to improve the decoding accuracy and computational time compared to conventional methods. Furthermore, the components of the sDM features informative for decoding showed similar characteristics to the high-γ power of the signals, but with higher trial-to-trial reproducibility. The proposed sDM features enable fast, accurate, and interpretable neural decoding.

End-to-end deep learning approach to mouse behavior classification from cortex-wide calcium imaging

Deep learning is a powerful tool for neural decoding, broadly applied to systems neuroscience and clinical studies. Interpretable and transparent models that can explain neural decoding for intended behaviors are crucial to identifying essential features of deep learning decoders in brain activity. In this study, we examine the performance of deep learning to classify mouse behavioral states from mesoscopic cortex-wide calcium imaging data. Our convolutional neural network (CNN)-based end-to-end decoder combined with recurrent neural network (RNN) classifies the behavioral states with high accuracy and robustness to individual differences on temporal scales of sub-seconds. Using the CNN-RNN decoder, we identify that the forelimb and hindlimb areas in the somatosensory cortex significantly contribute to behavioral classification. Our findings imply that the end-to-end approach has the potential to be an interpretable deep learning method with unbiased visualization of critical brain regions.

Data augmentation for invasive brain-computer interfaces based on stereo-electroencephalography (SEEG)

Objective.Deep learning is increasingly used for brain-computer interfaces (BCIs). However, the quantity of available data is sparse, especially for invasive BCIs. Data augmentation (DA) methods, such as generative models, can help to address this sparseness. However, all the existing studies on brain signals were based on convolutional neural networks and ignored the temporal dependence. This paper attempted to enhance generative models by capturing the temporal relationship from a time-series perspective.Approach. A conditional generative network (conditional transformer-based generative adversarial network (cTGAN)) based on the transformer model was proposed. The proposed method was tested using a stereo-electroencephalography (SEEG) dataset which was recorded from eight epileptic patients performing five different movements. Three other commonly used DA methods were also implemented: noise injection (NI), variational autoencoder (VAE), and conditional Wasserstein generative adversarial network with gradient penalty (cWGANGP). Using the proposed method, the artificial SEEG data was generated, and several metrics were used to compare the data quality, including visual inspection, cosine similarity (CS), Jensen-Shannon distance (JSD), and the effect on the performance of a deep learning-based classifier.Main results. Both the proposed cTGAN and the cWGANGP methods were able to generate realistic data, while NI and VAE outputted inferior samples when visualized as raw sequences and in a lower dimensional space. The cTGAN generated the best samples in terms of CS and JSD and outperformed cWGANGP significantly in enhancing the performance of a deep learning-based classifier (each of them yielding a significant improvement of 6% and 3.4%, respectively).Significance. This is the first time that DA methods have been applied to invasive BCIs based on SEEG. In addition, this study demonstrated the advantages of the model that preserves the temporal dependence from a time-series perspective.

The role of distinct ECoG frequency features in decoding finger movement

Objective.To identify the electrocorticography (ECoG) frequency features that encode distinct finger movement states during repeated finger flexions.Approach.We used the publicly available Stanford ECoG dataset of cue-based, repeated single finger flexions. Using linear regression, we identified the spectral features that contributed most to the encoding of movement dynamics and discriminating movement events from rest, and combined them to predict finger movement trajectories. Furthermore, we also looked into the effect of the used frequency range and the spatial distribution of the identified features.Main results.Two frequency features generate superior performance, each one for a different movement aspect: high gamma band activity distinguishes movement events from rest, whereas the local motor potential (LMP) codes for movement dynamics. Combining these two features in a finger movement decoder outperformed comparable prior work where the entire spectrum was used as the average correlation coefficient with the true trajectories increased from 0.45 to 0.5, both applied to the Stanford dataset, and erroneous predictions during rest were demoted. In addition, for the first time, our results show the influence of the upper cut-off frequency used to extract LMP, yielding a higher performance when this range is adjusted to the finger movement rate.Significance.This study shows the benefit of a detailed feature analysis prior to designing the finger movement decoder.

Nine decades of electrocorticography: A comparison between epidural and subdural recordings

In recent years, electrocorticography (ECoG) has arisen as a neural signal recording tool in the development of clinically viable neural interfaces. ECoG electrodes are generally placed below the dura mater (subdural) but can also be placed on top of the dura (epidural). In deciding which of these modalities best suits long-term implants, complications and signal quality are important considerations. Conceptually, epidural placement may present a lower risk of complications as the dura is left intact but also a lower signal quality due to the dura acting as a signal attenuator. The extent to which complications and signal quality are affected by the dura, however, has been a matter of debate. To improve our understanding of the effects of the dura on complications and signal quality, we conducted a literature review. We inventorized the effect of the dura on signal quality, decodability and longevity of acute and chronic ECoG recordings in humans and non-human primates. Also, we compared the incidence and nature of serious complications in studies that employed epidural and subdural ECoG. Overall, we found that, even though epidural recordings exhibit attenuated signal amplitude over subdural recordings, particularly for high-density grids, the decodability of epidural recorded signals does not seem to be markedly affected. Additionally, we found that the nature of serious complications was comparable between epidural and subdural recordings. These results indicate that both epidural and subdural ECoG may be suited for long-term neural signal recordings, at least for current generations of clinical and high-density ECoG grids.

Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance

Introduction

In brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.

Methods

We evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.

Results

Our results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.

Discussion

DL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.

An integrated perspective for the diagnosis and therapy of neurodevelopmental disorders - From an engineering point of view

Neurodevelopmental disorders (NDDs) are complex conditions with largely unknown pathophysiology. While many NDD symptoms are familiar, the cause of these disorders remains unclear and may involve a combination of genetic, biological, psychosocial, and environmental risk factors. Current diagnosis relies heavily on behaviorally defined criteria, which may be biased by the clinical team's professional and cultural expectations, thus a push for new biological-based biomarkers for NDDs diagnosis is underway. Emerging new research technologies offer an unprecedented view into the electrical, chemical, and physiological activity in the brain and with further development in humans may provide clinically relevant diagnoses. These could also be extended to new treatment options, which can start to address the underlying physiological issues. When combined with current speech, language, occupational therapy, and pharmacological treatment these could greatly improve patient outcomes. The current review will discuss the latest technologies that are being used or may be used for NDDs diagnosis and treatment. The aim is to provide an inspiring and forward-looking view for future research in the field.

Phase-amplitude coupling-based adaptive filters for neural signal decoding

Bandpass filters play a core role in ECoG signal processing. Commonly used frequency bands such as alpha, beta, and gamma bands can reflect the normal rhythm of the brain. However, the universally predefined bands might not be optimal for a specific task. Especially the gamma band usually covers a wide frequency span (i.e., 30-200 Hz) which can be too coarse to capture features that appear in narrow bands. An ideal option is to find the optimal frequency bands for specific tasks in real-time and dynamically. To tackle this problem, we propose an adaptive band filter that selects the useful frequency band in a data-driven way. Specifically, we leverage the phase-amplitude coupling (PAC) of the coupled working mechanism of synchronizing neuron and pyramidal neurons in neuronal oscillations, in which the phase of slower oscillations modulates the amplitude of faster ones, to help locate the fine frequency bands from the gamma range, in a task-specific and individual-specific way. Thus, the information can be more precisely extracted from ECoG signals to improve neural decoding performance. Based on this, an end-to-end decoder (PACNet) is proposed to construct a neural decoding application with adaptive filter banks in a uniform framework. Experiments show that PACNet can improve neural decoding performance universally with different tasks.

Movement decoding using spatio-spectral features of cortical and subcortical local field potentials

The first commercially sensing enabled deep brain stimulation (DBS) devices for the treatment of movement disorders have recently become available. In the future, such devices could leverage machine learning based brain signal decoding strategies to individualize and adapt therapy in real-time. As multi-channel recordings become available, spatial information may provide an additional advantage for informing machine learning models. To investigate this concept, we compared decoding performances from single channels vs. spatial filtering techniques using intracerebral multitarget electrophysiology in Parkinson's disease patients undergoing DBS implantation. We investigated the feasibility of spatial filtering in invasive neurophysiology and the putative utility of combined cortical ECoG and subthalamic local field potential signals for decoding grip-force, a well-defined and continuous motor readout. We found that adding spatial information to the model can improve decoding (6% gain in decoding), but the spatial patterns and additional benefit was highly individual. Beyond decoding performance results, spatial filters and patterns can be used to obtain meaningful neurophysiological information about the brain networks involved in target behavior. Our results highlight the importance of individualized approaches for brain signal decoding, for which multielectrode recordings and spatial filtering can improve precision medicine approaches for clinical brain computer interfaces.

Finger movement and coactivation predicted from intracranial brain activity using extended block-term tensor regression

Objective.We introduce extended Block-Term Tensor Regression (eBTTR), a novel regression method designed to account for the multilinear nature of human intracranial finger movement recordings.Approach.The proposed method relies on recursive Tucker decomposition combined with automatic component extraction.Main results.eBTTR outperforms state-of-the-art regression approaches, including multilinear and deep learning ones, in accurately predicting finger trajectories as well as unintentional finger coactivations.Significance.eBTTR rivals state-of-the-art approaches while being less computationally expensive which is an advantage when intracranial electrodes are implanted acutely, as part of the patient's presurgical workup, limiting time for decoder development and testing.

NeuralTree: A 256-Channel 0.227-μJ/Class Versatile Neural Activity Classification and Closed-Loop Neuromodulation SoC

Closed-loop neural interfaces with on-chip machine learning can detect and suppress disease symptoms in neurological disorders or restore lost functions in paralyzed patients. While high-density neural recording can provide rich neural activity information for accurate disease-state detection, existing systems have low channel counts and poor scalability, which could limit their therapeutic efficacy. This work presents a highly scalable and versatile closed-loop neural interface SoC that can overcome these limitations. A 256-channel time-division multiplexed (TDM) front-end with a two-step fast-settling mixed-signal DC servo loop (DSL) is proposed to record high-spatial-resolution neural activity and perform channel-selective brain-state inference. A tree-structured neural network (NeuralTree) classification processor extracts a rich set of neural biomarkers in a patient- and disease-specific manner. Trained with an energy-aware learning algorithm, the NeuralTree classifier detects the symptoms of underlying disorders (e.g., epilepsy and movement disorders) at an optimal energy-accuracy trade-off. A 16-channel high-voltage (HV) compliant neurostimulator closes the therapeutic loop by delivering charge-balanced biphasic current pulses to the brain. The proposed SoC was fabricated in 65nm CMOS and achieved a 0.227μJ/class energy efficiency in a compact area of 0.014mm2/channel. The SoC was extensively verified on human electroencephalography (EEG) and intracranial EEG (iEEG) epilepsy datasets, obtaining 95.6%/94% sensitivity and 96.8%/96.9% specificity, respectively. In-vivo neural recordings using soft μECoG arrays and multi-domain biomarker extraction were further performed on a rat model of epilepsy. In addition, for the first time in literature, on-chip classification of rest-state tremor in Parkinson's disease (PD) from human local field potentials (LFPs) was demonstrated.

Neurochemical Concentration Prediction Using Deep Learning vs Principal Component Regression in Fast Scan Cyclic Voltammetry: A Comparison Study

Neurotransmitters, such as dopamine and serotonin, are responsible for mediating a wide array of neurologic functions, from memory to motivation. From measurements using fast scan cyclic voltammetry (FSCV), one of the main tools used to detect synaptic efflux of neurochemicals in vivo, principal component regression (PCR), has been commonly used to predict the identity and concentrations of neurotransmitters. However, the sensitivity and discrimination performance of PCR have room for improvement, especially for analyzing mixtures of similar oxidizable neurochemicals. Deep learning may be able to address these challenges. To date, there have been a few studies to apply machine learning to FSCV, but no attempt to apply deep learning to neurotransmitter mixture discrimination and no comparative study have been performed between PCR and deep learning methods to demonstrate which is more accurate for FSCV analysis so far. In this study, we compared the neurochemical identification and concentration estimation performance of PCR and deep learning in an analysis of FSCV recordings of catecholamine and indolamine neurotransmitters. Both analysis methods were tested on in vitro FSCV data with a single or mixture of neurotransmitters at the desired concentration. In addition, the estimation performance of PCR and deep learning was compared in incorporation with in vivo experiments to evaluate the practical usage. Pharmacological tests were also conducted to see whether deep learning would track the increased amount of catecholamine levels in the brain. Using conventional FSCV, we used five electrodes and recorded in vitro background-subtracted cyclic voltammograms from four neurotransmitters, dopamine, epinephrine, norepinephrine, and serotonin, with five concentrations of each substance, as well as various mixtures of the four analytes. The results showed that the identification accuracy errors were reduced 5-20% by using deep learning compared to using PCR for mixture analysis, and the two methods were comparable for single analyte analysis. The applied deep-learning-based method demonstrated not only higher identification accuracy but also better discrimination performance than PCR for mixtures of neurochemicals and even for in vivo testing. Therefore, we suggest that deep learning should be chosen as a more reliable tool to analyze FSCV data compared to conventional PCR methods although further work is still needed on developing complete validation procedures prior to widespread use.

Decoding Intracranial EEG With Machine Learning: A Systematic Review

Advances in intracranial electroencephalography (iEEG) and neurophysiology have enabled the study of previously inaccessible brain regions with high fidelity temporal and spatial resolution. Studies of iEEG have revealed a rich neural code subserving healthy brain function and which fails in disease states. Machine learning (ML), a form of artificial intelligence, is a modern tool that may be able to better decode complex neural signals and enhance interpretation of these data. To date, a number of publications have applied ML to iEEG, but clinician awareness of these techniques and their relevance to neurosurgery, has been limited. The present work presents a review of existing applications of ML techniques in iEEG data, discusses the relative merits and limitations of the various approaches, and examines potential avenues for clinical translation in neurosurgery. One-hundred-seven articles examining artificial intelligence applications to iEEG were identified from 3 databases. Clinical applications of ML from these articles were categorized into 4 domains: i) seizure analysis, ii) motor tasks, iii) cognitive assessment, and iv) sleep staging. The review revealed that supervised algorithms were most commonly used across studies and often leveraged publicly available timeseries datasets. We conclude with recommendations for future work and potential clinical applications.

Electrocorticography is superior to subthalamic local field potentials for movement decoding in Parkinson's disease

Brain signal decoding promises significant advances in the development of clinical brain computer interfaces (BCI). In Parkinson's disease (PD), first bidirectional BCI implants for adaptive deep brain stimulation (DBS) are now available. Brain signal decoding can extend the clinical utility of adaptive DBS but the impact of neural source, computational methods and PD pathophysiology on decoding performance are unknown. This represents an unmet need for the development of future neurotechnology. To address this, we developed an invasive brain-signal decoding approach based on intraoperative sensorimotor electrocorticography (ECoG) and subthalamic LFP to predict grip-force, a representative movement decoding application, in 11 PD patients undergoing DBS. We demonstrate that ECoG is superior to subthalamic LFP for accurate grip-force decoding. Gradient boosted decision trees (XGBOOST) outperformed other model architectures. ECoG based decoding performance negatively correlated with motor impairment, which could be attributed to subthalamic beta bursts in the motor preparation and movement period. This highlights the impact of PD pathophysiology on the neural capacity to encode movement vigor. Finally, we developed a connectomic analysis that could predict grip-force decoding performance of individual ECoG channels across patients by using their connectomic fingerprints. Our study provides a neurophysiological and computational framework for invasive brain signal decoding to aid the development of an individualized precision-medicine approach to intelligent adaptive DBS.

A framework for motor imagery with LSTM neural network

BACKGROUND AND OBJECTIVE

How to learn robust representations from brain activities and to improve algorithm performance are the most significant issues for brain-computer interface systems.

METHODS

This study introduces a long short-term memory recurrent neural network to decode the multichannel electroencephalogram or electrocorticogram for implementing an effective motor imagery-based brain-computer interface system. The unique information processing mechanism of the long short-term memory network characterizes spatio-temporal dynamics in time sequences. This study evaluates the proposed method using publically available electroencephalogram/electrocorticogram datasets.

RESULTS

The decoded features coupled with a gradient boosting classifier could obtain high recognition accuracies of 99% for electroencephalogram and 100% for electrocorticogram, respectively.

CONCLUSIONS

The results demonstrated that the proposed model can estimate robust spatial-temporal features and obtain significant performance improvement for motor imagery-based brain-computer interface systems. Further, the proposed method is of low computational complexity.

Machine learning based brain signal decoding for intelligent adaptive deep brain stimulation

Sensing enabled implantable devices and next-generation neurotechnology allow real-time adjustments of invasive neuromodulation. The identification of symptom and disease-specific biomarkers in invasive brain signal recordings has inspired the idea of demand dependent adaptive deep brain stimulation (aDBS). Expanding the clinical utility of aDBS with machine learning may hold the potential for the next breakthrough in the therapeutic success of clinical brain computer interfaces. To this end, sophisticated machine learning algorithms optimized for decoding of brain states from neural time-series must be developed. To support this venture, this review summarizes the current state of machine learning studies for invasive neurophysiology. After a brief introduction to the machine learning terminology, the transformation of brain recordings into meaningful features for decoding of symptoms and behavior is described. Commonly used machine learning models are explained and analyzed from the perspective of utility for aDBS. This is followed by a critical review on good practices for training and testing to ensure conceptual and practical generalizability for real-time adaptation in clinical settings. Finally, first studies combining machine learning with aDBS are highlighted. This review takes a glimpse into the promising future of intelligent adaptive DBS (iDBS) and concludes by identifying four key ingredients on the road for successful clinical adoption: i) multidisciplinary research teams, ii) publicly available datasets, iii) open-source algorithmic solutions and iv) strong world-wide research collaborations.

Interpretable functional specialization emerges in deep convolutional networks trained on brain signals

OBJECTIVE

Functional specialization is fundamental to neural information processing. Here, we study whether and how functional specialization emerges in artificial deep convolutional neural networks (CNNs) during a brain-computer interfacing (BCI) task.

APPROACH

We trained CNNs to predict hand movement speed from intracranial EEG (iEEG) and delineated how units across the different CNN hidden layers learned to represent the iEEG signal.

MAIN RESULTS

We show that distinct, functionally interpretable neural populations emerged as a result of the training process. While some units became sensitive to either iEEG amplitude or phase, others showed bimodal behavior with significant sensitivity to both features. Pruning of highly-sensitive units resulted in a steep drop of decoding accuracy not observed for pruning of less sensitive units, highlighting the functional relevance of the amplitude- and phase-specialized populations.

SIGNIFICANCE

We anticipate that emergent functional specialization as uncovered here will become a key concept in research towards interpretable deep learning for neuroscience and BCI applications.

Decoding ECoG signal into 3D hand translation using deep learning

Objective.Motor brain-computer interfaces (BCIs) are a promising technology that may enable motor-impaired people to interact with their environment. BCIs would potentially compensate for arm and hand function loss, which is the top priority for individuals with tetraplegia. Designing real-time and accurate BCI is crucial to make such devices useful, safe, and easy to use by patients in a real-life environment. Electrocorticography (ECoG)-based BCIs emerge as a good compromise between invasiveness of the recording device and good spatial and temporal resolution of the recorded signal. However, most ECoG signal decoders used to predict continuous hand movements are linear models. These models have a limited representational capacity and may fail to capture the relationship between ECoG signal features and continuous hand movements. Deep learning (DL) models, which are state-of-the-art in many problems, could be a solution to better capture this relationship.Approach.In this study, we tested several DL-based architectures to predict imagined 3D continuous hand translation using time-frequency features extracted from ECoG signals. The dataset used in the analysis is a part of a long-term clinical trial (ClinicalTrials.gov identifier: NCT02550522) and was acquired during a closed-loop experiment with a tetraplegic subject. The proposed architectures include multilayer perceptron, convolutional neural networks (CNNs), and long short-term memory networks (LSTM). The accuracy of the DL-based and multilinear models was compared offline using cosine similarity.Main results.Our results show that CNN-based architectures outperform the current state-of-the-art multilinear model. The best architecture exploited the spatial correlation between neighboring electrodes with CNN and benefited from the sequential character of the desired hand trajectory by using LSTMs. Overall, DL increased the average cosine similarity, compared to the multilinear model, by up to 60%, from 0.189 to 0.302 and from 0.157 to 0.249 for the left and right hand, respectively.Significance.This study shows that DL-based models could increase the accuracy of BCI systems in the case of 3D hand translation prediction in a tetraplegic subject.

Decoding Bilateral Hindlimb Kinematics From Cat Spinal Signals Using Three-Dimensional Convolutional Neural Network

To date, decoding limb kinematic information mostly relies on neural signals recorded from the peripheral nerve, dorsal root ganglia (DRG), ventral roots, spinal cord gray matter, and the sensorimotor cortex. In the current study, we demonstrated that the neural signals recorded from the lateral and dorsal columns within the spinal cord have the potential to decode hindlimb kinematics during locomotion. Experiments were conducted using intact cats. The cats were trained to walk on a moving belt in a hindlimb-only condition, while their forelimbs were kept on the front body of the treadmill. The bilateral hindlimb joint angles were decoded using local field potential signals recorded using a microelectrode array implanted in the dorsal and lateral columns of both the left and right sides of the cat spinal cord. The results show that contralateral hindlimb kinematics can be decoded as accurately as ipsilateral kinematics. Interestingly, hindlimb kinematics of both legs can be accurately decoded from the lateral columns within one side of the spinal cord during hindlimb-only locomotion. The results indicated that there was no significant difference between the decoding performances obtained using neural signals recorded from the dorsal and lateral columns. The results of the time-frequency analysis show that event-related synchronization (ERS) and event-related desynchronization (ERD) patterns in all frequency bands could reveal the dynamics of the neural signals during movement. The onset and offset of the movement can be clearly identified by the ERD/ERS patterns. The results of the mutual information (MI) analysis showed that the theta frequency band contained significantly more limb kinematics information than the other frequency bands. Moreover, the theta power increased with a higher locomotion speed.

Fast and accurate decoding of finger movements from ECoG through Riemannian features and modern machine learning techniques

Objective: Accurate decoding of individual finger movements is crucial for advanced prosthetic control. In this work, we introduce the use of Riemannian-space features and temporal dynamics of electrocorticography (ECoG) signal combined with modern machine learning tools to improve the motor decoding accuracy at the level of individual fingers. Approach: We selected a set of informative biomarkers that correlated with finger movements and evaluated the performance of state-of-the-art machine learning algorithms on the BCI competition IV dataset (ECoG, three subjects) and a second ECoG dataset with a similar recording paradigm (Stanford, 9 subjects). We further explored the temporal concatenation of features to effectively capture the history of ECoG signal, which led to a significant improvement over single-epoch decoding in both classification (p<0.01) and regression tasks (p<0.01). Main results: Using feature concatenation and gradient boosted trees (the top-performing model), we achieved a classification accuracy of 77.0% in detecting individual finger movements (6-class task, including rest state), improving over the state-of-the-art conditional random fields (CRF) by 11.7% on the 3 BCI competition subjects. In continuous decoding of movement trajectory, our approach resulted in an average Pearson's correlation coefficient (r) of 0.537 across subjects and fingers, outperforming both the BCI competition winner and the state-of-the-art approach reported on the same dataset (CNN+LSTM). Furthermore, our proposed method features a low time complexity, with only <17.2s required for training and <50ms for inference. This enables about 250× speed-up in training compared to previously reported deep learning method with state-of-the-art performance. Significance: The proposed techniques enable fast, reliable, and high-performance prosthetic control through minimally-invasive cortical signals.

Foundations of Time Series Analysis

For almost a century, classical statistical methods including exponential smoothing and autoregression integrated moving averages (ARIMA) have been predominant in the analysis of time series (TS) and in the pursuit of forecasting future events from historical data. TS are chronological sequences of observations, and TS data are therefore prevalent in many aspects of clinical medicine and academic neuroscience. With the rise of highly complex and nonlinear datasets, machine learning (ML) methods have become increasingly popular for prediction or pattern detection and within neurosciences, including neurosurgery. ML methods regularly outperform classical methods and have been successfully applied to, inter alia, predict physiological responses in intracranial pressure monitoring or to identify seizures in EEGs. Implementing nonparametric methods for TS analysis in clinical practice can benefit clinical decision making and sharpen our diagnostic armory.

Generalized neural decoders for transfer learning across participants and recording modalities

OBJECTIVE

Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.

APPROACH

We introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.

MAIN RESULTS

HTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet's trained weights and demonstrate its ability to extract physiologically-relevant features.

SIGNIFICANCE

By generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Deep learning approaches for neural decoding across architectures and recording modalities

Decoding behavior, perception or cognitive state directly from neural signals is critical for brain-computer interface research and an important tool for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to functional magnetic resonance imaging. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

EEG-Based BCI System to Detect Fingers Movements

The advancement of assistive technologies toward the restoration of the mobility of paralyzed and/or amputated limbs will go a long way. Herein, we propose a system that adopts the brain-computer interface technology to control prosthetic fingers with the use of brain signals. To predict the movements of each finger, complex electroencephalogram (EEG) signal processing algorithms should be applied to remove the outliers, extract features, and be able to handle separately the five human fingers. The proposed method deals with a multi-class classification problem. Our machine learning strategy to solve this problem is built on an ensemble of one-class classifiers, each of which is dedicated to the prediction of the intention to move a specific finger. Regions of the brain that are sensitive to the movements of the fingers are identified and located. The average accuracy of the proposed EEG signal processing chain reached 81% for five subjects. Unlike the majority of existing prototypes that allow only one single finger to be controlled and only one movement to be performed at a time, the system proposed will enable multiple fingers to perform movements simultaneously. Although the proposed system classifies five tasks, the obtained accuracy is too high compared with a binary classification system. The proposed system contributes to the advancement of a novel prosthetic solution that allows people with severe disabilities to perform daily tasks in an easy manner.

Control for multifunctionality: bioinspired control based on feeding in Aplysia californica

Animals exhibit remarkable feats of behavioral flexibility and multifunctional control that remain challenging for robotic systems. The neural and morphological basis of multifunctionality in animals can provide a source of bioinspiration for robotic controllers. However, many existing approaches to modeling biological neural networks rely on computationally expensive models and tend to focus solely on the nervous system, often neglecting the biomechanics of the periphery. As a consequence, while these models are excellent tools for neuroscience, they fail to predict functional behavior in real time, which is a critical capability for robotic control. To meet the need for real-time multifunctional control, we have developed a hybrid Boolean model framework capable of modeling neural bursting activity and simple biomechanics at speeds faster than real time. Using this approach, we present a multifunctional model of Aplysia californica feeding that qualitatively reproduces three key feeding behaviors (biting, swallowing, and rejection), demonstrates behavioral switching in response to external sensory cues, and incorporates both known neural connectivity and a simple bioinspired mechanical model of the feeding apparatus. We demonstrate that the model can be used for formulating testable hypotheses and discuss the implications of this approach for robotic control and neuroscience.

The Representation of Finger Movement and Force in Human Motor and Premotor Cortices

The ability to grasp and manipulate objects requires controlling both finger movement kinematics and isometric force in rapid succession. Previous work suggests that these behavioral modes are controlled separately, but it is unknown whether the cerebral cortex represents them differently. Here, we asked the question of how movement and force were represented cortically, when executed sequentially with the same finger. We recorded high-density electrocorticography (ECoG) from the motor and premotor cortices of seven human subjects performing a movement-force motor task. We decoded finger movement [0.7 ± 0.3 fractional variance accounted for (FVAF)] and force (0.7 ± 0.2 FVAF) with high accuracy, yet found different spatial representations. In addition, we used a state-of-the-art deep learning method to uncover smooth, repeatable trajectories through ECoG state space during the movement-force task. We also summarized ECoG across trials and participants by developing a new metric, the neural vector angle (NVA). Thus, state-space techniques can help to investigate broad cortical networks. Finally, we were able to classify the behavioral mode from neural signals with high accuracy (90 ± 6%). Thus, finger movement and force appear to have distinct representations in motor/premotor cortices. These results inform our understanding of the neural control of movement, as well as the design of grasp brain-machine interfaces (BMIs).

Using Coherence-based spectro-spatial filters for stimulus features prediction from electro-corticographic recordings

The traditional approach in neuroscience relies on encoding models where brain responses are related to different stimuli in order to establish dependencies. In decoding tasks, on the contrary, brain responses are used to predict the stimuli, and traditionally, the signals are assumed stationary within trials, which is rarely the case for natural stimuli. We hypothesize that a decoding model assuming each experimental trial as a realization of a random process more likely reflects the statistical properties of the undergoing process compared to the assumption of stationarity. Here, we propose a Coherence-based spectro-spatial filter that allows for reconstructing stimulus features from brain signal's features. The proposed method extracts common patterns between features of the brain signals and the stimuli that produced them. These patterns, originating from different recording electrodes are combined, forming a spatial filter that produces a unified prediction of the presented stimulus. This approach takes into account frequency, phase, and spatial distribution of brain features, hence avoiding the need to predefine specific frequency bands of interest or phase relationships between stimulus and brain responses manually. Furthermore, the model does not require the tuning of hyper-parameters, reducing significantly the computational load attached to it. Using three different cognitive tasks (motor movements, speech perception, and speech production), we show that the proposed method consistently improves stimulus feature predictions in terms of correlation (group averages of 0.74 for motor movements, 0.84 for speech perception, and 0.74 for speech production) in comparison with other methods based on regularized multivariate regression, probabilistic graphical models and artificial neural networks. Furthermore, the model parameters revealed those anatomical regions and spectral components that were discriminant in the different cognitive tasks. This novel method does not only provide a useful tool to address fundamental neuroscience questions, but could also be applied to neuroprosthetics.

Decoding Movement From Electrocorticographic Activity: A Review

Electrocorticography (ECoG) holds promise to provide efficient neuroprosthetic solutions for people suffering from neurological disabilities. This recording technique combines adequate temporal and spatial resolution with the lower risks of medical complications compared to the other invasive methods. ECoG is routinely used in clinical practice for preoperative cortical mapping in epileptic patients. During the last two decades, research utilizing ECoG has considerably grown, including the paradigms where behaviorally relevant information is extracted from ECoG activity with decoding algorithms of different complexity. Several research groups have advanced toward the development of assistive devices driven by brain-computer interfaces (BCIs) that decode motor commands from multichannel ECoG recordings. Here we review the evolution of this field and its recent tendencies, and discuss the potential areas for future development.

Exploring the application of artificial intelligence technology for identification of water pollution characteristics and tracing the source of water quality pollutants

Point sources are important routes through which pollutants enter rivers. It is important to identify the characteristics of and trace the origins of water pollutants. In this study, an artificial intelligence system called the integrated long short-term memory network (LSTM), using cross-correlation and association rules (Apriori), was used to identify the characteristics of water pollutants and trace industrial point sources of pollutants. Water quality monitoring data from Shandong Province, China, were used to verify the applicability of the artificial intelligence system using a cross-correlation method to develop a water quality cross-correlation map. The map was used to identify highly correlated pollutants affecting water quality, then the association rules (Apriori) were used to track the pollutants to industries common in the study area. The highly correlated water pollutants and relevant industries were used as inputs for the LSTM to determine how well the LSTM traced sources of water pollutants. The results showed that (1) changes in water quality were affected in different ways by different industries and different distributions and production cycles of the pollutant point sources; (2) water quality correlation maps can be used to identify regular and abnormal fluctuations in point source pollutant emissions by identifying changes in water quality characteristics and frequent itemsets in water quality indices can be used to trace the industries that most strongly affect water quality; and (3) the LSTM accurately traced point sources of future changes in water quality. In conclusion, the artificial intelligence scheme described here can be applied to aquatic systems.

Human motor decoding from neural signals: a review

Many people suffer from movement disability due to amputation or neurological diseases. Fortunately, with modern neurotechnology now it is possible to intercept motor control signals at various points along the neural transduction pathway and use that to drive external devices for communication or control. Here we will review the latest developments in human motor decoding. We reviewed the various strategies to decode motor intention from human and their respective advantages and challenges. Neural control signals can be intercepted at various points in the neural signal transduction pathway, including the brain (electroencephalography, electrocorticography, intracortical recordings), the nerves (peripheral nerve recordings) and the muscles (electromyography). We systematically discussed the sites of signal acquisition, available neural features, signal processing techniques and decoding algorithms in each of these potential interception points. Examples of applications and the current state-of-the-art performance were also reviewed. Although great strides have been made in human motor decoding, we are still far away from achieving naturalistic and dexterous control like our native limbs. Concerted efforts from material scientists, electrical engineers, and healthcare professionals are needed to further advance the field and make the technology widely available in clinical use.

The Advantage of Low-Delta Electroencephalogram Phase Feature for Reconstructing the Center-Out Reaching Hand Movements

It is an emerging frontier of research on the use of neural signals for prosthesis control, in order to restore lost function to amputees and patients after spinal cord injury. Compared to the invasive neural signal based brain-machine interface (BMI), a non-invasive alternative, i.e., the electroencephalogram (EEG)-based BMI would be more widely accepted by the patients above. Ideally, a real-time continuous neuroprosthestic control is required for practical applications. However, conventional EEG-based BMIs mainly deal with the discrete brain activity classification. Until recently, the literature has reported several attempts for achieving the real-time continuous control by reconstructing the continuous movement parameters (e.g., speed, position, etc.) from the EEG recordings, and the low-frequency band EEG is consistently reported to encode the continuous motor control information. Previous studies with executed movement tasks have extensively relied on the amplitude representation of such slow oscillations of EEG signals for building models to decode kinematic parameters. Inspired by the recent successes of instantaneous phase of low-frequency invasive brain signals in the motor control and sensory processing domains, this study examines the extension of such a slow-oscillation phase representation to the reconstructing two-dimensional hand movements, with the non-invasive EEG signals for the first time. The data for analysis are collected on five healthy subjects performing 2D hand center-out reaching along four directions in two sessions. On representative channels over the cortices encoding the execution information of reaching movements, we show that the low-delta EEG phase representation is characterized by higher signal-to-noise ratio and stronger modulation by the movement tasks, compared to the low-delta EEG amplitude representation. Furthermore, we have tested the low-delta EEG phase representation with two commonly used linear decoding models. The results demonstrate that the low-delta EEG phase based decoders lead to superior performance for 2D executed movement reconstruction to its amplitude based counterparts, as well as the other-frequency band amplitude and power based features. Thus, our study contributes to improve the movement reconstruction from EEG by introducing a new feature set based on the low-delta EEG phase patterns, and demonstrates its potential for continuous fine motion control of neuroprostheses.

Toward Electrophysiology-Based Intelligent Adaptive Deep Brain Stimulation for Movement Disorders

Deep brain stimulation (DBS) represents one of the major clinical breakthroughs in the age of translational neuroscience. In 1987, Benabid and colleagues demonstrated that high-frequency stimulation can mimic the effects of ablative neurosurgery in Parkinson's disease (PD), while offering two key advantages to previous procedures: adjustability and reversibility. Deep brain stimulation is now an established therapeutic approach that robustly alleviates symptoms in patients with movement disorders, such as Parkinson's disease, essential tremor, and dystonia, who present with inadequate or adverse responses to medication. Currently, stimulation electrodes are implanted in specific target regions of the basal ganglia-thalamic circuit and stimulation pulses are delivered chronically. To achieve optimal therapeutic effect, stimulation frequency, amplitude, and pulse width must be adjusted on a patient-specific basis by a movement disorders specialist. The finding that pathological neural activity can be sampled directly from the target region using the DBS electrode has inspired a novel DBS paradigm: closed-loop adaptive DBS (aDBS). The goal of this strategy is to identify pathological and physiologically normal patterns of neuronal activity that can be used to adapt stimulation parameters to the concurrent therapeutic demand. This review will give detailed insight into potential biomarkers and discuss next-generation strategies, implementing advances in artificial intelligence, to further elevate the therapeutic potential of DBS by capitalizing on its modifiable nature. Development of intelligent aDBS, with an ability to deliver highly personalized treatment regimens and to create symptom-specific therapeutic strategies in real-time, could allow for significant further improvements in the quality of life for movement disorders patients with DBS that ultimately could outperform traditional drug treatment.

Rapid Decoding of Hand Gestures in Electrocorticography Using Recurrent Neural Networks

Brain-computer interface (BCI) is a direct communication pathway between brain and external devices, and BCI-based prosthetic devices are promising to provide new rehabilitation options for people with motor disabilities. Electrocorticography (ECoG) signals contain rich information correlated with motor activities, and have great potential in hand gesture decoding. However, most existing decoders use long time windows, thus ignore the temporal dynamics within the period. In this study, we propose to use recurrent neural networks (RNNs) to exploit the temporal information in ECoG signals for robust hand gesture decoding. With RNN's high nonlinearity modeling ability, our method can effectively capture the temporal information in ECoG time series for robust gesture recognition. In the experiments, we decode three hand gestures using ECoG signals of two participants, and achieve an accuracy of 90%. Specially, we investigate the possibility of recognizing the gestures in a time interval as short as possible after motion onsets. Our method rapidly recognizes gestures within 0.5 s after motion onsets with an accuracy of about 80%. Experimental results also indicate that the temporal dynamics is especially informative for effective and rapid decoding of hand gestures.