1
|
Levett JJ, Elkaim LM, Niazi F, Weber MH, Iorio-Morin C, Bonizzato M, Weil AG. Invasive Brain Computer Interface for Motor Restoration in Spinal Cord Injury: A Systematic Review. Neuromodulation 2024; 27:597-603. [PMID: 37943244 DOI: 10.1016/j.neurom.2023.10.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2023] [Revised: 09/10/2023] [Accepted: 10/09/2023] [Indexed: 11/10/2023]
Abstract
STUDY DESIGN Systematic review of the literature. OBJECTIVES In recent years, brain-computer interface (BCI) has emerged as a potential treatment for patients with spinal cord injury (SCI). This is the first systematic review of the literature on invasive closed-loop BCI technologies for the treatment of SCI in humans. MATERIALS AND METHODS A comprehensive search of PubMed MEDLINE, Web of Science, and Ovid EMBASE was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. RESULTS Of 8316 articles collected, 19 studies met all the inclusion criteria. Data from 21 patients were extracted from these studies. All patients sustained a cervical SCI and were treated using either a BCI with intracortical microelectrode arrays (n = 18, 85.7%) or electrocorticography (n = 3, 14.3%). To decode these neural signals, machine learning and statistical models were used: support vector machine in eight patients (38.1%), linear estimator in seven patients (33.3%), Hidden Markov Model in three patients (14.3%), and other in three patients (14.3%). As the outputs, ten patients (47.6%) underwent noninvasive functional electrical stimulation (FES) with a cuff; one (4.8%) had an invasive FES with percutaneous stimulation, and ten (47.6%) used an external device (neuroprosthesis or virtual avatar). Motor function was restored in all patients for each assigned task. Clinical outcome measures were heterogeneous across all studies. CONCLUSIONS Invasive techniques of BCI show promise for the treatment of SCI, but there is currently no technology that can restore complete functional autonomy in patients with SCI. The current techniques and outcomes of BCI vary greatly. Because invasive BCIs are still in the early stages of development, further clinical studies should be conducted to optimize the prognosis for patients with SCI.
Collapse
Affiliation(s)
- Jordan J Levett
- Faculty of Medicine, University of Montreal, Montreal, Quebec, Canada
| | - Lior M Elkaim
- Department of Neurology and Neurosurgery, McGill University, Montreal, Quebec, Canada
| | - Farbod Niazi
- Faculty of Medicine, University of Montreal, Montreal, Quebec, Canada
| | - Michael H Weber
- Department of Orthopaedic Surgery, McGill University, Montreal, Quebec, Canada
| | | | - Marco Bonizzato
- Department of Electrical Engineering and Institute of Biomedical Engineering, Polytechnique Montréal, Montreal, Quebec, Canada; Department of Neuroscience and Centre interdisciplinaire sur le cerveau et l'apprentissage, University of Montreal, Montreal, Quebec, Canada
| | - Alexander G Weil
- Division of Neurosurgery, St-Justine University Hospital, Montreal, Quebec, Canada.
| |
Collapse
|
2
|
Silva AB, Liu JR, Metzger SL, Bhaya-Grossman I, Dougherty ME, Seaton MP, Littlejohn KT, Tu-Chan A, Ganguly K, Moses DA, Chang EF. A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages. Nat Biomed Eng 2024:10.1038/s41551-024-01207-5. [PMID: 38769157 DOI: 10.1038/s41551-024-01207-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 04/01/2024] [Indexed: 05/22/2024]
Abstract
Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.
Collapse
Affiliation(s)
- Alexander B Silva
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Jessie R Liu
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Sean L Metzger
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Ilina Bhaya-Grossman
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Maximilian E Dougherty
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
| | - Margaret P Seaton
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
| | - Kaylo T Littlejohn
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA, USA
| | - Adelyn Tu-Chan
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - Karunesh Ganguly
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - David A Moses
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - Edward F Chang
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA.
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA.
| |
Collapse
|
3
|
Jung T, Zeng N, Fabbri JD, Eichler G, Li Z, Willeke K, Wingel KE, Dubey A, Huq R, Sharma M, Hu Y, Ramakrishnan G, Tien K, Mantovani P, Parihar A, Yin H, Oswalt D, Misdorp A, Uguz I, Shinn T, Rodriguez GJ, Nealley C, Gonzales I, Roukes M, Knecht J, Yoshor D, Canoll P, Spinazzi E, Carloni LP, Pesaran B, Patel S, Youngerman B, Cotton RJ, Tolias A, Shepard KL. Stable, chronic in-vivo recordings from a fully wireless subdural-contained 65,536-electrode brain-computer interface device. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.17.594333. [PMID: 38798494 PMCID: PMC11118429 DOI: 10.1101/2024.05.17.594333] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
Minimally invasive, high-bandwidth brain-computer-interface (BCI) devices can revolutionize human applications. With orders-of-magnitude improvements in volumetric efficiency over other BCI technologies, we developed a 50-μm-thick, mechanically flexible micro-electrocorticography (μECoG) BCI, integrating 256×256 electrodes, signal processing, data telemetry, and wireless powering on a single complementary metal-oxide-semiconductor (CMOS) substrate containing 65,536 recording and 16,384 stimulation channels, from which we can simultaneously record up to 1024 channels at a given time. Fully implanted below the dura, our chip is wirelessly powered, communicating bi-directionally with an external relay station outside the body. We demonstrated chronic, reliable recordings for up to two weeks in pigs and up to two months in behaving non-human primates from somatosensory, motor, and visual cortices, decoding brain signals at high spatiotemporal resolution.
Collapse
|
4
|
Wu X, Zhang D, Li G, Gao X, Metcalfe B, Chen L. Data augmentation for invasive brain-computer interfaces based on stereo-electroencephalography (SEEG). J Neural Eng 2024; 21:016026. [PMID: 38237174 DOI: 10.1088/1741-2552/ad200e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Accepted: 01/18/2024] [Indexed: 02/23/2024]
Abstract
Objective.Deep learning is increasingly used for brain-computer interfaces (BCIs). However, the quantity of available data is sparse, especially for invasive BCIs. Data augmentation (DA) methods, such as generative models, can help to address this sparseness. However, all the existing studies on brain signals were based on convolutional neural networks and ignored the temporal dependence. This paper attempted to enhance generative models by capturing the temporal relationship from a time-series perspective.Approach. A conditional generative network (conditional transformer-based generative adversarial network (cTGAN)) based on the transformer model was proposed. The proposed method was tested using a stereo-electroencephalography (SEEG) dataset which was recorded from eight epileptic patients performing five different movements. Three other commonly used DA methods were also implemented: noise injection (NI), variational autoencoder (VAE), and conditional Wasserstein generative adversarial network with gradient penalty (cWGANGP). Using the proposed method, the artificial SEEG data was generated, and several metrics were used to compare the data quality, including visual inspection, cosine similarity (CS), Jensen-Shannon distance (JSD), and the effect on the performance of a deep learning-based classifier.Main results. Both the proposed cTGAN and the cWGANGP methods were able to generate realistic data, while NI and VAE outputted inferior samples when visualized as raw sequences and in a lower dimensional space. The cTGAN generated the best samples in terms of CS and JSD and outperformed cWGANGP significantly in enhancing the performance of a deep learning-based classifier (each of them yielding a significant improvement of 6% and 3.4%, respectively).Significance. This is the first time that DA methods have been applied to invasive BCIs based on SEEG. In addition, this study demonstrated the advantages of the model that preserves the temporal dependence from a time-series perspective.
Collapse
Affiliation(s)
- Xiaolong Wu
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Dingguo Zhang
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Guangye Li
- School of Mechanical Engineering, Shanghai Jiao Tong University, People's Republic of China
| | - Xin Gao
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Benjamin Metcalfe
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Liang Chen
- Liang Chen is with Huashan Hospital, Fudan University, People's Republic of China
| |
Collapse
|
5
|
Brannigan JFM, Fry A, Opie NL, Campbell BCV, Mitchell PJ, Oxley TJ. Endovascular Brain-Computer Interfaces in Poststroke Paralysis. Stroke 2024; 55:474-483. [PMID: 38018832 DOI: 10.1161/strokeaha.123.037719] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2023]
Abstract
Stroke is a leading cause of paralysis, most frequently affecting the upper limbs and vocal folds. Despite recent advances in care, stroke recovery invariably reaches a plateau, after which there are permanent neurological impairments. Implantable brain-computer interface devices offer the potential to bypass permanent neurological lesions. They function by (1) recording neural activity, (2) decoding the neural signal occurring in response to volitional motor intentions, and (3) generating digital control signals that may be used to control external devices. While brain-computer interface technology has the potential to revolutionize neurological care, clinical translation has been limited. Endovascular arrays present a novel form of minimally invasive brain-computer interface devices that have been deployed in human subjects during early feasibility studies. This article provides an overview of endovascular brain-computer interface devices and critically evaluates the patient with stroke as an implant candidate. Future opportunities are mapped, along with the challenges arising when decoding neural activity following infarction. Limitations arise when considering intracerebral hemorrhage and motor cortex lesions; however, future directions are outlined that aim to address these challenges.
Collapse
Affiliation(s)
- Jamie F M Brannigan
- Nuffield Department of Clinical Neurosciences, University of Oxford, United Kingdom (J.F.M.B.)
| | - Adam Fry
- Synchron, Inc, New York, NY (A.F., N.L.O., T.J.O.)
| | - Nicholas L Opie
- Synchron, Inc, New York, NY (A.F., N.L.O., T.J.O.)
- Vascular Bionics Laboratory, Department of Medicine, The University of Melbourne, Victoria, Australia (N.L.O., T.J.O.)
| | - Bruce C V Campbell
- Department of Neurology (B.C.V.C.), The Royal Melbourne Hospital, The University of Melbourne, Parkville, Australia
- Melbourne Brain Centre (B.C.V.C.), The Royal Melbourne Hospital, The University of Melbourne, Parkville, Australia
| | - Peter J Mitchell
- Department of Radiology (P.J.M.), The Royal Melbourne Hospital, The University of Melbourne, Parkville, Australia
| | - Thomas J Oxley
- Synchron, Inc, New York, NY (A.F., N.L.O., T.J.O.)
- Vascular Bionics Laboratory, Department of Medicine, The University of Melbourne, Victoria, Australia (N.L.O., T.J.O.)
| |
Collapse
|
6
|
Liu H, Wei P, Wang H, Lv X, Duan W, Li M, Zhao Y, Wang Q, Chen X, Shi G, Han B, Hao J. An EEG motor imagery dataset for brain computer interface in acute stroke patients. Sci Data 2024; 11:131. [PMID: 38272904 PMCID: PMC10811218 DOI: 10.1038/s41597-023-02787-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2022] [Accepted: 11/24/2023] [Indexed: 01/27/2024] Open
Abstract
The brain-computer interface (BCI) is a technology that involves direct communication with parts of the brain and has evolved rapidly in recent years; it has begun to be used in clinical practice, such as for patient rehabilitation. Patient electroencephalography (EEG) datasets are critical for algorithm optimization and clinical applications of BCIs but are rare at present. We collected data from 50 acute stroke patients with wireless portable saline EEG devices during the performance of two tasks: 1) imagining right-handed movements and 2) imagining left-handed movements. The dataset consists of four types of data: 1) the motor imagery instructions, 2) raw recording data, 3) pre-processed data after removing artefacts and other manipulations, and 4) patient characteristics. This is the first open dataset to address left- and right-handed motor imagery in acute stroke patients. We believe that the dataset will be very helpful for analysing brain activation and designing decoding methods that are more applicable for acute stroke patients, which will greatly facilitate research in the field of motor imagery-BCI.
Collapse
Affiliation(s)
- Haijie Liu
- Department of Neurology, Xuanwu Hospital Capital Medical University, Beijing, 100053, China
- National Center for Neurological Disorders, Beijing, 100053, China
| | - Penghu Wei
- National Center for Neurological Disorders, Beijing, 100053, China
- Department of Neurosurgery, Xuanwu Hospital Capital Medical University, Beijing, 100053, China
| | - Haochong Wang
- The Key Laboratory of Biomedical Information Engineering of Ministry of Education, Institute of Health and Rehabilitation Science, School of Life Science and Technology, Xi'an Jiaotong University, Xi'an, Shanxi, 710049, China
| | - Xiaodong Lv
- Department of Neurology, Xuanwu Hospital Capital Medical University, Beijing, 100053, China
- National Center for Neurological Disorders, Beijing, 100053, China
| | - Wei Duan
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100000, China
| | - Meijie Li
- Department of Neurology, Xuanwu Hospital Capital Medical University, Beijing, 100053, China
- National Center for Neurological Disorders, Beijing, 100053, China
| | - Yan Zhao
- Department of Neurology, Xuanwu Hospital Capital Medical University, Beijing, 100053, China
- National Center for Neurological Disorders, Beijing, 100053, China
| | - Qingmei Wang
- Stroke Biological Recovery Laboratory, Department of Physical Medicine and Rehabilitation, Spaulding Rehabilitation Hospital, Harvard Medical School, Charlestown, MA, 02129, USA
| | - Xinyuan Chen
- Donders Institute of Cognition and Behaviour, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Gaige Shi
- The Key Laboratory of Biomedical Information Engineering of Ministry of Education, Institute of Health and Rehabilitation Science, School of Life Science and Technology, Xi'an Jiaotong University, Xi'an, Shanxi, 710049, China
| | - Bo Han
- Department of Neurology, Xuanwu Hospital Capital Medical University, Beijing, 100053, China
- National Center for Neurological Disorders, Beijing, 100053, China
| | - Junwei Hao
- Department of Neurology, Xuanwu Hospital Capital Medical University, Beijing, 100053, China.
- National Center for Neurological Disorders, Beijing, 100053, China.
- Chinese Institute for Brain Research, Beijing, 100053, China.
| |
Collapse
|
7
|
Todorov D, Schnitzler A, Hirschmann J. Parkinsonian rest tremor can be distinguished from voluntary hand movements based on subthalamic and cortical activity. Clin Neurophysiol 2024; 157:146-155. [PMID: 38030516 DOI: 10.1016/j.clinph.2023.10.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Revised: 10/19/2023] [Accepted: 10/31/2023] [Indexed: 12/01/2023]
Abstract
OBJECTIVE To distinguish Parkinsonian rest tremor and different voluntary hand movements by analyzing brain activity. METHODS We re-analyzed magnetoencephalography and local field potential recordings from the subthalamic nucleus of six patients with Parkinson's disease. Data were obtained after withdrawal from dopaminergic medication (Med Off) and after administration of levodopa (Med On). Using gradient-boosted tree learning, we classified epochs as tremor, fist-clenching, forearm extension or tremor-free rest. RESULTS Subthalamic activity alone was insufficient for distinguishing the four different motor states (balanced accuracy mean: 38%, std: 7%). The combination of cortical and subthalamic features, in contrast, allowed for a much more accurate classification (balanced accuracy mean: 75%, std: 17%). Adding a single cortical area improved balanced accuracy by 17% on average, as compared to classification based on subthalamic activity alone. In most patients, the most informative cortical areas were sensorimotor cortical regions. Decoding performance was similar in Med On and Med Off. CONCLUSIONS Electrophysiological recordings allow for distinguishing several motor states, provided that cortical signals are monitored in addition to subthalamic activity. SIGNIFICANCE By combining cortical recordings, subcortical recordings and machine learning, adaptive deep brain stimulation systems might be able to detect tremor specifically and to respond adequately to several motor states.
Collapse
Affiliation(s)
- Dmitrii Todorov
- Institute of Clinical Neuroscience and Medical Psychology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Germany; Centre de Recherche en Neurosciences de Lyon - Inserm U1028, 69675 Bron, France; Centre de Recerca Matemática, Campus UAB edifici C, 08193 Bellaterra, Barcelona, Spain
| | - Alfons Schnitzler
- Institute of Clinical Neuroscience and Medical Psychology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Germany; Center for Movement Disorders and Neuromodulation, Department of Neurology Medical Faculty, Heinrich Heine University, 40225 Düsseldorf, Germany
| | - Jan Hirschmann
- Institute of Clinical Neuroscience and Medical Psychology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Germany.
| |
Collapse
|
8
|
Merino EC, Faes A, Van Hulle MM. The role of distinct ECoG frequency features in decoding finger movement. J Neural Eng 2023; 20:066014. [PMID: 37963397 DOI: 10.1088/1741-2552/ad0c5e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Accepted: 11/14/2023] [Indexed: 11/16/2023]
Abstract
Objective.To identify the electrocorticography (ECoG) frequency features that encode distinct finger movement states during repeated finger flexions.Approach.We used the publicly available Stanford ECoG dataset of cue-based, repeated single finger flexions. Using linear regression, we identified the spectral features that contributed most to the encoding of movement dynamics and discriminating movement events from rest, and combined them to predict finger movement trajectories. Furthermore, we also looked into the effect of the used frequency range and the spatial distribution of the identified features.Main results.Two frequency features generate superior performance, each one for a different movement aspect: high gamma band activity distinguishes movement events from rest, whereas the local motor potential (LMP) codes for movement dynamics. Combining these two features in a finger movement decoder outperformed comparable prior work where the entire spectrum was used as the average correlation coefficient with the true trajectories increased from 0.45 to 0.5, both applied to the Stanford dataset, and erroneous predictions during rest were demoted. In addition, for the first time, our results show the influence of the upper cut-off frequency used to extract LMP, yielding a higher performance when this range is adjusted to the finger movement rate.Significance.This study shows the benefit of a detailed feature analysis prior to designing the finger movement decoder.
Collapse
Affiliation(s)
- Eva Calvo Merino
- Laboratory for Neuro- and Psychophysiology, KU Leuven, Leuven, Belgium
| | - A Faes
- Laboratory for Neuro- and Psychophysiology, KU Leuven, Leuven, Belgium
| | - M M Van Hulle
- Laboratory for Neuro- and Psychophysiology, KU Leuven, Leuven, Belgium
| |
Collapse
|
9
|
Setogawa S, Kanda R, Tada S, Hikima T, Saitoh Y, Ishikawa M, Nakada S, Seki F, Hikishima K, Matsumoto H, Mizuseki K, Fukayama O, Osanai M, Sekiguchi H, Ohkawa N. A novel micro-ECoG recording method for recording multisensory neural activity from the parietal to temporal cortices in mice. Mol Brain 2023; 16:38. [PMID: 37138338 PMCID: PMC10157930 DOI: 10.1186/s13041-023-01019-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Accepted: 03/09/2023] [Indexed: 05/05/2023] Open
Abstract
Characterization of inter-regional interactions in brain is essential for understanding the mechanism relevant to normal brain function and neurological disease. The recently developed flexible micro (μ)-electrocorticography (μECoG) device is one prominent method used to examine large-scale cortical activity across multiple regions. The sheet-shaped μECoG electrodes arrays can be placed on a relatively wide area of cortical surface beneath the skull by inserting the device into the space between skull and brain. Although rats and mice are useful tools for neuroscience, current μECoG recording methods in these animals are limited to the parietal region of cerebral cortex. Recording cortical activity from the temporal region of cortex in mice has proven difficult because of surgical barriers created by the skull and surrounding temporalis muscle anatomy. Here, we developed a sheet-shaped 64-channel μECoG device that allows access to the mouse temporal cortex, and we determined the factor determining the appropriate bending stiffness for the μECoG electrode array. We also established a surgical technique to implant the electrode arrays into the epidural space over a wide area of cerebral cortex covering from the barrel field to olfactory (piriform) cortex, which is the deepest region of the cerebral cortex. Using histology and computed tomography (CT) images, we confirmed that the tip of the μECoG device reached to the most ventral part of cerebral cortex without causing noticeable damage to the brain surface. Moreover, the device simultaneously recorded somatosensory and odor stimulus-evoked neural activity from dorsal and ventral parts of cerebral cortex in awake and anesthetized mice. These data indicate that our μECoG device and surgical techniques enable the recording of large-scale cortical activity from the parietal to temporal cortex in mice, including somatosensory and olfactory cortices. This system will provide more opportunities for the investigation of physiological functions from wider areas of the mouse cerebral cortex than those currently available with existing ECoG techniques.
Collapse
Affiliation(s)
- Susumu Setogawa
- Division for Memory and Cognitive Function, Research Center for Advanced Medical Science, Comprehensive Research Facilities for Advanced Medical Science, Dokkyo Medical University, Tochigi, 321-0293, Japan
- Department of Physiology, Osaka Metropolitan University Graduate School of Medicine, Osaka, 545-8585, Japan
| | - Ryota Kanda
- Department of Electrical and Electronic Information Engineering, Toyohashi University of Technology, Toyohashi, Aichi, 441-8580, Japan
| | - Shuto Tada
- Department of Electrical and Electronic Information Engineering, Toyohashi University of Technology, Toyohashi, Aichi, 441-8580, Japan
| | - Takuya Hikima
- Division for Memory and Cognitive Function, Research Center for Advanced Medical Science, Comprehensive Research Facilities for Advanced Medical Science, Dokkyo Medical University, Tochigi, 321-0293, Japan
| | - Yoshito Saitoh
- Division for Memory and Cognitive Function, Research Center for Advanced Medical Science, Comprehensive Research Facilities for Advanced Medical Science, Dokkyo Medical University, Tochigi, 321-0293, Japan
| | - Mikiko Ishikawa
- Division for Memory and Cognitive Function, Research Center for Advanced Medical Science, Comprehensive Research Facilities for Advanced Medical Science, Dokkyo Medical University, Tochigi, 321-0293, Japan
| | - Satoshi Nakada
- Japanese Center for Research on Women in Sport, Graduate School of Health and Sports Science, Juntendo University, Chiba, 270-1695, Japan
| | - Fumiko Seki
- Live Animal Imaging Center, Central Institutes for Experimental Animals (CIEA), Kanagawa, 210-0821, Japan
| | - Keigo Hikishima
- Medical Devices Research Group, Health and Medical Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, 305-8564, Japan
| | - Hideyuki Matsumoto
- Department of Physiology, Osaka Metropolitan University Graduate School of Medicine, Osaka, 545-8585, Japan
| | - Kenji Mizuseki
- Department of Physiology, Osaka Metropolitan University Graduate School of Medicine, Osaka, 545-8585, Japan
| | - Osamu Fukayama
- Center for Information and Neural Networks (CiNet), Advanced ICT Research Institute, National Institute of Information and Communications Technology, Suita, Osaka, 565-0871, Japan
| | - Makoto Osanai
- Laboratory for Physiological Functional Imaging, Department of Medical Physics and Engineering, Division of Health Sciences, Osaka University Graduate School of Medicine, Suita, Osaka, 565-0871, Japan
| | - Hiroto Sekiguchi
- Department of Electrical and Electronic Information Engineering, Toyohashi University of Technology, Toyohashi, Aichi, 441-8580, Japan.
- Japan Science and Technology, Precursory Research for Embryonic Science and Technology (PRESTO), Kawaguchi, Saitama, 332-0012, Japan.
| | - Noriaki Ohkawa
- Division for Memory and Cognitive Function, Research Center for Advanced Medical Science, Comprehensive Research Facilities for Advanced Medical Science, Dokkyo Medical University, Tochigi, 321-0293, Japan.
| |
Collapse
|
10
|
Śliwowski M, Martin M, Souloumiac A, Blanchart P, Aksenova T. Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance. Front Hum Neurosci 2023; 17:1111645. [PMID: 37007675 PMCID: PMC10061076 DOI: 10.3389/fnhum.2023.1111645] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Accepted: 02/27/2023] [Indexed: 03/18/2023] Open
Abstract
IntroductionIn brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.MethodsWe evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.ResultsOur results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.DiscussionDL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.
Collapse
Affiliation(s)
- Maciej Śliwowski
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- Université Paris-Saclay, CEA, List, Palaiseau, France
| | - Matthieu Martin
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
| | | | | | - Tetiana Aksenova
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- *Correspondence: Tetiana Aksenova
| |
Collapse
|
11
|
Peterson V, Merk T, Bush A, Nikulin V, Kühn AA, Neumann WJ, Richardson RM. Movement decoding using spatio-spectral features of cortical and subcortical local field potentials. Exp Neurol 2023; 359:114261. [PMID: 36349662 DOI: 10.1016/j.expneurol.2022.114261] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2022] [Revised: 09/26/2022] [Accepted: 10/25/2022] [Indexed: 12/30/2022]
Abstract
The first commercially sensing enabled deep brain stimulation (DBS) devices for the treatment of movement disorders have recently become available. In the future, such devices could leverage machine learning based brain signal decoding strategies to individualize and adapt therapy in real-time. As multi-channel recordings become available, spatial information may provide an additional advantage for informing machine learning models. To investigate this concept, we compared decoding performances from single channels vs. spatial filtering techniques using intracerebral multitarget electrophysiology in Parkinson's disease patients undergoing DBS implantation. We investigated the feasibility of spatial filtering in invasive neurophysiology and the putative utility of combined cortical ECoG and subthalamic local field potential signals for decoding grip-force, a well-defined and continuous motor readout. We found that adding spatial information to the model can improve decoding (6% gain in decoding), but the spatial patterns and additional benefit was highly individual. Beyond decoding performance results, spatial filters and patterns can be used to obtain meaningful neurophysiological information about the brain networks involved in target behavior. Our results highlight the importance of individualized approaches for brain signal decoding, for which multielectrode recordings and spatial filtering can improve precision medicine approaches for clinical brain computer interfaces.
Collapse
Affiliation(s)
- Victoria Peterson
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, USA.
| | - Timon Merk
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Alan Bush
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, USA
| | - Vadim Nikulin
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Andrea A Kühn
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Wolf-Julian Neumann
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - R Mark Richardson
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, USA
| |
Collapse
|
12
|
Petrosyan A, Voskoboinikov A, Sukhinin D, Makarova A, Skalnaya A, Arkhipova N, Sinkin M, Ossadtchi A. Speech decoding from a small set of spatially segregated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network. J Neural Eng 2022; 19. [PMID: 36356309 DOI: 10.1088/1741-2552/aca1e1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2022] [Accepted: 11/10/2022] [Indexed: 11/12/2022]
Abstract
Objective. Speech decoding, one of the most intriguing brain-computer interface applications, opens up plentiful opportunities from rehabilitation of patients to direct and seamless communication between human species. Typical solutions rely on invasive recordings with a large number of distributed electrodes implanted through craniotomy. Here we explored the possibility of creating speech prosthesis in a minimally invasive setting with a small number of spatially segregated intracranial electrodes.Approach. We collected one hour of data (from two sessions) in two patients implanted with invasive electrodes. We then used only the contacts that pertained to a single stereotactic electroencephalographic (sEEG) shaft or an electrocorticographic (ECoG) stripe to decode neural activity into 26 words and one silence class. We employed a compact convolutional network-based architecture whose spatial and temporal filter weights allow for a physiologically plausible interpretation.Mainresults. We achieved on average 55% accuracy using only six channels of data recorded with a single minimally invasive sEEG electrode in the first patient and 70% accuracy using only eight channels of data recorded for a single ECoG strip in the second patient in classifying 26+1 overtly pronounced words. Our compact architecture did not require the use of pre-engineered features, learned fast and resulted in a stable, interpretable and physiologically meaningful decision rule successfully operating over a contiguous dataset collected during a different time interval than that used for training. Spatial characteristics of the pivotal neuronal populations corroborate with active and passive speech mapping results and exhibit the inverse space-frequency relationship characteristic of neural activity. Compared to other architectures our compact solution performed on par or better than those recently featured in neural speech decoding literature.Significance. We showcase the possibility of building a speech prosthesis with a small number of electrodes and based on a compact feature engineering free decoder derived from a small amount of training data.
Collapse
Affiliation(s)
- Artur Petrosyan
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia
| | | | - Dmitrii Sukhinin
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia
| | - Anna Makarova
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia
| | | | | | - Mikhail Sinkin
- Moscow State University of Medicine and Dentistry, Scientific Research Institute of First Aid to them. N.V. Sklifosovsky, Moscow, Russia
| | - Alexei Ossadtchi
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia.,Artificial Intelligence Research Institute, AIRI, Moscow, Russia
| |
Collapse
|
13
|
Peterson SM, Rao RPN, Brunton BW. Learning neural decoders without labels using multiple data streams. J Neural Eng 2022; 19. [PMID: 35905727 DOI: 10.1088/1741-2552/ac857c] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2022] [Accepted: 07/29/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Recent advances in neural decoding have accelerated the development of brain-computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding. APPROACH We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Specifically, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models. MAIN RESULTS We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models. Significance: We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.
Collapse
Affiliation(s)
- Steven M Peterson
- Biology, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| | - Rajesh P N Rao
- Department of Computer Science and Engineering College of Engineering, University of Washington, Box 352350, Seattle, Washington, 98195, UNITED STATES
| | - Bingni W Brunton
- University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| |
Collapse
|
14
|
Dhawan V, Cui XT. Carbohydrate based biomaterials for neural interface applications. J Mater Chem B 2022; 10:4714-4740. [PMID: 35702979 DOI: 10.1039/d2tb00584k] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
Abstract
Neuroprosthetic devices that record and modulate neural activities have demonstrated immense potential for bypassing or restoring lost neurological functions due to neural injuries and disorders. However, implantable electrical devices interfacing with brain tissue are susceptible to a series of inflammatory tissue responses along with mechanical or electrical failures which can affect the device performance over time. Several biomaterial strategies have been implemented to improve device-tissue integration for high quality and stable performance. Ranging from developing smaller, softer, and more flexible electrode designs to introducing bioactive coatings and drug-eluting layers on the electrode surface, such strategies have shown different degrees of success but with limitations. With their hydrophilic properties and specific bioactivities, carbohydrates offer a potential solution for addressing some of the limitations of the existing biomolecular approaches. In this review, we summarize the role of polysaccharides in the central nervous system, with a primary focus on glycoproteins and proteoglycans, to shed light on their untapped potential as biomaterials for neural implants. Utilization of glycosaminoglycans for neural interface and tissue regeneration applications is comprehensively reviewed to provide the current state of carbohydrate-based biomaterials for neural implants. Finally, we will discuss the challenges and opportunities of applying carbohydrate-based biomaterials for neural tissue interfaces.
Collapse
Affiliation(s)
- Vaishnavi Dhawan
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. .,Center for Neural Basis of Cognition, Pittsburgh, PA, USA
| | - Xinyan Tracy Cui
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. .,Center for Neural Basis of Cognition, Pittsburgh, PA, USA.,McGowan Institute for Regenerative Medicine, Pittsburgh, PA, USA
| |
Collapse
|
15
|
Merk T, Peterson V, Köhler R, Haufe S, Richardson RM, Neumann WJ. Machine learning based brain signal decoding for intelligent adaptive deep brain stimulation. Exp Neurol 2022; 351:113993. [PMID: 35104499 PMCID: PMC10521329 DOI: 10.1016/j.expneurol.2022.113993] [Citation(s) in RCA: 21] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2021] [Revised: 11/18/2021] [Accepted: 01/22/2022] [Indexed: 12/30/2022]
Abstract
Sensing enabled implantable devices and next-generation neurotechnology allow real-time adjustments of invasive neuromodulation. The identification of symptom and disease-specific biomarkers in invasive brain signal recordings has inspired the idea of demand dependent adaptive deep brain stimulation (aDBS). Expanding the clinical utility of aDBS with machine learning may hold the potential for the next breakthrough in the therapeutic success of clinical brain computer interfaces. To this end, sophisticated machine learning algorithms optimized for decoding of brain states from neural time-series must be developed. To support this venture, this review summarizes the current state of machine learning studies for invasive neurophysiology. After a brief introduction to the machine learning terminology, the transformation of brain recordings into meaningful features for decoding of symptoms and behavior is described. Commonly used machine learning models are explained and analyzed from the perspective of utility for aDBS. This is followed by a critical review on good practices for training and testing to ensure conceptual and practical generalizability for real-time adaptation in clinical settings. Finally, first studies combining machine learning with aDBS are highlighted. This review takes a glimpse into the promising future of intelligent adaptive DBS (iDBS) and concludes by identifying four key ingredients on the road for successful clinical adoption: i) multidisciplinary research teams, ii) publicly available datasets, iii) open-source algorithmic solutions and iv) strong world-wide research collaborations.
Collapse
Affiliation(s)
- Timon Merk
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité - Universitätsmedizin Berlin, Chariteplatz 1, 10117 Berlin, Germany
| | - Victoria Peterson
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, United States
| | - Richard Köhler
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité - Universitätsmedizin Berlin, Chariteplatz 1, 10117 Berlin, Germany
| | - Stefan Haufe
- Berlin Center for Advanced Neuroimaging (BCAN), Charité - Universitätsmedizin Berlin, Chariteplatz 1, 10117 Berlin, Germany
| | - R Mark Richardson
- Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, United States
| | - Wolf-Julian Neumann
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Charité - Universitätsmedizin Berlin, Chariteplatz 1, 10117 Berlin, Germany.
| |
Collapse
|
16
|
Hammer J, Schirrmeister RT, Hartmann K, Marusic P, Schulze-Bonhage A, Ball T. Interpretable functional specialization emerges in deep convolutional networks trained on brain signals. J Neural Eng 2022; 19. [PMID: 35421857 DOI: 10.1088/1741-2552/ac6770] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Accepted: 04/14/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Functional specialization is fundamental to neural information processing. Here, we study whether and how functional specialization emerges in artificial deep convolutional neural networks (CNNs) during a brain-computer interfacing (BCI) task. APPROACH We trained CNNs to predict hand movement speed from intracranial EEG (iEEG) and delineated how units across the different CNN hidden layers learned to represent the iEEG signal. MAIN RESULTS We show that distinct, functionally interpretable neural populations emerged as a result of the training process. While some units became sensitive to either iEEG amplitude or phase, others showed bimodal behavior with significant sensitivity to both features. Pruning of highly-sensitive units resulted in a steep drop of decoding accuracy not observed for pruning of less sensitive units, highlighting the functional relevance of the amplitude- and phase-specialized populations. SIGNIFICANCE We anticipate that emergent functional specialization as uncovered here will become a key concept in research towards interpretable deep learning for neuroscience and BCI applications.
Collapse
Affiliation(s)
- Jiri Hammer
- Neuromedical AI Lab, Department of Neurosurgery, University of Freiburg, Engelbergerstraße 21, Freiburg, 79106, GERMANY
| | | | - Kay Hartmann
- Neuromedical AI Lab, Department of Neurosurgery, University of Freiburg, Engelbergerstraße 21, Freiburg, 79106, GERMANY
| | - Petr Marusic
- Department of Neurology, Motol University Hospital, V Úvalu 84, Prague, 150 06, CZECH REPUBLIC
| | - Andreas Schulze-Bonhage
- Epilepsy Center, University Clinics, Albert-Ludwigs-Universitaet Freiburg, Albert-Ludwigs-University,, 79095 Freiburg, Germany, Freiburg, 79095, GERMANY
| | - Tonio Ball
- Epilepsy Center, University Clinics, Albert-Ludwigs-Universitaet Freiburg, Albert-Ludwigs-University,, 79095 Freiburg, Germany, Freiburg, 79106, GERMANY
| |
Collapse
|
17
|
Kiang L, Woodington B, Carnicer-Lombarte A, Malliaras G, Barone DG. Spinal cord bioelectronic interfaces: opportunities in neural recording and clinical challenges. J Neural Eng 2022; 19. [PMID: 35320780 DOI: 10.1088/1741-2552/ac605f] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Accepted: 03/23/2022] [Indexed: 11/11/2022]
Abstract
Bioelectronic stimulation of the spinal cord has demonstrated significant progress in restoration of motor function in spinal cord injury (SCI). The proximal, uninjured spinal cord presents a viable target for the recording and generation of control signals to drive targeted stimulation. Signals have been directly recorded from the spinal cord in behaving animals and correlated with limb kinematics. Advances in flexible materials, electrode impedance and signal analysis will allow SCR to be used in next-generation neuroprosthetics. In this review, we summarize the technological advances enabling progress in SCR and describe systematically the clinical challenges facing spinal cord bioelectronic interfaces and potential solutions, from device manufacture, surgical implantation to chronic effects of foreign body reaction and stress-strain mismatches between electrodes and neural tissue. Finally, we establish our vision of bi-directional closed-loop spinal cord bioelectronic bypass interfaces that enable the communication of disrupted sensory signals and restoration of motor function in SCI.
Collapse
Affiliation(s)
- Lei Kiang
- Orthopaedic Surgery, Singapore General Hospital, Outram Road, Singapore, Singapore, 169608, SINGAPORE
| | - Ben Woodington
- Department of Engineering, University of Cambridge, Electrical Engineering Division, 9 JJ Thomson Ave, Cambridge, Cambridge, CB2 1TN, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Alejandro Carnicer-Lombarte
- Clinical Neurosciences, University of Cambridge, Bioelectronics Laboratory, Cambridge, CB2 0PY, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - George Malliaras
- University of Cambridge, University of Cambridge, Cambridge, CB2 1TN, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Damiano G Barone
- Department of Engineering, University of Cambridge, Electrical Engineering Division, 9 JJ Thomson Ave, Cambridge, Cambridge, Cambridgeshire, CB2 1TN, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| |
Collapse
|
18
|
Moly A, Costecalde T, Martel F, Martin M, Larzabal C, Karakas S, Verney A, Charvet G, Chabardès S, Benabid AL, Aksenova T. An adaptive closed-loop ECoG decoder for long-term and stable bimanual control of an exoskeleton by a tetraplegic. J Neural Eng 2022; 19. [PMID: 35234665 DOI: 10.1088/1741-2552/ac59a0] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2021] [Accepted: 03/01/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE The article aims at addressing 2 challenges to step motor BCI out of laboratories: asynchronous control of complex bimanual effectors with large numbers of degrees of freedom, using chronic and safe recorders, and the decoding performance stability over time without frequent decoder recalibration. APPROACH Closed-loop adaptive/incremental decoder training is one strategy to create a model stable over time. Adaptive decoders update their parameters with new incoming data, optimizing the model parameters in real time. It allows cross-session training with multiple recording conditions during closed loop BCI experiments. In the article, an adaptive tensor-based Recursive Exponentially Weighted Markov-Switching multi-Linear Model (REW-MSLM) decoder is proposed. REW-MSLM uses a Mixture of Expert (ME) architecture, mixing or switching independent decoders (experts) according to the probability estimated by a "gating" model. A Hidden Markov model approach is employed as gating model to improve the decoding robustness and to provide strong idle state support. The ME architecture fits the multi-limb paradigm associating an expert to a particular limb or action. MAIN RESULTS Asynchronous control of an exoskeleton by a tetraplegic patient using a chronically implanted epidural electrocorticography (EpiCoG) recorder is reported. The stable over a period of 6 months (without decoder recalibration) 8-dimensional alternative bimanual control of the exoskeleton and its virtual avatar is demonstrated. SIGNIFICANCE Based on the long-term (>36 months) chronic bilateral epidural ECoG recordings in a tetraplegic (ClinicalTrials.gov, NCT02550522), we addressed the poorly explored field of asynchronous bimanual BCI. The new decoder was designed to meet to several challenges: the high-dimensional control of a complex effector in experiments closer to real-world behaviour (point-to-point pursuit versus conventional center-out tasks), with the ability of the BCI system to act as a stand-alone device switching between idle and control states, and a stable performance over a long period of time without decoder recalibration.
Collapse
Affiliation(s)
- Alexandre Moly
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Thomas Costecalde
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Félix Martel
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Matthieu Martin
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des Martyrs, Grenoble, 38000, FRANCE
| | - Christelle Larzabal
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Serpil Karakas
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Alexandre Verney
- Université Paris-Saclay, Palaiseau, Palaiseau, Île-de-France, 91120, FRANCE
| | - Guillaume Charvet
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Stephan Chabardès
- CHU Grenoble Alpes, Boulevard de la Chantourne, La Tronche, Auvergne-Rhône-Alpes, 38700, FRANCE
| | - Alim-Louis Benabid
- Univ. Grenoble Alpes, CEA, LETI, Clinatec, 17, avenue des Martyrs, Grenoble, 38000, FRANCE
| | - Tatiana Aksenova
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| |
Collapse
|
19
|
Kim HH, Jeong J. An electrocorticographic decoder for arm movement for brain–machine interface using an echo state network and Gaussian readout. Appl Soft Comput 2022. [DOI: 10.1016/j.asoc.2021.108393] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
|
20
|
Li G, Jiang S, Meng J, Chai G, Wu Z, Fan Z, Hu J, Sheng X, Zhang D, Chen L, Zhu X. Assessing differential representation of hand movements in multiple domains using stereo-electroencephalographic recordings. Neuroimage 2022; 250:118969. [DOI: 10.1016/j.neuroimage.2022.118969] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2021] [Revised: 01/28/2022] [Accepted: 02/02/2022] [Indexed: 01/03/2023] Open
|
21
|
Choi GY, Han CH, Lee HT, Paik NJ, Kim WS, Hwang HJ. An artificial neural-network approach to identify motor hotspot for upper-limb based on electroencephalography: a proof-of-concept study. J Neuroeng Rehabil 2021; 18:176. [PMID: 34930380 PMCID: PMC8686235 DOI: 10.1186/s12984-021-00972-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2021] [Accepted: 12/08/2021] [Indexed: 11/26/2022] Open
Abstract
Background To apply transcranial electrical stimulation (tES) to the motor cortex, motor hotspots are generally identified using motor evoked potentials by transcranial magnetic stimulation (TMS). The objective of this study is to validate the feasibility of a novel electroencephalography (EEG)-based motor-hotspot-identification approach using a machine learning technique as a potential alternative to TMS. Methods EEG data were measured using 63 channels from thirty subjects as they performed a simple finger tapping task. Power spectral densities of the EEG data were extracted from six frequency bands (delta, theta, alpha, beta, gamma, and full) and were independently used to train and test an artificial neural network for motor hotspot identification. The 3D coordinate information of individual motor hotspots identified by TMS were quantitatively compared with those estimated by our EEG-based motor-hotspot-identification approach to assess its feasibility. Results The minimum mean error distance between the motor hotspot locations identified by TMS and our proposed motor-hotspot-identification approach was 0.22 ± 0.03 cm, demonstrating the proof-of-concept of our proposed EEG-based approach. A mean error distance of 1.32 ± 0.15 cm was measured when using only nine channels attached to the middle of the motor cortex, showing the possibility of practically using the proposed motor-hotspot-identification approach based on a relatively small number of EEG channels. Conclusion We demonstrated the feasibility of our novel EEG-based motor-hotspot-identification method. It is expected that our approach can be used as an alternative to TMS for motor hotspot identification. In particular, its usability would significantly increase when using a recently developed portable tES device integrated with an EEG device.
Collapse
Affiliation(s)
- Ga-Young Choi
- Department of Electronics and Information Engineering, Korea University, Sejong, 30019, Republic of Korea
| | - Chang-Hee Han
- Department of Software, College of Software Convergence, Dongseo University, Busan, 47011, South Korea
| | - Hyung-Tak Lee
- Department of Electronics and Information Engineering, Korea University, Sejong, 30019, Republic of Korea.,Interdisciplinary Graduate Program for Artificial Intelligence Smart Convergence Technology, Korea University, Sejong, 30019, South Korea
| | - Nam-Jong Paik
- Department of Rehabilitation Medicine, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongnam-si, 13620, Republic of Korea
| | - Won-Seok Kim
- Department of Rehabilitation Medicine, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongnam-si, 13620, Republic of Korea.
| | - Han-Jeong Hwang
- Department of Electronics and Information Engineering, Korea University, Sejong, 30019, Republic of Korea. .,Interdisciplinary Graduate Program for Artificial Intelligence Smart Convergence Technology, Korea University, Sejong, 30019, South Korea.
| |
Collapse
|
22
|
Gaudry KS, Ayaz H, Bedows A, Celnik P, Eagleman D, Grover P, Illes J, Rao RPN, Robinson JT, Thyagarajan K. Projections and the Potential Societal Impact of the Future of Neurotechnologies. Front Neurosci 2021; 15:658930. [PMID: 34867139 PMCID: PMC8634831 DOI: 10.3389/fnins.2021.658930] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2021] [Accepted: 10/04/2021] [Indexed: 12/17/2022] Open
Abstract
Traditionally, recording from and stimulating the brain with high spatial and temporal resolution required invasive means. However, recently, the technical capabilities of less invasive and non-invasive neuro-interfacing technology have been dramatically improving, and laboratories and funders aim to further improve these capabilities. These technologies can facilitate functions such as multi-person communication, mood regulation and memory recall. We consider a potential future where the less invasive technology is in high demand. Will this demand match that the current-day demand for a smartphone? Here, we draw upon existing research to project which particular neuroethics issues may arise in this potential future and what preparatory steps may be taken to address these issues.
Collapse
Affiliation(s)
- Kate S. Gaudry
- Kilpatrick Townsend & Stockton LLP, Washington, DC, United States
| | - Hasan Ayaz
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, United States
- Department of Psychology, College of Arts and Sciences, Drexel University, Philadelphia, PA, United States
- Drexel Solutions Institute, Drexel University, Philadelphia, PA, United States
- Department of Family and Community Health, University of Pennsylvania, Philadelphia, PA, United States
- Center for Injury Research and Prevention, Children’s Hospital of Philadelphia, Philadelphia, PA, United States
| | | | - Pablo Celnik
- Department of Physical Medicine and Rehabilitation, Johns Hopkins, School of Medicine, Baltimore, MD, United States
| | - David Eagleman
- Department of Psychiatry, Stanford University School of Medicine, Stanford, CA, United States
| | - Pulkit Grover
- Center for the Neural Basis of Cognition, Carnegie Mellon University, Pittsburgh, PA, United States
- Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Judy Illes
- Department of Medicine, University of British Columbia, Vancouver, BC, Canada
- Neuroethics Canada, University of British Columbia, Vancouver, BC, Canada
| | - Rajesh P. N. Rao
- Center for Neurotechnology, Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, DC, United States
| | - Jacob T. Robinson
- Department of Bioengineering, Rice University, Houston, TX, United States
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, United States
- Applied Physics Program, Rice University, Houston, TX, United States
- Department of Neuroscience, Baylor College of Medicine, Houston, TX, United States
| | | |
Collapse
|
23
|
Liu S, Li G, Jiang S, Wu X, Hu J, Zhang D, Chen L. Investigating Data Cleaning Methods to Improve Performance of Brain-Computer Interfaces Based on Stereo-Electroencephalography. Front Neurosci 2021; 15:725384. [PMID: 34690673 PMCID: PMC8528199 DOI: 10.3389/fnins.2021.725384] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2021] [Accepted: 09/01/2021] [Indexed: 11/13/2022] Open
Abstract
Stereo-electroencephalography (SEEG) utilizes localized and penetrating depth electrodes to directly measure electrophysiological brain activity. The implanted electrodes generally provide a sparse sampling of multiple brain regions, including both cortical and subcortical structures, making the SEEG neural recordings a potential source for the brain–computer interface (BCI) purpose in recent years. For SEEG signals, data cleaning is an essential preprocessing step in removing excessive noises for further analysis. However, little is known about what kinds of effect that different data cleaning methods may exert on BCI decoding performance and, moreover, what are the reasons causing the differentiated effects. To address these questions, we adopted five different data cleaning methods, including common average reference, gray–white matter reference, electrode shaft reference, bipolar reference, and Laplacian reference, to process the SEEG data and evaluated the effect of these methods on improving BCI decoding performance. Additionally, we also comparatively investigated the changes of SEEG signals induced by these different methods from multiple-domain (e.g., spatial, spectral, and temporal domain). The results showed that data cleaning methods could improve the accuracy of gesture decoding, where the Laplacian reference produced the best performance. Further analysis revealed that the superiority of the data cleaning method with excellent performance might be attributed to the increased distinguishability in the low-frequency band. The findings of this work highlighted the importance of applying proper data clean methods for SEEG signals and proposed the application of Laplacian reference for SEEG-based BCI.
Collapse
Affiliation(s)
- Shengjie Liu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, China
| | - Guangye Li
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, China
| | - Shize Jiang
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| | - Xiaolong Wu
- Department of Electronic and Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Jie Hu
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| | - Dingguo Zhang
- Department of Electronic and Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Liang Chen
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| |
Collapse
|
24
|
Olsen S, Alder G, Williams M, Chambers S, Jochumsen M, Signal N, Rashid U, Niazi IK, Taylor D. Electroencephalographic Recording of the Movement-Related Cortical Potential in Ecologically Valid Movements: A Scoping Review. Front Neurosci 2021; 15:721387. [PMID: 34650399 PMCID: PMC8505671 DOI: 10.3389/fnins.2021.721387] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2021] [Accepted: 08/27/2021] [Indexed: 12/05/2022] Open
Abstract
The movement-related cortical potential (MRCP) is a brain signal that can be recorded using surface electroencephalography (EEG) and represents the cortical processes involved in movement preparation. The MRCP has been widely researched in simple, single-joint movements, however, these movements often lack ecological validity. Ecological validity refers to the generalizability of the findings to real-world situations, such as neurological rehabilitation. This scoping review aimed to synthesize the research evidence investigating the MRCP in ecologically valid movement tasks. A search of six electronic databases identified 102 studies that investigated the MRCP during multi-joint movements; 59 of these studies investigated ecologically valid movement tasks and were included in the review. The included studies investigated 15 different movement tasks that were applicable to everyday situations, but these were largely carried out in healthy populations. The synthesized findings suggest that the recording and analysis of MRCP signals is possible in ecologically valid movements, however the characteristics of the signal appear to vary across different movement tasks (i.e., those with greater complexity, increased cognitive load, or a secondary motor task) and different populations (i.e., expert performers, people with Parkinson’s Disease, and older adults). The scarcity of research in clinical populations highlights the need for further research in people with neurological and age-related conditions to progress our understanding of the MRCPs characteristics and to determine its potential as a measure of neurological recovery and intervention efficacy. MRCP-based neuromodulatory interventions applied during ecologically valid movements were only represented in one study in this review as these have been largely delivered during simple joint movements. No studies were identified that used ecologically valid movements to control BCI-driven external devices; this may reflect the technical challenges associated with accurately classifying functional movements from MRCPs. Future research investigating MRCP-based interventions should use movement tasks that are functionally relevant to everyday situations. This will facilitate the application of this knowledge into the rehabilitation setting.
Collapse
Affiliation(s)
- Sharon Olsen
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand
| | - Gemma Alder
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand
| | - Mitra Williams
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand
| | - Seth Chambers
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand
| | - Mads Jochumsen
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Nada Signal
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand
| | - Usman Rashid
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand.,Centre for Chiropractic Research, New Zealand College of Chiropractic, Auckland, New Zealand
| | - Imran Khan Niazi
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand.,Department of Health Science and Technology, Aalborg University, Aalborg, Denmark.,Centre for Chiropractic Research, New Zealand College of Chiropractic, Auckland, New Zealand
| | - Denise Taylor
- Rehabilitation Innovation Centre, Health and Rehabilitation Research Institute, Auckland University of Technology, Auckland, New Zealand
| |
Collapse
|
25
|
Abstract
Brain-machine interfaces (BMI) are being developed to restore upper limb function for persons with spinal cord injury or other motor degenerative conditions. BMI and implantable sensors for myoelectric prostheses directly extract information from the central or peripheral nervous system to provide users with high fidelity control of their prosthetic device. Control algorithms have been highly transferable between the 2 technologies but also face common issues. In this review of the current state of the art in each field, the authors point out similarities and differences between the 2 technologies that may guide the implementation of common solutions to these challenges.
Collapse
Affiliation(s)
- Alex K Vaskov
- Robotics Institute, University of Michigan, 2505 Hayward St, Ann Arbor, MI 48109, USA
| | - Cynthia A Chestek
- Robotics Institute, University of Michigan, 2505 Hayward St, Ann Arbor, MI 48109, USA; Department of Biomedical Engineering, University of Michigan, 2200 Bonisteel Blvd, Ann Arbor, MI 48109, USA; Department of Electrical Engineering and Computer Science, University of Michigan, 1301 Beal Ave, Ann Arbor, MI 48109, USA; Neuroscience Graduate Program, University of Michigan, 204 Washtenaw Ave, Ann Arbor, MI 48109, USA.
| |
Collapse
|
26
|
Arif S, Khan MJ, Naseer N, Hong KS, Sajid H, Ayaz Y. Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain-Computer Interface. Front Hum Neurosci 2021; 15:658444. [PMID: 33994983 PMCID: PMC8121150 DOI: 10.3389/fnhum.2021.658444] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2021] [Accepted: 03/09/2021] [Indexed: 11/13/2022] Open
Abstract
A passive brain-computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects' data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.
Collapse
Affiliation(s)
- Saad Arif
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan
| | - Muhammad Jawad Khan
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan.,National Center of Artificial Intelligence (NCAI), Islamabad, Pakistan
| | - Noman Naseer
- Department of Mechatronics Engineering, Air University, Islamabad, Pakistan
| | - Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| | - Hasan Sajid
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan.,National Center of Artificial Intelligence (NCAI), Islamabad, Pakistan
| | - Yasar Ayaz
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan.,National Center of Artificial Intelligence (NCAI), Islamabad, Pakistan
| |
Collapse
|
27
|
Singh SH, Peterson SM, Rao RPN, Brunton BW. Mining naturalistic human behaviors in long-term video and neural recordings. J Neurosci Methods 2021; 358:109199. [PMID: 33910024 DOI: 10.1016/j.jneumeth.2021.109199] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2020] [Revised: 04/07/2021] [Accepted: 04/19/2021] [Indexed: 10/21/2022]
Abstract
BACKGROUND Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. NEW METHOD Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video. RESULTS Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7-9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain-computer interfacing applications, and we show decoding results from a movement initiation detection task. COMPARISON WITH EXISTING METHODS Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution. CONCLUSIONS Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements.
Collapse
Affiliation(s)
- Satpreet H Singh
- Department of Electrical and Computer Engineering, University of Washington, Seattle, USA
| | - Steven M Peterson
- Department of Biology, University of Washington, Seattle, USA; eScience Institute, University of Washington, Seattle, USA
| | - Rajesh P N Rao
- Department of Electrical and Computer Engineering, University of Washington, Seattle, USA; Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, USA; Center for Neurotechnology, University of Washington, Seattle, USA; University of Washington Institute for Neuroengineering, Seattle, USA
| | - Bingni W Brunton
- Department of Biology, University of Washington, Seattle, USA; eScience Institute, University of Washington, Seattle, USA; University of Washington Institute for Neuroengineering, Seattle, USA.
| |
Collapse
|
28
|
Gao H, Sun M, Li M, Wang C, Yu C, Wang Y, Xu K. Force Decoding of Caudal Forelimb Area and Rostral Forelimb Area in Chronic Stroke Rats. IEEE Trans Biomed Eng 2021; 68:3078-3086. [PMID: 33661731 DOI: 10.1109/tbme.2021.3063903] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Brain machine interfaces (BMIs) used for movement restoration primarily rely on studies of motor decoding. It has been proved that local field potentials (LFPs) from primary motor cortex and premotor cortex of normal rodents could be used for decoding motor signals. However, few studies have explored the decoding performance of these brain areas under motor cortex damage. In this work, we focus on force decoding performance of LFPs spectrum from both ipsilesional caudal forelimb area (CFA) and rostral forelimb area (RFA) of rodents with ischemia over CFA. After three months of ischemia induced by photothrombosis over CFA, the power of high-frequency bands (>120 Hz) from both CFA and RFA can decode force signals by Kalman filters. The fair performance of CFA indicates motor reorganization over penumbra. Further exploration of RFA decoding ability proves that at least four electrodes of RFA should be used on decoding and electrodes far from CFA of stroke rats could achieve almost as good results as those close to CFA of normal rats, which indicates the motor remapping. Experimental results show the long-term stability of PM LFPs decoding performance of stroke rats as the trained Kalman model could be used to accurately decode force some days later which provides a possibility for online decoding system. In conclusion, our work shows that even under CFA ischemia, high-frequency power of LFPs from RFA is still able to accurately decode force signals and has long stability, which provides the possibility of BMIs for motor function reconstruction of chronic stroke patients.
Collapse
|
29
|
Petrosyan A, Sinkin M, Lebedev MA, Ossadtchi A. Decoding and interpreting cortical signals with a compact convolutional neural network. J Neural Eng 2021; 18. [PMID: 33524962 DOI: 10.1088/1741-2552/abe20e] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2020] [Accepted: 02/01/2021] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) decode information from neural activity and send it to external devices. The use of Deep Learning approaches for decoding allows for automatic feature engineering within the specific decoding task. Physiologically plausible interpretation of the network parameters ensures the robustness of the learned decision rules and opens the exciting opportunity for automatic knowledge discovery. APPROACH We describe a compact convolutional network-based architecture for adaptive decoding of electrocorticographic (ECoG) data into finger kinematics. We also propose a novel theoretically justified approach to interpreting the spatial and temporal weights in the architectures that combine adaptation in both space and time. The obtained spatial and frequency patterns characterizing the neuronal populations pivotal to the specific decoding task can then be interpreted by fitting appropriate spatial and dynamical models. MAIN RESULTS We first tested our solution using realistic Monte-Carlo simulations. Then, when applied to the ECoG data from Berlin BCI competition IV dataset, our architecture performed comparably to the competition winners without requiring explicit feature engineering. Using the proposed approach to the network weights interpretation we could unravel the spatial and the spectral patterns of the neuronal processes underlying the successful decoding of finger kinematics from an ECoG dataset. Finally we have also applied the entire pipeline to the analysis of a 32-channel EEG motor-imagery dataset and observed physiologically plausible patterns specific to the task. SIGNIFICANCE We described a compact and interpretable CNN architecture derived from the basic principles and encompassing the knowledge in the field of neural electrophysiology. For the first time in the context of such multibranch architectures with factorized spatial and temporal processing we presented theoretically justified weights interpretation rules. We verified our recipes using simulations and real data and demonstrated that the proposed solution offers a good decoder and a tool for investigating motor control neural mechanisms.
Collapse
Affiliation(s)
- Artur Petrosyan
- Center for Bioelectric Interfaces, Higher School of Economics, Krivokolennyi per., 3, Moscow, Russia, 10100, RUSSIAN FEDERATION
| | - Mikhail Sinkin
- A I Yevdokimov Moscow State University of Medicine and Dentistry of the Ministry of Healthcare of the Russian Federation Faculty of Dentistry, Delegatskaya St., 20, p. 1, Moskva, Moskva, 127473, RUSSIAN FEDERATION
| | - M A Lebedev
- Neurobiology, Duke University, Hudson Hall 136, Durham, NC 27708-0281, USA, Durham, 27517, UNITED STATES
| | - Alexei Ossadtchi
- Center for Bioelectric Interfaces, Higher School of Economics, Krivokolennyi per., 3, Moscow, Russia, 101000, RUSSIAN FEDERATION
| |
Collapse
|
30
|
Peterson SM, Steine-Hanson Z, Davis N, Rao RPN, Brunton BW. Generalized neural decoders for transfer learning across participants and recording modalities. J Neural Eng 2021; 18. [PMID: 33418552 DOI: 10.1088/1741-2552/abda0b] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2020] [Accepted: 01/08/2021] [Indexed: 02/07/2023]
Abstract
OBJECTIVE Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants. APPROACH We introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant. MAIN RESULTS HTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet's trained weights and demonstrate its ability to extract physiologically-relevant features. SIGNIFICANCE By generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.
Collapse
Affiliation(s)
- Steven M Peterson
- Biology, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| | - Zoe Steine-Hanson
- Computer Science and Engineering, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| | - Nathan Davis
- Computer Science and Engineering, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| | - Rajesh P N Rao
- Computer Science and Engineering, University of Washington, 185 E Stevens Way NE, Seattle, Washington, 98195, UNITED STATES
| | - Bingni W Brunton
- Biology, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| |
Collapse
|
31
|
Foodeh R, Ebadollahi S, Daliri MR. Regularized Partial Least Square Regression for Continuous Decoding in Brain-Computer Interfaces. Neuroinformatics 2020; 18:465-477. [PMID: 32107734 DOI: 10.1007/s12021-020-09455-x] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
Continuous decoding is a crucial step in many types of brain-computer interfaces (BCIs). Linear regression techniques have been widely used to determine a linear relation between the input and desired output. A serious issue in this technique is the over-fitting phenomenon. Partial least square (PLS) is a well-known and popular method which tries to overcome this problem. PLS calculates a set of latent variables which are maximally correlated to the output and determines a linear relation between a low-rank estimation of the input and output data. However, this method has shown its potential to overfit the training data in many cases. In this paper, a regularized version of PLS (RPLS) is proposed which tries to determine a linear relation between the latent vector of the input and desired output using the regularized least square instead of the ordinary one. This approach is able to control the effect of non-efficient and non-generalized latent vectors in prediction. We have shown that the proposed method outperforms Ridge regression (RR), PLS, and PLS with regularized weights (PLSRW) in estimating the output in two different real BCI datasets, Neurotycho public electrocorticogram (ECoG) dataset for decoding trajectory of hand movements in monkeys and our own local field potential (LFP) dataset for decoding applied force performed by rats. Furthermore, the results indicate that RPLS is more robust against the increase in the number of latent vectors compared to PLS and PLSRW. Next, we evaluated the resistance of our proposed method against the presence of different noise levels in a BCI application and compared it to other techniques using a semi-simulated dataset. This approach revealed that RPLS offered a higher performance compared with other techniques in all levels of noise. Finally, to illustrate the usability of RPLS in other type of data, we presented the application of this method in predicting relative active substance content of pharmaceutical tablets using near-infrared (NIR) transmittance spectroscopy data. This application showed a superior performance of the proposed method compared to other decoding methods.
Collapse
Affiliation(s)
- Reza Foodeh
- Neuroscience and Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science & Technology (IUST), Narmak, Tehran, 16846-13114, Iran
| | - Saeed Ebadollahi
- Control Engineering Department, School of Electrical Engineering, Iran University of Science & Technology (IUST), Tehran, Iran
| | - Mohammad Reza Daliri
- Neuroscience and Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science & Technology (IUST), Narmak, Tehran, 16846-13114, Iran.
| |
Collapse
|
32
|
Ahmadi N, Constandinou T, Bouganis CS. Impact of referencing scheme on decoding performance of LFP-based brain-machine interface. J Neural Eng 2020; 18. [PMID: 33242850 DOI: 10.1088/1741-2552/abce3c] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2020] [Accepted: 11/26/2020] [Indexed: 12/19/2022]
Abstract
OBJECTIVE There has recently been an increasing interest in local field potential (LFP) for brain-machine interface (BMI) applications due to its desirable properties (signal stability and low bandwidth). LFP is typically recorded with respect to a single unipolar reference which is susceptible to common noise. Several referencing schemes have been proposed to eliminate the common noise, such as bipolar reference, current source density (CSD), and common average reference (CAR). However, to date, there have not been any studies to investigate the impact of these referencing schemes on decoding performance of LFP-based BMIs. APPROACH To address this issue, we comprehensively examined the impact of different referencing schemes and LFP features on the performance of hand kinematic decoding using a deep learning method. We used LFPs chronically recorded from the motor cortex area of a monkey while performing reaching tasks. MAIN RESULTS Experimental results revealed that local motor potential (LMP) emerged as the most informative feature regardless of the referencing schemes. Using LMP as the feature, CAR was found to yield consistently better decoding performance than other referencing schemes over long-term recording sessions. Significance Overall, our results suggest the potential use of LMP coupled with CAR for enhancing the decoding performance of LFP-based BMIs.
Collapse
Affiliation(s)
- Nur Ahmadi
- Electrical and Electronic Engineering, Imperial College London, South Kensington Campus, London, London, SW7 2AZ, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Timothy Constandinou
- Electrical & Electronic Engineering, Imperial College London, London, London, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Christos-Savvas Bouganis
- Electrical and Electronic Engineering, Imperial College London, London, London, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| |
Collapse
|
33
|
Ranjandish R, Schmid A. A Review of Microelectronic Systems and Circuit Techniques for Electrical Neural Recording Aimed at Closed-Loop Epilepsy Control. SENSORS (BASEL, SWITZERLAND) 2020; 20:E5716. [PMID: 33050032 PMCID: PMC7583980 DOI: 10.3390/s20195716] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/29/2020] [Revised: 09/07/2020] [Accepted: 10/02/2020] [Indexed: 12/21/2022]
Abstract
Closed-loop implantable electronics offer a new trend in therapeutic systems aimed at controlling some neurological diseases such as epilepsy. Seizures are detected and electrical stimulation applied to the brain or groups of nerves. To this aim, the signal recording chain must be very carefully designed so as to operate in low-power and low-latency, while enhancing the probability of correct event detection. This paper reviews the electrical characteristics of the target brain signals pertaining to epilepsy detection. Commercial systems are presented and discussed. Finally, the major blocks of the signal acquisition chain are presented with a focus on the circuit architecture and a careful attention to solutions to issues related to data acquisition from multi-channel arrays of cortical sensors.
Collapse
Affiliation(s)
- Reza Ranjandish
- Department of Information Technology and Electrical Engineering, ETH Zürich, CH-8092 Zürich, Switzerland;
| | - Alexandre Schmid
- Institute of Electrical Engineering, EPF Lausanne, CH-1015 Lausanne, Switzerland
| |
Collapse
|
34
|
Shiraishi Y, Kawahara Y, Yamashita O, Fukuma R, Yamamoto S, Saitoh Y, Kishima H, Yanagisawa T. Neural decoding of electrocorticographic signals using dynamic mode decomposition. J Neural Eng 2020; 17:036009. [DOI: 10.1088/1741-2552/ab8910] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
|
35
|
Idowu OP, Huang J, Zhao Y, Samuel OW, Yu M, Fang P, Li G. A stacked sparse auto-encoder and back propagation network model for sensory event detection via a flexible ECoG. Cogn Neurodyn 2020; 14:591-607. [PMID: 33014175 DOI: 10.1007/s11571-020-09603-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2019] [Revised: 04/22/2020] [Accepted: 05/22/2020] [Indexed: 01/22/2023] Open
Abstract
Current prostheses are limited in their ability to provide direct sensory feedback to users with missing limb. Several efforts have been made to restore tactile sensation to amputees but the somatotopic tactile feedback often results in unnatural sensations, and it is yet unclear how and what information the somatosensory system receives during voluntary movement. The present study proposes an efficient model of stacked sparse autoencoder and back propagation neural network for detecting sensory events from a highly flexible electrocorticography (ECoG) electrode. During the mechanical stimulation with Von Frey (VF) filament on the plantar surface of rats' foot, simultaneous recordings of tactile afferent signals were obtained from primary somatosensory cortex (S1) in the brain. In order to achieve a model with optimal performance, Particle Swarm Optimization and Adaptive Moment Estimation (Adam) were adopted to select the appropriate number of neurons, hidden layers and learning rate of each sparse auto-encoder. We evaluated the stimulus-evoked sensation by using an automated up-down (UD) method otherwise called UDReader. The assessment of tactile thresholds with VF shows that the right side of the hind-paw was significantly more sensitive at the tibia-(p = 6.50 × 10-4), followed by the saphenous-(p = 7.84 × 10-4), and sural-(p = 8.24 × 10-4). We then validated our proposed model by comparing with the state-of-the-art methods, and recorded accuracy of 98.8%, sensitivity of 96.8%, and specificity of 99.1%. Hence, we demonstrated the effectiveness of our algorithms in detecting sensory events through flexible ECoG recordings which could be a viable option in restoring somatosensory feedback.
Collapse
Affiliation(s)
- Oluwagbenga Paul Idowu
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen Engineering Laboratory of Neural Rehabilitation Technology, Shenzhen, 518055 China
| | - Jianping Huang
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen Engineering Laboratory of Neural Rehabilitation Technology, Shenzhen, 518055 China
| | - Yang Zhao
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen Engineering Laboratory of Neural Rehabilitation Technology, Shenzhen, 518055 China
| | - Oluwarotimi William Samuel
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen Engineering Laboratory of Neural Rehabilitation Technology, Shenzhen, 518055 China
| | - Mei Yu
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen Engineering Laboratory of Neural Rehabilitation Technology, Shenzhen, 518055 China
| | - Peng Fang
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen Engineering Laboratory of Neural Rehabilitation Technology, Shenzhen, 518055 China
| | - Guanglin Li
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, 518055 China.,Shenzhen Engineering Laboratory of Neural Rehabilitation Technology, Shenzhen, 518055 China
| |
Collapse
|