1
|
Cao B, Huang Y, Chen L, Jia W, Li D, Jiang Y. Soft bioelectronics for diagnostic and therapeutic applications in neurological diseases. Biosens Bioelectron 2024; 259:116378. [PMID: 38759308 DOI: 10.1016/j.bios.2024.116378] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2024] [Revised: 04/13/2024] [Accepted: 05/09/2024] [Indexed: 05/19/2024]
Abstract
Physical and chemical signals in the central nervous system yield crucial information that is clinically relevant under both physiological and pathological conditions. The emerging field of bioelectronics focuses on the monitoring and manipulation of neurophysiological signals with high spatiotemporal resolution and minimal invasiveness. Significant advances have been realized through innovations in materials and structural design, which have markedly enhanced mechanical and electrical properties, biocompatibility, and overall device performance. The diagnostic and therapeutic potential of soft bioelectronics has been corroborated across a diverse array of pre-clinical settings. This review summarizes recent studies that underscore the developments and applications of soft bioelectronics in neurological disorders, including neuromonitoring, neuromodulation, tumor treatment, and biosensing. Limitations and outlooks of soft devices are also discussed in terms of power supply, wireless control, biocompatibility, and the integration of artificial intelligence. This review highlights the potential of soft bioelectronics as a future platform to promote deciphering brain functions and clinical outcomes of neurological diseases.
Collapse
Affiliation(s)
- Bowen Cao
- Department of Neurosurgery, Beijing Tiantan Hospital, National Center for Neurological Disorders, Capital Medical University, Beijing, China; Department of Materials Science and Engineering, University of Pennsylvania, Philadelphia, United States
| | - Yewei Huang
- Department of Materials Science and Engineering, University of Pennsylvania, Philadelphia, United States
| | - Liangpeng Chen
- Department of Neurosurgery, Beijing Tiantan Hospital, National Center for Neurological Disorders, Capital Medical University, Beijing, China
| | - Wang Jia
- Department of Neurosurgery, Beijing Tiantan Hospital, National Center for Neurological Disorders, Capital Medical University, Beijing, China; Beijing Neurosurgical Institute, Capital Medical University, Beijing, China; China National Clinical Research Center for Neurological Diseases (NCRC-ND), Beijing, China.
| | - Deling Li
- Department of Neurosurgery, Beijing Tiantan Hospital, National Center for Neurological Disorders, Capital Medical University, Beijing, China; Beijing Neurosurgical Institute, Capital Medical University, Beijing, China; China National Clinical Research Center for Neurological Diseases (NCRC-ND), Beijing, China.
| | - Yuanwen Jiang
- Department of Materials Science and Engineering, University of Pennsylvania, Philadelphia, United States.
| |
Collapse
|
2
|
Tesink V, Douglas T, Forsberg L, Ligthart S, Meynen G. Right to mental integrity and neurotechnologies: implications of the extended mind thesis. JOURNAL OF MEDICAL ETHICS 2024:jme-2023-109645. [PMID: 38408854 DOI: 10.1136/jme-2023-109645] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Accepted: 01/14/2024] [Indexed: 02/28/2024]
Abstract
The possibility of neurotechnological interference with our brain and mind raises questions about the moral rights that would protect against the (mis)use of these technologies. One such moral right that has received recent attention is the right to mental integrity. Though the metaphysical boundaries of the mind are a matter of live debate, most defences of this moral right seem to assume an internalist (brain-based) view of the mind. In this article, we will examine what an extended account of the mind might imply for the right to mental integrity and the protection it provides against neurotechnologies. We argue that, on an extended account of the mind, the scope of the right to mental integrity would expand significantly, implying that neurotechnologies would no longer pose a uniquely serious threat to the right. In addition, some neurotechnologies may even be protected by the right to mental integrity, as the technologies would become part of the mind. We conclude that adopting an extended account of the mind has significant implications for the right to mental integrity in terms of its protective scope and capacity to protect against neurotechnologies, demonstrating that metaphysical assumptions about the mind play an important role in determining the moral protection provided by the right.
Collapse
Affiliation(s)
- Vera Tesink
- Department of Philosophy, Faculty of Humanities, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
| | - Thomas Douglas
- Oxford Uehiro Centre for Practical Ethics, Faculty of Philosophy, University of Oxford, Oxford, UK
- Jesus College, University of Oxford, Oxford, UK
| | - Lisa Forsberg
- Oxford Uehiro Centre for Practical Ethics, Faculty of Philosophy, University of Oxford, Oxford, UK
| | - Sjors Ligthart
- Department of Criminal Law, Tilburg University, Tilburg, Netherlands
- Willem Pompe Institute for Criminal Law and Criminology and UCALL, Utrecht University, Utrecht, Netherlands
| | - Gerben Meynen
- Department of Philosophy, Faculty of Humanities, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Willem Pompe Institute for Criminal Law and Criminology and UCALL, Utrecht University, Utrecht, Netherlands
| |
Collapse
|
3
|
Oxley TJ. The Promise of Endovascular Neurotechnology: A Brain-Computer Interface to Restore Autonomy to People With Motor Impairment. Am J Phys Med Rehabil 2024; 103:465-470. [PMID: 38377064 DOI: 10.1097/phm.0000000000002463] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/22/2024]
Abstract
ABSTRACT This Joel A. DeLisa Lecture on endovascular brain-computer interfaces was presented by Dr Thomas Oxley on February 23, 2023, at the Association of Academic Physiatrists Annual Scientific Meeting. The lecture described how brain-computer interfaces replace lost physiological function to enable direct communication between the brain and external digital devices, such as computers, smartphones, and robotic limbs. Specifically, the potential of a novel endovascular brain-computer interface technology was discussed. The brain-computer interface uses a stent-electrode array delivered via the jugular vein and is permanently implanted in a vein adjacent to the motor cortex. In a first-in-human clinical trial, participants with upper limb paralysis who received the endovascular brain-computer interface could use the system independently and at home to operate laptop computers for various instrumental activities of daily living. A Food and Drug Administration-approved trial of the endovascular brain-computer interface in the United States is in progress. Future development of the system will provide recipients with continuous autonomy through digital access with minimal caregiver assistance. Physiatrists and occupational therapists will have a vital role in helping people with paralysis achieve the potential of implantable brain-computer interfaces.
Collapse
Affiliation(s)
- Thomas J Oxley
- From the Vascular Bionics Laboratory, Department of Medicine, The University of Melbourne, Melbourne, Australia
| |
Collapse
|
4
|
Young MJ, Kazazian K, Fischer D, Lissak IA, Bodien YG, Edlow BL. Disclosing Results of Tests for Covert Consciousness: A Framework for Ethical Translation. Neurocrit Care 2024; 40:865-878. [PMID: 38243150 PMCID: PMC11147696 DOI: 10.1007/s12028-023-01899-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2023] [Accepted: 11/22/2023] [Indexed: 01/21/2024]
Abstract
The advent of neurotechnologies including advanced functional magnetic resonance imaging and electroencephalography to detect states of awareness not detectable by traditional bedside neurobehavioral techniques (i.e., covert consciousness) promises to transform neuroscience research and clinical practice for patients with brain injury. As these interventions progress from research tools into actionable, guideline-endorsed clinical tests, ethical guidance for clinicians on how to responsibly communicate the sensitive results they yield is crucial yet remains underdeveloped. Drawing on insights from empirical and theoretical neuroethics research and our clinical experience with advanced neurotechnologies to detect consciousness in behaviorally unresponsive patients, we critically evaluate ethical promises and perils associated with disclosing the results of clinical covert consciousness assessments and describe a semistructured approach to responsible data sharing to mitigate potential risks.
Collapse
Affiliation(s)
- Michael J Young
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital and Harvard Medical School, 101 Merrimac Street, Suite 310, Boston, MA, 02114, USA.
| | - Karnig Kazazian
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital and Harvard Medical School, 101 Merrimac Street, Suite 310, Boston, MA, 02114, USA
- Western Institute of Neuroscience, Western University, London, ON, Canada
| | - David Fischer
- University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA
| | - India A Lissak
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital and Harvard Medical School, 101 Merrimac Street, Suite 310, Boston, MA, 02114, USA
| | - Yelena G Bodien
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital and Harvard Medical School, 101 Merrimac Street, Suite 310, Boston, MA, 02114, USA
| | - Brian L Edlow
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital and Harvard Medical School, 101 Merrimac Street, Suite 310, Boston, MA, 02114, USA
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA
| |
Collapse
|
5
|
Silva AB, Liu JR, Metzger SL, Bhaya-Grossman I, Dougherty ME, Seaton MP, Littlejohn KT, Tu-Chan A, Ganguly K, Moses DA, Chang EF. A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages. Nat Biomed Eng 2024:10.1038/s41551-024-01207-5. [PMID: 38769157 DOI: 10.1038/s41551-024-01207-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 04/01/2024] [Indexed: 05/22/2024]
Abstract
Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.
Collapse
Affiliation(s)
- Alexander B Silva
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Jessie R Liu
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Sean L Metzger
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Ilina Bhaya-Grossman
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Maximilian E Dougherty
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
| | - Margaret P Seaton
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
| | - Kaylo T Littlejohn
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA, USA
| | - Adelyn Tu-Chan
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - Karunesh Ganguly
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - David A Moses
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - Edward F Chang
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA.
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA.
| |
Collapse
|
6
|
Ikegawa Y, Fukuma R, Sugano H, Oshino S, Tani N, Tamura K, Iimura Y, Suzuki H, Yamamoto S, Fujita Y, Nishimoto S, Kishima H, Yanagisawa T. Text and image generation from intracranial electroencephalography using an embedding space for text and images. J Neural Eng 2024; 21:036019. [PMID: 38648781 DOI: 10.1088/1741-2552/ad417a] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2023] [Accepted: 04/22/2024] [Indexed: 04/25/2024]
Abstract
Objective.Invasive brain-computer interfaces (BCIs) are promising communication devices for severely paralyzed patients. Recent advances in intracranial electroencephalography (iEEG) coupled with natural language processing have enhanced communication speed and accuracy. It should be noted that such a speech BCI uses signals from the motor cortex. However, BCIs based on motor cortical activities may experience signal deterioration in users with motor cortical degenerative diseases such as amyotrophic lateral sclerosis. An alternative approach to using iEEG of the motor cortex is necessary to support patients with such conditions.Approach. In this study, a multimodal embedding of text and images was used to decode visual semantic information from iEEG signals of the visual cortex to generate text and images. We used contrastive language-image pretraining (CLIP) embedding to represent images presented to 17 patients implanted with electrodes in the occipital and temporal cortices. A CLIP image vector was inferred from the high-γpower of the iEEG signals recorded while viewing the images.Main results.Text was generated by CLIPCAP from the inferred CLIP vector with better-than-chance accuracy. Then, an image was created from the generated text using StableDiffusion with significant accuracy.Significance.The text and images generated from iEEG through the CLIP embedding vector can be used for improved communication.
Collapse
Affiliation(s)
- Yuya Ikegawa
- Institute for Advanced Co-Creation Studies, Osaka University, Suita, Japan
| | - Ryohei Fukuma
- Institute for Advanced Co-Creation Studies, Osaka University, Suita, Japan
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Hidenori Sugano
- Department of Neurosurgery, Juntendo University, Tokyo, Japan
| | - Satoru Oshino
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Naoki Tani
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Kentaro Tamura
- Department of Neurosurgery, Nara Medical University, Kashihara, Japan
| | - Yasushi Iimura
- Department of Neurosurgery, Juntendo University, Tokyo, Japan
| | - Hiroharu Suzuki
- Department of Neurosurgery, Juntendo University, Tokyo, Japan
| | - Shota Yamamoto
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Yuya Fujita
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Shinji Nishimoto
- National Institute of Information and Communications Technology (NICT), Center for Information and Neural Networks (CiNet), Suita, Japan
- Graduate School of Frontier Biosciences, Osaka University, Suita, Japan
| | - Haruhiko Kishima
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Takufumi Yanagisawa
- Institute for Advanced Co-Creation Studies, Osaka University, Suita, Japan
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| |
Collapse
|
7
|
Fukuma R, Majima K, Kawahara Y, Yamashita O, Shiraishi Y, Kishima H, Yanagisawa T. Fast, accurate, and interpretable decoding of electrocorticographic signals using dynamic mode decomposition. Commun Biol 2024; 7:595. [PMID: 38762683 PMCID: PMC11102437 DOI: 10.1038/s42003-024-06294-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2023] [Accepted: 05/06/2024] [Indexed: 05/20/2024] Open
Abstract
Dynamic mode (DM) decomposition decomposes spatiotemporal signals into basic oscillatory components (DMs). DMs can improve the accuracy of neural decoding when used with the nonlinear Grassmann kernel, compared to conventional power features. However, such kernel-based machine learning algorithms have three limitations: large computational time preventing real-time application, incompatibility with non-kernel algorithms, and low interpretability. Here, we propose a mapping function corresponding to the Grassmann kernel that explicitly transforms DMs into spatial DM (sDM) features, which can be used in any machine learning algorithm. Using electrocorticographic signals recorded during various movement and visual perception tasks, the sDM features were shown to improve the decoding accuracy and computational time compared to conventional methods. Furthermore, the components of the sDM features informative for decoding showed similar characteristics to the high-γ power of the signals, but with higher trial-to-trial reproducibility. The proposed sDM features enable fast, accurate, and interpretable neural decoding.
Collapse
Affiliation(s)
- Ryohei Fukuma
- Institute for Advanced Co-Creation Studies, Osaka University, Suita, Japan
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Kei Majima
- Institute for Quantum Life Science, National Institutes for Quantum Science and Technology, Chiba, Japan
- JST PRESTO, Saitama, Japan
| | - Yoshinobu Kawahara
- RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
- Graduate School of Information Science and Technology, Osaka University, Suita, Japan
| | - Okito Yamashita
- RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
- Department of Computational Brain Imaging, Neural Information Analysis Laboratories, ATR, Kyoto, Japan
| | | | - Haruhiko Kishima
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan
| | - Takufumi Yanagisawa
- Institute for Advanced Co-Creation Studies, Osaka University, Suita, Japan.
- Department of Neurosurgery, Graduate School of Medicine, Osaka University, Suita, Japan.
| |
Collapse
|
8
|
Jung T, Zeng N, Fabbri JD, Eichler G, Li Z, Willeke K, Wingel KE, Dubey A, Huq R, Sharma M, Hu Y, Ramakrishnan G, Tien K, Mantovani P, Parihar A, Yin H, Oswalt D, Misdorp A, Uguz I, Shinn T, Rodriguez GJ, Nealley C, Gonzales I, Roukes M, Knecht J, Yoshor D, Canoll P, Spinazzi E, Carloni LP, Pesaran B, Patel S, Youngerman B, Cotton RJ, Tolias A, Shepard KL. Stable, chronic in-vivo recordings from a fully wireless subdural-contained 65,536-electrode brain-computer interface device. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.17.594333. [PMID: 38798494 PMCID: PMC11118429 DOI: 10.1101/2024.05.17.594333] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
Minimally invasive, high-bandwidth brain-computer-interface (BCI) devices can revolutionize human applications. With orders-of-magnitude improvements in volumetric efficiency over other BCI technologies, we developed a 50-μm-thick, mechanically flexible micro-electrocorticography (μECoG) BCI, integrating 256×256 electrodes, signal processing, data telemetry, and wireless powering on a single complementary metal-oxide-semiconductor (CMOS) substrate containing 65,536 recording and 16,384 stimulation channels, from which we can simultaneously record up to 1024 channels at a given time. Fully implanted below the dura, our chip is wirelessly powered, communicating bi-directionally with an external relay station outside the body. We demonstrated chronic, reliable recordings for up to two weeks in pigs and up to two months in behaving non-human primates from somatosensory, motor, and visual cortices, decoding brain signals at high spatiotemporal resolution.
Collapse
|
9
|
Silva AB, Littlejohn KT, Liu JR, Moses DA, Chang EF. The speech neuroprosthesis. Nat Rev Neurosci 2024:10.1038/s41583-024-00819-9. [PMID: 38745103 DOI: 10.1038/s41583-024-00819-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/12/2024] [Indexed: 05/16/2024]
Abstract
Loss of speech after paralysis is devastating, but circumventing motor-pathway injury by directly decoding speech from intact cortical activity has the potential to restore natural communication and self-expression. Recent discoveries have defined how key features of speech production are facilitated by the coordinated activity of vocal-tract articulatory and motor-planning cortical representations. In this Review, we highlight such progress and how it has led to successful speech decoding, first in individuals implanted with intracranial electrodes for clinical epilepsy monitoring and subsequently in individuals with paralysis as part of early feasibility clinical trials to restore speech. We discuss high-spatiotemporal-resolution neural interfaces and the adaptation of state-of-the-art speech computational algorithms that have driven rapid and substantial progress in decoding neural activity into text, audible speech, and facial movements. Although restoring natural speech is a long-term goal, speech neuroprostheses already have performance levels that surpass communication rates offered by current assistive-communication technology. Given this accelerated rate of progress in the field, we propose key evaluation metrics for speed and accuracy, among others, to help standardize across studies. We finish by highlighting several directions to more fully explore the multidimensional feature space of speech and language, which will continue to accelerate progress towards a clinically viable speech neuroprosthesis.
Collapse
Affiliation(s)
- Alexander B Silva
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - Kaylo T Littlejohn
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA, USA
| | - Jessie R Liu
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - David A Moses
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - Edward F Chang
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA.
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
| |
Collapse
|
10
|
Wandelt SK, Bjånes DA, Pejsa K, Lee B, Liu C, Andersen RA. Representation of internal speech by single neurons in human supramarginal gyrus. Nat Hum Behav 2024:10.1038/s41562-024-01867-y. [PMID: 38740984 DOI: 10.1038/s41562-024-01867-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 03/16/2024] [Indexed: 05/16/2024]
Abstract
Speech brain-machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.
Collapse
Affiliation(s)
- Sarah K Wandelt
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA.
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA.
| | - David A Bjånes
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA
- Rancho Los Amigos National Rehabilitation Center, Downey, CA, USA
| | - Kelsie Pejsa
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA
| | - Brian Lee
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- Department of Neurological Surgery, Keck School of Medicine of USC, Los Angeles, CA, USA
- USC Neurorestoration Center, Keck School of Medicine of USC, Los Angeles, CA, USA
| | - Charles Liu
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- Rancho Los Amigos National Rehabilitation Center, Downey, CA, USA
- Department of Neurological Surgery, Keck School of Medicine of USC, Los Angeles, CA, USA
- USC Neurorestoration Center, Keck School of Medicine of USC, Los Angeles, CA, USA
| | - Richard A Andersen
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA
| |
Collapse
|
11
|
Brain-machine-interface device translates internal speech into text. Nat Hum Behav 2024:10.1038/s41562-024-01869-w. [PMID: 38740991 DOI: 10.1038/s41562-024-01869-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
|
12
|
Tankus A, Rosenberg N, Ben-Hamo O, Stern E, Strauss I. Machine learning decoding of single neurons in the thalamus for speech brain-machine interfaces. J Neural Eng 2024; 21:036009. [PMID: 38648783 DOI: 10.1088/1741-2552/ad4179] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2023] [Accepted: 04/22/2024] [Indexed: 04/25/2024]
Abstract
Objective. Our goal is to decode firing patterns of single neurons in the left ventralis intermediate nucleus (Vim) of the thalamus, related to speech production, perception, and imagery. For realistic speech brain-machine interfaces (BMIs), we aim to characterize the amount of thalamic neurons necessary for high accuracy decoding.Approach. We intraoperatively recorded single neuron activity in the left Vim of eight neurosurgical patients undergoing implantation of deep brain stimulator or RF lesioning during production, perception and imagery of the five monophthongal vowel sounds. We utilized the Spade decoder, a machine learning algorithm that dynamically learns specific features of firing patterns and is based on sparse decomposition of the high dimensional feature space.Main results. Spade outperformed all algorithms compared with, for all three aspects of speech: production, perception and imagery, and obtained accuracies of 100%, 96%, and 92%, respectively (chance level: 20%) based on pooling together neurons across all patients. The accuracy was logarithmic in the amount of neurons for all three aspects of speech. Regardless of the amount of units employed, production gained highest accuracies, whereas perception and imagery equated with each other.Significance. Our research renders single neuron activity in the left Vim a promising source of inputs to BMIs for restoration of speech faculties for locked-in patients or patients with anarthria or dysarthria to allow them to communicate again. Our characterization of how many neurons are necessary to achieve a certain decoding accuracy is of utmost importance for planning BMI implantation.
Collapse
Affiliation(s)
- Ariel Tankus
- Functional Neurosurgery Unit, Tel Aviv Sourasky Medical Center, Tel Aviv 6423906, Israel
- Department of Neurology and Neurosurgery, School of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Noam Rosenberg
- School of Electrical Engineering, Iby and Aladar Fleischman Faculty of Engineering, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Oz Ben-Hamo
- School of Electrical Engineering, Iby and Aladar Fleischman Faculty of Engineering, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Einat Stern
- Department of Neurology and Neurosurgery, School of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Ido Strauss
- Functional Neurosurgery Unit, Tel Aviv Sourasky Medical Center, Tel Aviv 6423906, Israel
- Department of Neurology and Neurosurgery, School of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| |
Collapse
|
13
|
Sadras N, Pesaran B, Shanechi MM. Event detection and classification from multimodal time series with application to neural data. J Neural Eng 2024; 21:026049. [PMID: 38513289 DOI: 10.1088/1741-2552/ad3678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Accepted: 03/21/2024] [Indexed: 03/23/2024]
Abstract
The detection of events in time-series data is a common signal-processing problem. When the data can be modeled as a known template signal with an unknown delay in Gaussian noise, detection of the template signal can be done with a traditional matched filter. However, in many applications, the event of interest is represented in multimodal data consisting of both Gaussian and point-process time series. Neuroscience experiments, for example, can simultaneously record multimodal neural signals such as local field potentials (LFPs), which can be modeled as Gaussian, and neuronal spikes, which can be modeled as point processes. Currently, no method exists for event detection from such multimodal data, and as such our objective in this work is to develop a method to meet this need. Here we address this challenge by developing the multimodal event detector (MED) algorithm which simultaneously estimates event times and classes. To do this, we write a multimodal likelihood function for Gaussian and point-process observations and derive the associated maximum likelihood estimator of simultaneous event times and classes. We additionally introduce a cross-modal scaling parameter to account for model mismatch in real datasets. We validate this method in extensive simulations as well as in a neural spike-LFP dataset recorded during an eye-movement task, where the events of interest are eye movements with unknown times and directions. We show that the MED can successfully detect eye movement onset and classify eye movement direction. Further, the MED successfully combines information across data modalities, with multimodal performance exceeding unimodal performance. This method can facilitate applications such as the discovery of latent events in multimodal neural population activity and the development of brain-computer interfaces for naturalistic settings without constrained tasks or prior knowledge of event times.
Collapse
Affiliation(s)
- Nitin Sadras
- Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA, United States of America
| | - Bijan Pesaran
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America
| | - Maryam M Shanechi
- Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA, United States of America
- Thomas Lord Department of Computer Science, Alfred E. Mann Department of Biomedical Engineering, and the Neuroscience Graduate Program, University of Southern California, Los Angeles, CA, United States of America
| |
Collapse
|
14
|
Angrick M, Luo S, Rabbani Q, Candrea DN, Shah S, Milsap GW, Anderson WS, Gordon CR, Rosenblatt KR, Clawson L, Tippett DC, Maragakis N, Tenore FV, Fifer MS, Hermansky H, Ramsey NF, Crone NE. Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS. Sci Rep 2024; 14:9617. [PMID: 38671062 PMCID: PMC11053081 DOI: 10.1038/s41598-024-60277-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2023] [Accepted: 04/21/2024] [Indexed: 04/28/2024] Open
Abstract
Brain-computer interfaces (BCIs) that reconstruct and synthesize speech using brain activity recorded with intracranial electrodes may pave the way toward novel communication interfaces for people who have lost their ability to speak, or who are at high risk of losing this ability, due to neurological disorders. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a man with impaired articulation due to ALS, participating in a clinical trial (ClinicalTrials.gov, NCT03567213) exploring different strategies for BCI communication. The 3-stage approach reported here relies on recurrent neural networks to identify, decode and synthesize speech from electrocorticographic (ECoG) signals acquired across motor, premotor and somatosensory cortices. We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the participant from a vocabulary of 6 keywords previously used for decoding commands to control a communication board. Evaluation of the intelligibility of the synthesized speech indicates that 80% of the words can be correctly recognized by human listeners. Our results show that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words while preserving the participant's voice profile, and provide further evidence for the stability of ECoG for speech-based BCIs.
Collapse
Affiliation(s)
- Miguel Angrick
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA.
| | - Shiyu Luo
- Department of Biomedical Engineering, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Qinwan Rabbani
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD, USA
| | - Daniel N Candrea
- Department of Biomedical Engineering, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Samyak Shah
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Griffin W Milsap
- Research and Exploratory Development Department, Johns Hopkins Applied Physics Laboratory, Laurel, MD, USA
| | - William S Anderson
- Department of Neurosurgery, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Chad R Gordon
- Department of Neurosurgery, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
- Section of Neuroplastic and Reconstructive Surgery, Department of Plastic Surgery, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Kathryn R Rosenblatt
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
- Department of Anesthesiology & Critical Care Medicine, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Lora Clawson
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Donna C Tippett
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
- Department of Otolaryngology-Head and Neck Surgery, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
- Department of Physical Medicine and Rehabilitation, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Nicholas Maragakis
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Francesco V Tenore
- Research and Exploratory Development Department, Johns Hopkins Applied Physics Laboratory, Laurel, MD, USA
| | - Matthew S Fifer
- Research and Exploratory Development Department, Johns Hopkins Applied Physics Laboratory, Laurel, MD, USA
| | - Hynek Hermansky
- Center for Language and Speech Processing, The Johns Hopkins University, Baltimore, MD, USA
- Human Language Technology Center of Excellence, The Johns Hopkins University, Baltimore, MD, USA
| | - Nick F Ramsey
- UMC Utrecht Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Nathan E Crone
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA.
| |
Collapse
|
15
|
Shah NP, Willsey MS, Hahn N, Kamdar F, Avansino DT, Fan C, Hochberg LR, Willett FR, Henderson JM. A flexible intracortical brain-computer interface for typing using finger movements. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.22.590630. [PMID: 38712189 PMCID: PMC11071346 DOI: 10.1101/2024.04.22.590630] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2024]
Abstract
Keyboard typing with finger movements is a versatile digital interface for users with diverse skills, needs, and preferences. Currently, such an interface does not exist for people with paralysis. We developed an intracortical brain-computer interface (BCI) for typing with attempted flexion/extension movements of three finger groups on the right hand, or both hands, and demonstrated its flexibility in two dominant typing paradigms. The first paradigm is "point-and-click" typing, where a BCI user selects one key at a time using continuous real-time control, allowing selection of arbitrary sequences of symbols. During cued character selection with this paradigm, a human research participant with paralysis achieved 30-40 selections per minute with nearly 90% accuracy. The second paradigm is "keystroke" typing, where the BCI user selects each character by a discrete movement without real-time feedback, often giving a faster speed for natural language sentences. With 90 cued characters per minute, decoding attempted finger movements and correcting errors using a language model resulted in more than 90% accuracy. Notably, both paradigms matched the state-of-the-art for BCI performance and enabled further flexibility by the simultaneous selection of multiple characters as well as efficient decoder estimation across paradigms. Overall, the high-performance interface is a step towards the wider accessibility of BCI technology by addressing unmet user needs for flexibility.
Collapse
|
16
|
Micera S, Menciassi A, Cianferotti L, Gruppioni E, Lionetti V. Organ Neuroprosthetics: Connecting Transplanted and Artificial Organs with the Nervous System. Adv Healthc Mater 2024:e2302896. [PMID: 38656615 DOI: 10.1002/adhm.202302896] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Revised: 04/01/2024] [Indexed: 04/26/2024]
Abstract
Implantable neural interfaces with the central and peripheral nervous systems are currently used to restore sensory, motor, and cognitive functions in disabled people with very promising results. They have also been used to modulate autonomic activities to treat diseases such as diabetes or hypertension. Here, this study proposes to extend the use of these technologies to (re-)establish the connection between new (transplanted or artificial) organs and the nervous system in order to increase the long-term efficacy and the effective biointegration of these solutions. In this perspective paper, some clinically relevant applications of this approach are briefly described. Then, the choices that neural engineers must implement about the type, implantation location, and closed-loop control algorithms to successfully realize this approach are highlighted. It is believed that these new "organ neuroprostheses" are going to become more and more valuable and very effective solutions in the years to come.
Collapse
Affiliation(s)
- Silvestro Micera
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Pisa, 56127, Italy
- Interdisciplinary Research Center Health Science, Scuola Superiore Sant'Anna, Pisa, 56127, Italy
- Bertarelli Foundation Chair in Translational Neuroengineering, Neuro-X Institute, School of Engineering, Ecole Polytechnique Federale de Lausanne (EPFL), Lausanne, 1015, Switzerland
| | - Arianna Menciassi
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Pisa, 56127, Italy
- Interdisciplinary Research Center Health Science, Scuola Superiore Sant'Anna, Pisa, 56127, Italy
| | - Luisella Cianferotti
- Department of Experimental and Clinical Biomedical Sciences, University of Florence, Florence, 50121, Italy
| | | | - Vincenzo Lionetti
- Interdisciplinary Research Center Health Science, Scuola Superiore Sant'Anna, Pisa, 56127, Italy
- UOSVD Anesthesia and Resuscitation, Fondazione Toscana G. Monasterio, Pisa, 56127, Italy
| |
Collapse
|
17
|
Huang C, Shi N, Miao Y, Chen X, Wang Y, Gao X. Visual tracking brain-computer interface. iScience 2024; 27:109376. [PMID: 38510138 PMCID: PMC10951983 DOI: 10.1016/j.isci.2024.109376] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Revised: 01/25/2024] [Accepted: 02/27/2024] [Indexed: 03/22/2024] Open
Abstract
Brain-computer interfaces (BCIs) offer a way to interact with computers without relying on physical movements. Non-invasive electroencephalography-based visual BCIs, known for efficient speed and calibration ease, face limitations in continuous tasks due to discrete stimulus design and decoding methods. To achieve continuous control, we implemented a novel spatial encoding stimulus paradigm and devised a corresponding projection method to enable continuous modulation of decoded velocity. Subsequently, we conducted experiments involving 17 participants and achieved Fitt's information transfer rate (ITR) of 0.55 bps for the fixed tracking task and 0.37 bps for the random tracking task. The proposed BCI with a high Fitt's ITR was then integrated into two applications, including painting and gaming. In conclusion, this study proposed a visual BCI based-control method to go beyond discrete commands, allowing natural continuous control based on neural activity.
Collapse
Affiliation(s)
- Changxing Huang
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| | - Nanlin Shi
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| | - Yining Miao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| | - Xiaogang Chen
- Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin 300192, China
| | - Yijun Wang
- State Key Laboratory on Integrated Optoelectronics, Institute of Semiconductors, Chinese Academy of Sciences Beijing, Beijing 100083, China
| | - Xiaorong Gao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| |
Collapse
|
18
|
Zohny H, Savulescu J. When Two Become One: Singular Duos and the Neuroethical Frontiers of Brain-to-Brain Interfaces. Camb Q Healthc Ethics 2024:1-13. [PMID: 38606432 DOI: 10.1017/s0963180124000197] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/13/2024]
Abstract
Advances in brain-brain interface technologies raise the possibility that two or more individuals could directly link their minds, sharing thoughts, emotions, and sensory experiences. This paper explores conceptual and ethical issues posed by such mind-merging technologies in the context of clinical neuroethics. Using hypothetical examples along a spectrum from loosely connected pairs to fully merged minds, the authors sketch out a range of factors relevant to identifying the degree of a merger. They then consider potential new harms like loss of identity, psychological domination, loss of mental privacy, and challenges for notions of autonomy and patient benefit when applied to merged minds. While radical technologies may seem to necessitate new ethical paradigms, the authors suggest the individual-focus underpinning clinical ethics can largely accommodate varying degrees of mind mergers so long as individual patient interests remain identifiable. However, advanced decisionmaking and directives may have limitations in addressing the dilemmas posed. Overall, mind-merging possibilities amplify existing challenges around loss of identity, relating to others, autonomy, privacy, and the delineation of patient interests. This paper lays the groundwork for developing resources to address the novel issues raised, while suggesting the technologies reveal continuity with current healthcare ethics tensions.
Collapse
Affiliation(s)
- Hazem Zohny
- Oxford Uehiro Centre for Practical Ethics, University of Oxford, Oxford, UK
| | - Julian Savulescu
- Chen Su Lan Centennial Professor in Medical Ethics, Centre for Biomedical Ethics, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore
- Uehiro Chair in Practical Ethics, Faculty of Philosophy, University of Oxford, Oxford, UK
| |
Collapse
|
19
|
Card NS, Wairagkar M, Iacobacci C, Hou X, Singer-Clark T, Willett FR, Kunz EM, Fan C, Nia MV, Deo DR, Srinivasan A, Choi EY, Glasser MF, Hochberg LR, Henderson JM, Shahlaie K, Brandman DM, Stavisky SD. An accurate and rapidly calibrating speech neuroprosthesis. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2023.12.26.23300110. [PMID: 38645254 PMCID: PMC11030484 DOI: 10.1101/2023.12.26.23300110] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/23/2024]
Abstract
Brain-computer interfaces can enable rapid, intuitive communication for people with paralysis by transforming the cortical activity associated with attempted speech into text on a computer screen. Despite recent advances, communication with brain-computer interfaces has been restricted by extensive training data requirements and inaccurate word output. A man in his 40's with ALS with tetraparesis and severe dysarthria (ALSFRS-R = 23) was enrolled into the BrainGate2 clinical trial. He underwent surgical implantation of four microelectrode arrays into his left precentral gyrus, which recorded neural activity from 256 intracortical electrodes. We report a speech neuroprosthesis that decoded his neural activity as he attempted to speak in both prompted and unstructured conversational settings. Decoded words were displayed on a screen, then vocalized using text-to-speech software designed to sound like his pre-ALS voice. On the first day of system use, following 30 minutes of attempted speech training data, the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. On the second day, the size of the possible output vocabulary increased to 125,000 words, and, after 1.4 additional hours of training data, the neuroprosthesis achieved 90.2% accuracy. With further training data, the neuroprosthesis sustained 97.5% accuracy beyond eight months after surgical implantation. The participant has used the neuroprosthesis to communicate in self-paced conversations for over 248 hours. In an individual with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore naturalistic communication after a brief training period.
Collapse
Affiliation(s)
- Nicholas S Card
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Maitreyee Wairagkar
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Carrina Iacobacci
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Xianda Hou
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Computer Science, University of California Davis, Davis, CA, USA
| | - Tyler Singer-Clark
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Biomedical Engineering, University of California Davis, Davis, CA, USA
| | - Francis R Willett
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
- Departments of Electrical Engineering, Stanford University, Stanford, CA, USA
- Howard Hughes Medical Institute, Stanford University, Stanford, CA, USA
| | - Erin M Kunz
- Departments of Electrical Engineering, Stanford University, Stanford, CA, USA
- Departments of Mechanical Engineering, Stanford University, Stanford, CA, USA
| | - Chaofei Fan
- Departments of Computer Science, Stanford University, Stanford, CA, USA
| | - Maryam Vahdati Nia
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Computer Science, University of California Davis, Davis, CA, USA
| | - Darrel R Deo
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Aparna Srinivasan
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Biomedical Engineering, University of California Davis, Davis, CA, USA
| | - Eun Young Choi
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Matthew F Glasser
- Departments of Radiology and Neuroscience, Washington University School of Medicine, Saint Louis, MO, USA
| | - Leigh R Hochberg
- School of Engineering and Carney Institute for Brain Sciences, Brown University, Providence, RI, USA
- VA RR&D Center for Neurorestoration and Neurotechnology, VA Providence Healthcare, Providence, RI
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA
| | - Jaimie M Henderson
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | - Kiarash Shahlaie
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - David M Brandman
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Sergey D Stavisky
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| |
Collapse
|
20
|
Soldado-Magraner J, Antonietti A, French J, Higgins N, Young MJ, Larrivee D, Monteleone R. Applying the IEEE BRAIN neuroethics framework to intra-cortical brain-computer interfaces. J Neural Eng 2024; 21:022001. [PMID: 38537269 DOI: 10.1088/1741-2552/ad3852] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 03/27/2024] [Indexed: 04/05/2024]
Abstract
Objective. Brain-computer interfaces (BCIs) are neuroprosthetic devices that allow for direct interaction between brains and machines. These types of neurotechnologies have recently experienced a strong drive in research and development, given, in part, that they promise to restore motor and communication abilities in individuals experiencing severe paralysis. While a rich literature analyzes the ethical, legal, and sociocultural implications (ELSCI) of these novel neurotechnologies, engineers, clinicians and BCI practitioners often do not have enough exposure to these topics.Approach. Here, we present the IEEE Neuroethics Framework, an international, multiyear, iterative initiative aimed at developing a robust, accessible set of considerations for diverse stakeholders.Main results. Using the framework, we provide practical examples of ELSCI considerations for BCI neurotechnologies. We focus on invasive technologies, and in particular, devices that are implanted intra-cortically for medical research applications.Significance. We demonstrate the utility of our framework in exposing a wide range of implications across different intra-cortical BCI technology modalities and conclude with recommendations on how to utilize this knowledge in the development and application of ethical guidelines for BCI neurotechnologies.
Collapse
Affiliation(s)
- Joana Soldado-Magraner
- Department of Electrical and Computer Engineering and the Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, United States of America
| | - Alberto Antonietti
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano 20131, Italy
| | - Jennifer French
- Neurotech Network, St. Petersburg, FL 33733, United States of America
| | - Nathan Higgins
- School of Psychological Sciences, Monash University, Clayton, VIC 3800, Australia
| | - Michael J Young
- Department of Neurology, Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, United States of America
| | - Denis Larrivee
- Mind and Brain Institute, University of Navarra Medical School, Pamplona, Navarra 31008, Spain
- Loyola University, Chicago, IL 60611, United States of America
| | - Rebecca Monteleone
- Disability Studies Program, University of Toledo, Toledo, OH 43606, United States of America
| |
Collapse
|
21
|
Saway BF, Palmer C, Hughes C, Triano M, Suresh RE, Gilmore J, George M, Kautz SA, Rowland NC. The evolution of neuromodulation for chronic stroke: From neuroplasticity mechanisms to brain-computer interfaces. Neurotherapeutics 2024; 21:e00337. [PMID: 38377638 PMCID: PMC11103214 DOI: 10.1016/j.neurot.2024.e00337] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Revised: 02/05/2024] [Accepted: 02/13/2024] [Indexed: 02/22/2024] Open
Abstract
Stroke is one of the most common and debilitating neurological conditions worldwide. Those who survive experience motor, sensory, speech, vision, and/or cognitive deficits that severely limit remaining quality of life. While rehabilitation programs can help improve patients' symptoms, recovery is often limited, and patients frequently continue to experience impairments in functional status. In this review, invasive neuromodulation techniques to augment the effects of conventional rehabilitation methods are described, including vagus nerve stimulation (VNS), deep brain stimulation (DBS) and brain-computer interfaces (BCIs). In addition, the evidence base for each of these techniques, pivotal trials, and future directions are explored. Finally, emerging technologies such as functional near-infrared spectroscopy (fNIRS) and the shift to artificial intelligence-enabled implants and wearables are examined. While the field of implantable devices for chronic stroke recovery is still in a nascent stage, the data reviewed are suggestive of immense potential for reducing the impact and impairment from this globally prevalent disorder.
Collapse
Affiliation(s)
- Brian F Saway
- Department of Neurosurgery, Medical University of South Carolina, SC 29425, USA.
| | - Charles Palmer
- Department of Psychiatry, Medical University of South Carolina, SC 29425, USA
| | - Christopher Hughes
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA 15260, USA
| | - Matthew Triano
- Department of Neurosurgery, Medical University of South Carolina, SC 29425, USA
| | - Rishishankar E Suresh
- College of Medicine, Medical University of South Carolina, Charleston, SC 29425, USA
| | - Jordon Gilmore
- Department of Bioengineering, Clemson University, Clemson, SC 29634, USA
| | - Mark George
- Department of Psychiatry, Medical University of South Carolina, SC 29425, USA; Ralph H Johnson VA Health Care System, Charleston, SC 29425, USA
| | - Steven A Kautz
- Department of Health Science and Research, Medical University of South Carolina, SC 29425, USA; Ralph H Johnson VA Health Care System, Charleston, SC 29425, USA
| | - Nathan C Rowland
- Department of Neurosurgery, Medical University of South Carolina, SC 29425, USA; MUSC Institute for Neuroscience Discovery (MIND), Medical University of South Carolina, SC 29425, USA
| |
Collapse
|
22
|
Chen Y, Wang F, Li T, Zhao L, Gong A, Nan W, Ding P, Fu Y. Several inaccurate or erroneous conceptions and misleading propaganda about brain-computer interfaces. Front Hum Neurosci 2024; 18:1391550. [PMID: 38601800 PMCID: PMC11004276 DOI: 10.3389/fnhum.2024.1391550] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Accepted: 03/18/2024] [Indexed: 04/12/2024] Open
Abstract
Brain-computer interface (BCI) is a revolutionizing human-computer interaction, which has potential applications for specific individuals or groups in specific scenarios. Extensive research has been conducted on the principles and implementation methods of BCI, and efforts are currently being made to bridge the gap from research to real-world applications. However, there are inaccurate or erroneous conceptions about BCI among some members of the public, and certain media outlets, as well as some BCI researchers, developers, manufacturers, and regulators, propagate misleading or overhyped claims about BCI technology. Therefore, this article summarizes the several misconceptions and misleading propaganda about BCI, including BCI being capable of "mind-controlled," "controlling brain," "mind reading," and the ability to "download" or "upload" information from or to the brain using BCI, among others. Finally, the limitations (shortcomings) and limits (boundaries) of BCI, as well as the necessity of conducting research aimed at countering BCI systems are discussed, and several suggestions are offered to reduce misconceptions and misleading claims about BCI.
Collapse
Affiliation(s)
- Yanxiao Chen
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People’s Armed Police Force Engineering University, Xi’an, China
| | - Wenya Nan
- Department of Psychology, School of Education, Shanghai Normal University, Shanghai, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| |
Collapse
|
23
|
Chen J, Chen X, Wang R, Le C, Khalilian-Gourtani A, Jensen E, Dugan P, Doyle W, Devinsky O, Friedman D, Flinker A, Wang Y. Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.11.584533. [PMID: 38559163 PMCID: PMC10980022 DOI: 10.1101/2024.03.11.584533] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Objective This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. Approach We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes, by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train both subject-specific models using data from a single participant as well as multi-patient models exploiting data from multiple participants. Main Results The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. Significance The proposed SwinTW decoder enables future speech neuroprostheses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests the exciting possibility of developing speech neuroprostheses for people with speech disability without relying on their own neural data for training, which is not always feasible.
Collapse
Affiliation(s)
- Junbo Chen
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Xupeng Chen
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Ran Wang
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Chenqian Le
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | | | - Erika Jensen
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Patricia Dugan
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Werner Doyle
- Neurosurgery Department, New York University, 550 1st Avenue, Manhattan, 10016, NY, USA
| | - Orrin Devinsky
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Daniel Friedman
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Adeen Flinker
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
- Biomedical Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Yao Wang
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
- Biomedical Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| |
Collapse
|
24
|
Temmar H, Willsey MS, Costello JT, Mender MJ, Cubillos LH, Lam JL, Wallace DM, Kelberman MM, Patil PG, Chestek CA. Artificial neural network for brain-machine interface consistently produces more naturalistic finger movements than linear methods. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.01.583000. [PMID: 38496403 PMCID: PMC10942378 DOI: 10.1101/2024.03.01.583000] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/19/2024]
Abstract
Brain-machine interfaces (BMI) aim to restore function to persons living with spinal cord injuries by 'decoding' neural signals into behavior. Recently, nonlinear BMI decoders have outperformed previous state-of-the-art linear decoders, but few studies have investigated what specific improvements these nonlinear approaches provide. In this study, we compare how temporally convolved feedforward neural networks (tcFNNs) and linear approaches predict individuated finger movements in open and closed-loop settings. We show that nonlinear decoders generate more naturalistic movements, producing distributions of velocities 85.3% closer to true hand control than linear decoders. Addressing concerns that neural networks may come to inconsistent solutions, we find that regularization techniques improve the consistency of tcFNN convergence by 194.6%, along with improving average performance, and training speed. Finally, we show that tcFNN can leverage training data from multiple task variations to improve generalization. The results of this study show that nonlinear methods produce more naturalistic movements and show potential for generalizing over less constrained tasks. Teaser A neural network decoder produces consistent naturalistic movements and shows potential for real-world generalization through task variations.
Collapse
|
25
|
Londoño‐Ramírez H, Huang X, Cools J, Chrzanowska A, Brunner C, Ballini M, Hoffman L, Steudel S, Rolin C, Mora Lopez C, Genoe J, Haesler S. Multiplexed Surface Electrode Arrays Based on Metal Oxide Thin-Film Electronics for High-Resolution Cortical Mapping. ADVANCED SCIENCE (WEINHEIM, BADEN-WURTTEMBERG, GERMANY) 2024; 11:e2308507. [PMID: 38145348 PMCID: PMC10933637 DOI: 10.1002/advs.202308507] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Revised: 12/15/2023] [Indexed: 12/26/2023]
Abstract
Electrode grids are used in neuroscience research and clinical practice to record electrical activity from the surface of the brain. However, existing passive electrocorticography (ECoG) technologies are unable to offer both high spatial resolution and wide cortical coverage, while ensuring a compact acquisition system. The electrode count and density are restricted by the fact that each electrode must be individually wired. This work presents an active micro-electrocorticography (µECoG) implant that tackles this limitation by incorporating metal oxide thin-film transistors (TFTs) into a flexible electrode array, allowing to address multiple electrodes through a single shared readout line. By combining the array with an incremental-ΔΣ readout integrated circuit (ROIC), the system is capable of recording from up to 256 electrodes virtually simultaneously, thanks to the implemented 16:1 time-division multiplexing scheme, offering lower noise levels than existing active µECoG arrays. In vivo validation is demonstrated acutely in mice by recording spontaneous activity and somatosensory evoked potentials over a cortical surface of ≈8×8 mm2 . The proposed neural interface overcomes the wiring bottleneck limiting ECoG arrays, holding promise as a powerful tool for improved mapping of the cerebral cortex and as an enabling technology for future brain-machine interfaces.
Collapse
Affiliation(s)
- Horacio Londoño‐Ramírez
- Department of Neuroscience, Leuven Brain InstituteKatholieke Universiteit (KU) LeuvenLeuven3001Belgium
- Neuroelectronics Research Flanders (NERF)Leuven3001Belgium
- imecLeuven3001Belgium
- Flanders Institute for Biotechnology (VIB)Gent9052Belgium
| | - Xiaohua Huang
- imecLeuven3001Belgium
- Department of Electrical Engineering (ESAT)Katholieke Universiteit (KU) LeuvenLeuven3001Belgium
| | - Jordi Cools
- Neuroelectronics Research Flanders (NERF)Leuven3001Belgium
- imecLeuven3001Belgium
- Flanders Institute for Biotechnology (VIB)Gent9052Belgium
- Present address:
Thermo Fisher Scientific3001LeuvenBelgium
| | - Anna Chrzanowska
- Neuroelectronics Research Flanders (NERF)Leuven3001Belgium
- Flanders Institute for Biotechnology (VIB)Gent9052Belgium
- Department of BiologyKatholieke Universiteit (KU) LeuvenLeuven3001Belgium
| | - Clément Brunner
- Department of Neuroscience, Leuven Brain InstituteKatholieke Universiteit (KU) LeuvenLeuven3001Belgium
- Neuroelectronics Research Flanders (NERF)Leuven3001Belgium
- Flanders Institute for Biotechnology (VIB)Gent9052Belgium
| | - Marco Ballini
- imecLeuven3001Belgium
- Present address:
Microphone Business Unit, TDK InvenSense20057MilanItaly
| | - Luis Hoffman
- Neuroelectronics Research Flanders (NERF)Leuven3001Belgium
- imecLeuven3001Belgium
- Present address:
Swave Photonics3001LeuvenBelgium
| | - Soeren Steudel
- imecLeuven3001Belgium
- Present address:
MICLEDI Microdisplays3001LeuvenBelgium
| | | | | | - Jan Genoe
- Department of Electrical Engineering (ESAT)Katholieke Universiteit (KU) LeuvenLeuven3001Belgium
| | - Sebastian Haesler
- Department of Neuroscience, Leuven Brain InstituteKatholieke Universiteit (KU) LeuvenLeuven3001Belgium
- Neuroelectronics Research Flanders (NERF)Leuven3001Belgium
- Flanders Institute for Biotechnology (VIB)Gent9052Belgium
| |
Collapse
|
26
|
Wu H, Cai C, Ming W, Chen W, Zhu Z, Feng C, Jiang H, Zheng Z, Sawan M, Wang T, Zhu J. Speech decoding using cortical and subcortical electrophysiological signals. Front Neurosci 2024; 18:1345308. [PMID: 38486966 PMCID: PMC10937352 DOI: 10.3389/fnins.2024.1345308] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 02/12/2024] [Indexed: 03/17/2024] Open
Abstract
Introduction Language impairments often result from severe neurological disorders, driving the development of neural prosthetics utilizing electrophysiological signals to restore comprehensible language. Previous decoding efforts primarily focused on signals from the cerebral cortex, neglecting subcortical brain structures' potential contributions to speech decoding in brain-computer interfaces. Methods In this study, stereotactic electroencephalography (sEEG) was employed to investigate subcortical structures' role in speech decoding. Two native Mandarin Chinese speakers, undergoing sEEG implantation for epilepsy treatment, participated. Participants read Chinese text, with 1-30, 30-70, and 70-150 Hz frequency band powers of sEEG signals extracted as key features. A deep learning model based on long short-term memory assessed the contribution of different brain structures to speech decoding, predicting consonant articulatory place, manner, and tone within single syllable. Results Cortical signals excelled in articulatory place prediction (86.5% accuracy), while cortical and subcortical signals performed similarly for articulatory manner (51.5% vs. 51.7% accuracy). Subcortical signals provided superior tone prediction (58.3% accuracy). The superior temporal gyrus was consistently relevant in speech decoding for consonants and tone. Combining cortical and subcortical inputs yielded the highest prediction accuracy, especially for tone. Discussion This study underscores the essential roles of both cortical and subcortical structures in different aspects of speech decoding.
Collapse
Affiliation(s)
- Hemmings Wu
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
- Clinical Research Center for Neurological Disease of Zhejiang Province, Hangzhou, China
| | - Chengwei Cai
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Wenjie Ming
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
- Department of Neurology, Epilepsy Center, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Wangyu Chen
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Zhoule Zhu
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Chen Feng
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Hongjie Jiang
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Zhe Zheng
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Mohamad Sawan
- CenBRAIN Lab, School of Engineering, Westlake University, Hangzhou, China
| | - Ting Wang
- School of Foreign Languages, Tongji University, Shanghai, China
- Center for Speech and Language Processing, Tongji University, Shanghai, China
| | - Junming Zhu
- Department of Neurosurgery, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| |
Collapse
|
27
|
张 喆, 陈 衍, 赵 旭, 王 帆, 丁 鹏, 赵 磊, 伏 云. [Ethical considerations for medical applications of implantable brain-computer interfaces]. SHENG WU YI XUE GONG CHENG XUE ZA ZHI = JOURNAL OF BIOMEDICAL ENGINEERING = SHENGWU YIXUE GONGCHENGXUE ZAZHI 2024; 41:177-183. [PMID: 38403619 PMCID: PMC10894729 DOI: 10.7507/1001-5515.202309083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Revised: 12/18/2023] [Indexed: 02/27/2024]
Abstract
Implantable brain-computer interfaces (BCIs) have potentially important clinical applications due to the high spatial resolution and signal-to-noise ratio of electrodes that are closer to or implanted in the cerebral cortex. However, the surgery and electrodes of implantable BCIs carry safety risks of brain tissue damage, and their medical applications face ethical challenges, with little literature to date systematically considering ethical norms for the medical applications of implantable BCIs. In order to promote the clinical translation of this type of BCI, we considered the ethics of practice for the medical application of implantable BCIs, including: reducing the risk of brain tissue damage from implantable BCI surgery and electrodes, providing patients with customized and personalized implantable BCI treatments, ensuring multidisciplinary collaboration in the clinical application of implantable BCIs, and the responsible use of implantable BCIs, among others. It is expected that this article will provide thoughts and references for the research and development of ethics of the medical application of implantable BCI.
Collapse
Affiliation(s)
- 喆 张
- 昆明理工大学 马克思主义学院(昆明 650500)Faculty of Marxism, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 衍肖 陈
- 昆明理工大学 马克思主义学院(昆明 650500)Faculty of Marxism, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 旭 赵
- 昆明理工大学 马克思主义学院(昆明 650500)Faculty of Marxism, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 帆 王
- 昆明理工大学 马克思主义学院(昆明 650500)Faculty of Marxism, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 鹏 丁
- 昆明理工大学 马克思主义学院(昆明 650500)Faculty of Marxism, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 磊 赵
- 昆明理工大学 马克思主义学院(昆明 650500)Faculty of Marxism, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 云发 伏
- 昆明理工大学 马克思主义学院(昆明 650500)Faculty of Marxism, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| |
Collapse
|
28
|
Wan C, Pei M, Shi K, Cui H, Long H, Qiao L, Xing Q, Wan Q. Toward a Brain-Neuromorphics Interface. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2024:e2311288. [PMID: 38339866 DOI: 10.1002/adma.202311288] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Revised: 01/17/2024] [Indexed: 02/12/2024]
Abstract
Brain-computer interfaces (BCIs) that enable human-machine interaction have immense potential in restoring or augmenting human capabilities. Traditional BCIs are realized based on complementary metal-oxide-semiconductor (CMOS) technologies with complex, bulky, and low biocompatible circuits, and suffer with the low energy efficiency of the von Neumann architecture. The brain-neuromorphics interface (BNI) would offer a promising solution to advance the BCI technologies and shape the interactions with machineries. Neuromorphic devices and systems are able to provide substantial computation power with extremely high energy-efficiency by implementing in-materia computing such as in situ vector-matrix multiplication (VMM) and physical reservoir computing. Recent progresses on integrating neuromorphic components with sensing and/or actuating modules, give birth to the neuromorphic afferent nerve, efferent nerve, sensorimotor loop, and so on, which has advanced the technologies for future neurorobotics by achieving sophisticated sensorimotor capabilities as the biological system. With the development on the compact artificial spiking neuron and bioelectronic interfaces, the seamless communication between a BNI and a bioentity is reasonably expectable. In this review, the upcoming BNIs are profiled by introducing the brief history of neuromorphics, reviewing the recent progresses on related areas, and discussing the future advances and challenges that lie ahead.
Collapse
Affiliation(s)
- Changjin Wan
- Yongjiang Laboratory (Y-LAB), Ningbo, Zhejiang, 315202, China
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, 315201, China
| | - Mengjiao Pei
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Kailu Shi
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Hangyuan Cui
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Haotian Long
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Lesheng Qiao
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Qianye Xing
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| | - Qing Wan
- Yongjiang Laboratory (Y-LAB), Ningbo, Zhejiang, 315202, China
- School of Electronic Science and Engineering, National Laboratory of Solid-State Microstructures, Collaborative Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, 315201, China
| |
Collapse
|
29
|
Vitória MA, Fernandes FG, van den Boom M, Ramsey N, Raemaekers M. Decoding Single and Paired Phonemes Using 7T Functional MRI. Brain Topogr 2024:10.1007/s10548-024-01034-6. [PMID: 38261272 DOI: 10.1007/s10548-024-01034-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 01/12/2024] [Indexed: 01/24/2024]
Abstract
Several studies have shown that mouth movements related to the pronunciation of individual phonemes are represented in the sensorimotor cortex. This would theoretically allow for brain computer interfaces that are capable of decoding continuous speech by training classifiers based on the activity in the sensorimotor cortex related to the production of individual phonemes. To address this, we investigated the decodability of trials with individual and paired phonemes (pronounced consecutively with one second interval) using activity in the sensorimotor cortex. Fifteen participants pronounced 3 different phonemes and 3 combinations of two of the same phonemes in a 7T functional MRI experiment. We confirmed that support vector machine (SVM) classification of single and paired phonemes was possible. Importantly, by combining classifiers trained on single phonemes, we were able to classify paired phonemes with an accuracy of 53% (33% chance level), demonstrating that activity of isolated phonemes is present and distinguishable in combined phonemes. A SVM searchlight analysis showed that the phoneme representations are widely distributed in the ventral sensorimotor cortex. These findings provide insights about the neural representations of single and paired phonemes. Furthermore, it supports the notion that speech BCI may be feasible based on machine learning algorithms trained on individual phonemes using intracranial electrode grids.
Collapse
Affiliation(s)
- Maria Araújo Vitória
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Francisco Guerreiro Fernandes
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Max van den Boom
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
- Department of Physiology and Biomedical Engineering, Mayo Clinic, Rochester, MN, USA
| | - Nick Ramsey
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Mathijs Raemaekers
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands.
| |
Collapse
|
30
|
Schippers A, Vansteensel MJ, Freudenburg ZV, Ramsey NF. Don't put words in my mouth: Speech perception can generate False Positive activation of a speech BCI. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.01.21.23300437. [PMID: 38343801 PMCID: PMC10854295 DOI: 10.1101/2024.01.21.23300437] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/17/2024]
Abstract
Recent studies have demonstrated that speech can be decoded from brain activity and used for brain-computer interface (BCI)-based communication. It is however also known that the area often used as a signal source for speech decoding BCIs, the sensorimotor cortex (SMC), is also engaged when people perceive speech, thus making speech perception a potential source of false positive activation of the BCI. The current study investigated if and how speech perception may interfere with reliable speech BCI control. We recorded high-density electrocorticography (HD-ECoG) data from five subjects while they performed a speech perception and speech production task and trained a support-vector machine (SVM) on the produced speech data. Our results show that decoders that are highly reliable at detecting self-produced speech from brain signals also generate false positives during the perception of speech. We conclude that speech perception interferes with reliable BCI control, and that efforts to limit the occurrence of false positives during daily-life BCI use should be implemented in BCI design to increase the likelihood of successful adaptation by end users.
Collapse
Affiliation(s)
- A Schippers
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht, The Netherlands
| | - M J Vansteensel
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht, The Netherlands
| | - Z V Freudenburg
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht, The Netherlands
| | - N F Ramsey
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht, The Netherlands
| |
Collapse
|
31
|
Deo DR, Willett FR, Avansino DT, Hochberg LR, Henderson JM, Shenoy KV. Brain control of bimanual movement enabled by recurrent neural networks. Sci Rep 2024; 14:1598. [PMID: 38238386 PMCID: PMC10796685 DOI: 10.1038/s41598-024-51617-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Accepted: 01/07/2024] [Indexed: 01/22/2024] Open
Abstract
Brain-computer interfaces have so far focused largely on enabling the control of a single effector, for example a single computer cursor or robotic arm. Restoring multi-effector motion could unlock greater functionality for people with paralysis (e.g., bimanual movement). However, it may prove challenging to decode the simultaneous motion of multiple effectors, as we recently found that a compositional neural code links movements across all limbs and that neural tuning changes nonlinearly during dual-effector motion. Here, we demonstrate the feasibility of high-quality bimanual control of two cursors via neural network (NN) decoders. Through simulations, we show that NNs leverage a neural 'laterality' dimension to distinguish between left and right-hand movements as neural tuning to both hands become increasingly correlated. In training recurrent neural networks (RNNs) for two-cursor control, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously. Our results suggest that neural network decoders may be advantageous for multi-effector decoding, provided they are designed to transfer to the online setting.
Collapse
Affiliation(s)
- Darrel R Deo
- Department of Neurosurgery, Stanford University, Stanford, CA, USA.
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA.
| | - Francis R Willett
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Donald T Avansino
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Leigh R Hochberg
- School of Engineering, Brown University, Providence, RI, USA
- Carney Institute for Brain Science, Brown University, Providence, RI, USA
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
| | - Krishna V Shenoy
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Department of Bioengineering, Stanford University, Stanford, CA, USA
- Department of Neurobiology, Stanford University, Stanford, CA, USA
| |
Collapse
|
32
|
Donati E, Valle G. Neuromorphic hardware for somatosensory neuroprostheses. Nat Commun 2024; 15:556. [PMID: 38228580 PMCID: PMC10791662 DOI: 10.1038/s41467-024-44723-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Accepted: 01/03/2024] [Indexed: 01/18/2024] Open
Abstract
In individuals with sensory-motor impairments, missing limb functions can be restored using neuroprosthetic devices that directly interface with the nervous system. However, restoring the natural tactile experience through electrical neural stimulation requires complex encoding strategies. Indeed, they are presently limited in effectively conveying or restoring tactile sensations by bandwidth constraints. Neuromorphic technology, which mimics the natural behavior of neurons and synapses, holds promise for replicating the encoding of natural touch, potentially informing neurostimulation design. In this perspective, we propose that incorporating neuromorphic technologies into neuroprostheses could be an effective approach for developing more natural human-machine interfaces, potentially leading to advancements in device performance, acceptability, and embeddability. We also highlight ongoing challenges and the required actions to facilitate the future integration of these advanced technologies.
Collapse
Affiliation(s)
- Elisa Donati
- Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland.
| | - Giacomo Valle
- Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA.
| |
Collapse
|
33
|
Tai P, Ding P, Wang F, Gong A, Li T, Zhao L, Su L, Fu Y. Brain-computer interface paradigms and neural coding. Front Neurosci 2024; 17:1345961. [PMID: 38287988 PMCID: PMC10822902 DOI: 10.3389/fnins.2023.1345961] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Accepted: 12/28/2023] [Indexed: 01/31/2024] Open
Abstract
Brain signal patterns generated in the central nervous system of brain-computer interface (BCI) users are closely related to BCI paradigms and neural coding. In BCI systems, BCI paradigms and neural coding are critical elements for BCI research. However, so far there have been few references that clearly and systematically elaborated on the definition and design principles of the BCI paradigm as well as the definition and modeling principles of BCI neural coding. Therefore, these contents are expounded and the existing main BCI paradigms and neural coding are introduced in the review. Finally, the challenges and future research directions of BCI paradigm and neural coding were discussed, including user-centered design and evaluation for BCI paradigms and neural coding, revolutionizing the traditional BCI paradigms, breaking through the existing techniques for collecting brain signals and combining BCI technology with advanced AI technology to improve brain signal decoding performance. It is expected that the review will inspire innovative research and development of the BCI paradigm and neural coding.
Collapse
Affiliation(s)
- Pengrui Tai
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People’s Armed Police Force Engineering University, Xi’an, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Su
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| |
Collapse
|
34
|
Eisenstein M. Seven technologies to watch in 2024. Nature 2024; 625:844-848. [PMID: 38253763 DOI: 10.1038/d41586-024-00173-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/24/2024]
|
35
|
Boulingre M, Portillo-Lara R, Green RA. Biohybrid neural interfaces: improving the biological integration of neural implants. Chem Commun (Camb) 2023; 59:14745-14758. [PMID: 37991846 PMCID: PMC10720954 DOI: 10.1039/d3cc05006h] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Accepted: 11/10/2023] [Indexed: 11/24/2023]
Abstract
Implantable neural interfaces (NIs) have emerged in the clinic as outstanding tools for the management of a variety of neurological conditions caused by trauma or disease. However, the foreign body reaction triggered upon implantation remains one of the major challenges hindering the safety and longevity of NIs. The integration of tools and principles from biomaterial design and tissue engineering has been investigated as a promising strategy to develop NIs with enhanced functionality and performance. In this Feature Article, we highlight the main bioengineering approaches for the development of biohybrid NIs with an emphasis on relevant device design criteria. Technical and scientific challenges associated with the fabrication and functional assessment of technologies composed of both artificial and biological components are discussed. Lastly, we provide future perspectives related to engineering, regulatory, and neuroethical challenges to be addressed towards the realisation of the promise of biohybrid neurotechnology.
Collapse
Affiliation(s)
- Marjolaine Boulingre
- Department of Bioengineering, Imperial College London, South Kensington, London, SW7 2AZ, UK
| | - Roberto Portillo-Lara
- Department of Bioengineering, Imperial College London, South Kensington, London, SW7 2AZ, UK
| | - Rylie A Green
- Department of Bioengineering, Imperial College London, South Kensington, London, SW7 2AZ, UK
| |
Collapse
|
36
|
Canny E, Vansteensel MJ, van der Salm SMA, Müller-Putz GR, Berezutskaya J. Boosting brain-computer interfaces with functional electrical stimulation: potential applications in people with locked-in syndrome. J Neuroeng Rehabil 2023; 20:157. [PMID: 37980536 PMCID: PMC10656959 DOI: 10.1186/s12984-023-01272-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Accepted: 10/23/2023] [Indexed: 11/20/2023] Open
Abstract
Individuals with a locked-in state live with severe whole-body paralysis that limits their ability to communicate with family and loved ones. Recent advances in brain-computer interface (BCI) technology have presented a potential alternative for these people to communicate by detecting neural activity associated with attempted hand or speech movements and translating the decoded intended movements to a control signal for a computer. A technique that could potentially enrich the communication capacity of BCIs is functional electrical stimulation (FES) of paralyzed limbs and face to restore body and facial movements of paralyzed individuals, allowing to add body language and facial expression to communication BCI utterances. Here, we review the current state of the art of existing BCI and FES work in people with paralysis of body and face and propose that a combined BCI-FES approach, which has already proved successful in several applications in stroke and spinal cord injury, can provide a novel promising mode of communication for locked-in individuals.
Collapse
Affiliation(s)
- Evan Canny
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Mariska J Vansteensel
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Sandra M A van der Salm
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Laboratory of Brain-Computer Interfaces, Graz University of Technology, Graz, Austria
| | - Julia Berezutskaya
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands.
| |
Collapse
|
37
|
Andorno R, Lavazza A. How to deal with mind-reading technologies. Front Psychol 2023; 14:1290478. [PMID: 38034284 PMCID: PMC10682168 DOI: 10.3389/fpsyg.2023.1290478] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2023] [Accepted: 10/30/2023] [Indexed: 12/02/2023] Open
Affiliation(s)
- Roberto Andorno
- Institute of Biomedical Ethics and History of Medicine, University of Zurich, Zürich, Switzerland
| | - Andrea Lavazza
- Centro Universitario Internazionale, Arezzo, Italy
- Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy
| |
Collapse
|
38
|
Drew L. The rise of brain-reading technology: what you need to know. Nature 2023; 623:241-243. [PMID: 37938700 DOI: 10.1038/d41586-023-03423-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2023]
|
39
|
Ohkubo M. The emergence of non-cryogenic quantum magnetic sensors: Synergistic advancement in magnetography together with SQUID. THE REVIEW OF SCIENTIFIC INSTRUMENTS 2023; 94:111501. [PMID: 38010159 DOI: 10.1063/5.0167372] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Accepted: 11/01/2023] [Indexed: 11/29/2023]
Abstract
Emerging non-superconductor quantum magnetic sensors, such as optically pumped magnetometer, fluxgate, magnetic tunnel junction, and diamond nitrogen-vacancy center, are approaching the performance of superconductor quantum interference devices (SQUIDs). These sensors are enabling magnetography for human bodies and brain-computer interface. Will they completely replace the SQUID magnetography in the near future?
Collapse
Affiliation(s)
- Masataka Ohkubo
- National Institute of Advanced Industrial Science and Technology (AIST), 1-1-1, Umezono, Tsukuba, Ibaraki 305-8568, Japan
- Faculty of Pure and Applied Sciences, University of Tsukuba, 1-1-1, Tenodai, Tsukuba, Ibaraki 305-8571, Japan
| |
Collapse
|
40
|
Whalley K. Restoring speech. Nat Rev Neurosci 2023; 24:653. [PMID: 37740095 DOI: 10.1038/s41583-023-00746-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/24/2023]
|
41
|
Sankaran N, Moses D, Chiong W, Chang EF. Recommendations for promoting user agency in the design of speech neuroprostheses. Front Hum Neurosci 2023; 17:1298129. [PMID: 37920562 PMCID: PMC10619159 DOI: 10.3389/fnhum.2023.1298129] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Accepted: 10/04/2023] [Indexed: 11/04/2023] Open
Abstract
Brain-computer interfaces (BCI) that directly decode speech from brain activity aim to restore communication in people with paralysis who cannot speak. Despite recent advances, neural inference of speech remains imperfect, limiting the ability for speech BCIs to enable experiences such as fluent conversation that promote agency - that is, the ability for users to author and transmit messages enacting their intentions. Here, we make recommendations for promoting agency based on existing and emerging strategies in neural engineering. The focus is on achieving fast, accurate, and reliable performance while ensuring volitional control over when a decoder is engaged, what exactly is decoded, and how messages are expressed. Additionally, alongside neuroscientific progress within controlled experimental settings, we argue that a parallel line of research must consider how to translate experimental successes into real-world environments. While such research will ultimately require input from prospective users, here we identify and describe design choices inspired by human-factors work conducted in existing fields of assistive technology, which address practical issues likely to emerge in future real-world speech BCI applications.
Collapse
Affiliation(s)
- Narayan Sankaran
- Kavli Center for Ethics, Science and the Public, University of California, Berkeley, Berkeley, CA, United States
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, United States
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, United States
| | - David Moses
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, United States
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, United States
| | - Winston Chiong
- Memory and Aging Center, Department of Neurology, University of California, San Francisco, San Francisco, CA, United States
| | - Edward F. Chang
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, United States
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, United States
| |
Collapse
|
42
|
Ramsey NF, Crone NE. Brain implants that enable speech pass performance milestones. Nature 2023; 620:954-955. [PMID: 37612488 DOI: 10.1038/d41586-023-02546-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/25/2023]
|
43
|
Naddaf M. Brain-reading devices allow paralysed people to talk using their thoughts. Nature 2023; 620:930-931. [PMID: 37612493 DOI: 10.1038/d41586-023-02682-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/25/2023]
|