1
|
Vitória MA, Fernandes FG, van den Boom M, Ramsey N, Raemaekers M. Decoding Single and Paired Phonemes Using 7T Functional MRI. Brain Topogr 2024:10.1007/s10548-024-01034-6. [PMID: 38261272 DOI: 10.1007/s10548-024-01034-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 01/12/2024] [Indexed: 01/24/2024]
Abstract
Several studies have shown that mouth movements related to the pronunciation of individual phonemes are represented in the sensorimotor cortex. This would theoretically allow for brain computer interfaces that are capable of decoding continuous speech by training classifiers based on the activity in the sensorimotor cortex related to the production of individual phonemes. To address this, we investigated the decodability of trials with individual and paired phonemes (pronounced consecutively with one second interval) using activity in the sensorimotor cortex. Fifteen participants pronounced 3 different phonemes and 3 combinations of two of the same phonemes in a 7T functional MRI experiment. We confirmed that support vector machine (SVM) classification of single and paired phonemes was possible. Importantly, by combining classifiers trained on single phonemes, we were able to classify paired phonemes with an accuracy of 53% (33% chance level), demonstrating that activity of isolated phonemes is present and distinguishable in combined phonemes. A SVM searchlight analysis showed that the phoneme representations are widely distributed in the ventral sensorimotor cortex. These findings provide insights about the neural representations of single and paired phonemes. Furthermore, it supports the notion that speech BCI may be feasible based on machine learning algorithms trained on individual phonemes using intracranial electrode grids.
Collapse
Affiliation(s)
- Maria Araújo Vitória
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Francisco Guerreiro Fernandes
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Max van den Boom
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
- Department of Physiology and Biomedical Engineering, Mayo Clinic, Rochester, MN, USA
| | - Nick Ramsey
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Mathijs Raemaekers
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands.
| |
Collapse
|
2
|
Easthope E, Shamei A, Liu Y, Gick B, Fels S. Cortical control of posture in fine motor skills: evidence from inter-utterance rest position. Front Hum Neurosci 2023; 17:1139569. [PMID: 37662639 PMCID: PMC10469778 DOI: 10.3389/fnhum.2023.1139569] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2023] [Accepted: 06/12/2023] [Indexed: 09/05/2023] Open
Abstract
The vocal tract continuously employs tonic muscle activity in the maintenance of postural configurations. Gamma-band activity in the sensorimotor cortex underlies transient movements during speech production, yet little is known about the neural control of postural states in the vocal tract. Simultaneously, there is evidence that sensorimotor beta-band activations contribute to a system of inhibition and state maintenance that is integral to postural control in the body. Here we use electrocorticography to assess the contribution of sensorimotor beta-band activity during speech articulation and postural maintenance, and demonstrate that beta-band activity corresponds to the inhibition of discrete speech movements and the maintenance of tonic postural states in the vocal tract. Our findings identify consistencies between the neural control of posture in speech and what is previously reported in gross motor contexts, providing support for a unified theory of postural control across gross and fine motor skills.
Collapse
Affiliation(s)
- Eric Easthope
- Human Communication Technologies Lab, Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC, Canada
| | - Arian Shamei
- Integrated Speech Research Lab, Department of Linguistics, University of British Columbia, Vancouver, BC, Canada
| | - Yadong Liu
- Integrated Speech Research Lab, Department of Linguistics, University of British Columbia, Vancouver, BC, Canada
| | - Bryan Gick
- Integrated Speech Research Lab, Department of Linguistics, University of British Columbia, Vancouver, BC, Canada
- Haskins Laboratories, New Haven, CT, United States
| | - Sidney Fels
- Human Communication Technologies Lab, Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
4
|
Huggins JE, Guger C, Aarnoutse E, Allison B, Anderson CW, Bedrick S, Besio W, Chavarriaga R, Collinger JL, Do AH, Herff C, Hohmann M, Kinsella M, Lee K, Lotte F, Müller-Putz G, Nijholt A, Pels E, Peters B, Putze F, Rupp R, Schalk G, Scott S, Tangermann M, Tubig P, Zander T. Workshops of the Seventh International Brain-Computer Interface Meeting: Not Getting Lost in Translation. BRAIN-COMPUTER INTERFACES 2019; 6:71-101. [PMID: 33033729 PMCID: PMC7539697 DOI: 10.1080/2326263x.2019.1697163] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Accepted: 10/30/2019] [Indexed: 12/11/2022]
Abstract
The Seventh International Brain-Computer Interface (BCI) Meeting was held May 21-25th, 2018 at the Asilomar Conference Grounds, Pacific Grove, California, United States. The interactive nature of this conference was embodied by 25 workshops covering topics in BCI (also called brain-machine interface) research. Workshops covered foundational topics such as hardware development and signal analysis algorithms, new and imaginative topics such as BCI for virtual reality and multi-brain BCIs, and translational topics such as clinical applications and ethical assumptions of BCI development. BCI research is expanding in the diversity of applications and populations for whom those applications are being developed. BCI applications are moving toward clinical readiness as researchers struggle with the practical considerations to make sure that BCI translational efforts will be successful. This paper summarizes each workshop, providing an overview of the topic of discussion, references for additional information, and identifying future issues for research and development that resulted from the interactions and discussion at the workshop.
Collapse
Affiliation(s)
- Jane E Huggins
- Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan, United States, 325 East Eisenhower, Room 3017; Ann Arbor, Michigan 48108-5744
| | - Christoph Guger
- g.tec medical engineering GmbH/Guger Technologies OG, Austria, Sierningstrasse 14, 4521 Schiedlberg, Austria
| | - Erik Aarnoutse
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| | - Brendan Allison
- Dept. of Cognitive Science, Mail Code 0515, University of California at San Diego, La Jolla, United States
| | - Charles W Anderson
- Department of Computer Science, Molecular, Cellular and Integrative Neurosience Program, Colorado State University, Fort Collins, CO 80523
| | - Steven Bedrick
- Center for Spoken Language Understanding, Oregon Health & Science University, Portland, OR 97239
| | - Walter Besio
- Department of Electrical, Computer, & Biomedical Engineering and Interdisciplinary Neuroscience Program, University of Rhode Island, Kingston, Rhode Island, USA, CREmedical Corp. Kingston, Rhode Island, USA
| | - Ricardo Chavarriaga
- Defitech Chair in Brain-Machine Interface (CNBI), Center for Neuroprosthetics, Ecole Polytechnique Fédérale de Lausanne - EPFL, Switzerland
| | - Jennifer L Collinger
- University of Pittsburgh, Department of Physical Medicine and Rehabilitation, VA Pittsburgh Healthcare System, Department of Veterans Affairs, 3520 5th Ave, Pittsburgh, PA, 15213
| | - An H Do
- UC Irvine Brain Computer Interface Lab, Department of Neurology, University of California, Irvine
| | - Christian Herff
- School of Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | - Matthias Hohmann
- Max Planck Institute for Intelligent Systems, Department for Empirical Inference, Max-Planck-Ring 4, 72074 Tübingen, Germany
| | - Michelle Kinsella
- Oregon Health & Science University, Institute on Development & Disability, 707 SW Gaines St, #1290, Portland, OR 97239
| | - Kyuhwa Lee
- Swiss Federal Institute of Technology in Lausanne-EPFL
| | - Fabien Lotte
- Inria Bordeaux Sud-Ouest, LaBRI (Univ. Bordeaux/CNRS/Bordeaux INP), 200 avenue de la vieille tour, 33405, Talence Cedex, France
| | | | - Anton Nijholt
- Faculty EEMCS, University of Twente, Enschede, The Netherlands
| | - Elmar Pels
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| | - Betts Peters
- Oregon Health & Science University, Institute on Development & Disability, 707 SW Gaines St, #1290, Portland, OR 97239
| | - Felix Putze
- University of Bremen, Germany, Cognitive Systems Lab, University of Bremen, Enrique-Schmidt-Straße 5 (Cartesium), 28359 Bremen
| | - Rüdiger Rupp
- Spinal Cord Injury Center, Heidelberg University Hospital
| | - Gerwin Schalk
- National Center for Adaptive Neurotechnologies, Wadsworth Center, NYS Dept. of Health, Dept. of Neurology, Albany Medical College, Dept. of Biomed. Sci., State Univ. of New York at Albany, Center for Medical Sciences 2003, 150 New Scotland Avenue, Albany, New York 12208
| | - Stephanie Scott
- Department of Media Communications, Colorado State University, Fort Collins, CO 80523
| | - Michael Tangermann
- Brain State Decoding Lab, Cluster of Excellence BrainLinks-BrainTools, Computer Science Dept., University of Freiburg, Germany, Autonomous Intelligent Systems Lab, Computer Science Dept., University of Freiburg, Germany
| | - Paul Tubig
- Department of Philosophy, Center for Neurotechnology, University of Washington, Savery Hall, Room 361, Seattle, WA 98195
| | - Thorsten Zander
- Team PhyPA, Biological Psychology and Neuroergonomics, Technische Universität Berlin, Berlin, Germany, 7 Zander Laboratories B.V., Amsterdam, The Netherlands
| |
Collapse
|
5
|
Salari E, Freudenburg ZV, Vansteensel MJ, Ramsey NF. The influence of prior pronunciations on sensorimotor cortex activity patterns during vowel production. J Neural Eng 2018; 15:066025. [PMID: 30238924 PMCID: PMC6240458 DOI: 10.1088/1741-2552/aae329] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
Abstract
OBJECTIVE In recent years, brain-computer interface (BCI) systems have been investigated for their potential as a communication device to assist people with severe paralysis. Decoding speech sensorimotor cortex activity is a promising avenue for the generation of BCI control signals, but is complicated by variability in neural patterns, leading to suboptimal decoding. We investigated whether neural pattern variability associated with sound pronunciation can be explained by prior pronunciations and determined to what extent prior speech affects BCI decoding accuracy. APPROACH Neural patterns in speech motor areas were evaluated with electrocorticography in five epilepsy patients, who performed a simple speech task that involved pronunciation of the /i/ sound, preceded by either silence, the /a/ sound or the /u/ sound. MAIN RESULTS The neural pattern related to the /i/ sound depends on previous sounds and is therefore associated with multiple distinct sensorimotor patterns, which is likely to reflect differences in the movements towards this sound. We also show that these patterns still contain a commonality that is distinct from the other vowel sounds (/a/ and /u/). Classification accuracies for the decoding of different sounds do increase, however, when the multiple patterns for the /i/ sound are taken into account. Simply including multiple forms of the /i/ vowel in the training set for the creation of a single /i/ model performs as well as training individual models for each /i/ variation. SIGNIFICANCE Our results are of interest for the development of BCIs that aim to decode speech sounds from the sensorimotor cortex, since they argue that a multitude of cortical activity patterns associated with speech movements can be reduced to a basis set of models which reflect meaningful language units (vowels), yet it is important to account for the variety of neural patterns associated with a single sound in the training process.
Collapse
Affiliation(s)
- E Salari
- Department of Neurology and Neurosurgery, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht, Netherlands
| | | | | | | |
Collapse
|