1
|
Simistira Liwicki F, Gupta V, Saini R, De K, Abid N, Rakesh S, Wellington S, Wilson H, Liwicki M, Eriksson J. Bimodal electroencephalography-functional magnetic resonance imaging dataset for inner-speech recognition. Sci Data 2023; 10:378. [PMID: 37311807 DOI: 10.1038/s41597-023-02286-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2022] [Accepted: 06/01/2023] [Indexed: 06/15/2023] Open
Abstract
The recognition of inner speech, which could give a 'voice' to patients that have no ability to speak or move, is a challenge for brain-computer interfaces (BCIs). A shortcoming of the available datasets is that they do not combine modalities to increase the performance of inner speech recognition. Multimodal datasets of brain data enable the fusion of neuroimaging modalities with complimentary properties, such as the high spatial resolution of functional magnetic resonance imaging (fMRI) and the temporal resolution of electroencephalography (EEG), and therefore are promising for decoding inner speech. This paper presents the first publicly available bimodal dataset containing EEG and fMRI data acquired nonsimultaneously during inner-speech production. Data were obtained from four healthy, right-handed participants during an inner-speech task with words in either a social or numerical category. Each of the 8-word stimuli were assessed with 40 trials, resulting in 320 trials in each modality for each participant. The aim of this work is to provide a publicly available bimodal dataset on inner speech, contributing towards speech prostheses.
Collapse
Affiliation(s)
- Foteini Simistira Liwicki
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden.
| | - Vibha Gupta
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Rajkumar Saini
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Kanjar De
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Nosheen Abid
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Sumit Rakesh
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | | | - Holly Wilson
- University of Bath, Department of Computer Science, Bath, UK
| | - Marcus Liwicki
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Johan Eriksson
- Umeå University, Department of Integrative Medical Biology (IMB) and Umeå Center for Functional Brain Imaging (UFBI), Umeå, Sweden
| |
Collapse
|
2
|
Bedel HA, Sivgin I, Dalmaz O, Dar SUH, Çukur T. BolT: Fused window transformers for fMRI time series analysis. Med Image Anal 2023; 88:102841. [PMID: 37224718 DOI: 10.1016/j.media.2023.102841] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 02/07/2023] [Accepted: 05/10/2023] [Indexed: 05/26/2023]
Abstract
Deep-learning models have enabled performance leaps in analysis of high-dimensional functional MRI (fMRI) data. Yet, many previous methods are suboptimally sensitive for contextual representations across diverse time scales. Here, we present BolT, a blood-oxygen-level-dependent transformer model, for analyzing multi-variate fMRI time series. BolT leverages a cascade of transformer encoders equipped with a novel fused window attention mechanism. Encoding is performed on temporally-overlapped windows within the time series to capture local representations. To integrate information temporally, cross-window attention is computed between base tokens in each window and fringe tokens from neighboring windows. To gradually transition from local to global representations, the extent of window overlap and thereby number of fringe tokens are progressively increased across the cascade. Finally, a novel cross-window regularization is employed to align high-level classification features across the time series. Comprehensive experiments on large-scale public datasets demonstrate the superior performance of BolT against state-of-the-art methods. Furthermore, explanatory analyses to identify landmark time points and regions that contribute most significantly to model decisions corroborate prominent neuroscientific findings in the literature.
Collapse
Affiliation(s)
- Hasan A Bedel
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Irmak Sivgin
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Onat Dalmaz
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Salman U H Dar
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Neuroscience Program, Bilkent University, Ankara 06800, Turkey.
| |
Collapse
|
3
|
Bruurmijn LCM, Raemaekers M, Branco MP, Vansteensel MJ, Ramsey NF. Decoding attempted phantom hand movements from ipsilateral sensorimotor areas after amputation. J Neural Eng 2021; 18. [PMID: 34433158 DOI: 10.1088/1741-2552/ac20e4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2021] [Accepted: 08/25/2021] [Indexed: 11/12/2022]
Abstract
Objective.The sensorimotor cortex is often selected as target in the development of a Brain-Computer Interface, as activation patterns from this region can be robustly decoded to discriminate between different movements the user executes. Up until recently, such BCIs were primarily based on activity in the contralateral hemisphere, where decoding movements still works even years after denervation. However, there is increasing evidence for a role of the sensorimotor cortex in controlling the ipsilateral body. The aim of this study is to investigate the effects of denervation on the movement representation on the ipsilateral sensorimotor cortex.Approach.Eight subjects with acquired above-elbow arm amputation and nine controls performed a task in which they made (or attempted to make with their phantom hand) six different gestures from the American Manual Alphabet. Brain activity was measured using 7T functional MRI, and a classifier was trained to discriminate between activation patterns on four different regions of interest (ROIs) on the ipsilateral sensorimotor cortex.Main results.Classification scores showed that decoding was possible and significantly better than chance level for both the phantom and intact hands from all ROIs. Decoding both the left (intact) and right (phantom) hand from the same hemisphere was also possible with above-chance level classification score.Significance.The possibility to decode both hands from the same hemisphere, even years after denervation, indicates that implantation of motor-electrodes for BCI control possibly need only cover a single hemisphere, making surgery less invasive, and increasing options for people with lateralized damage to motor cortex like after stroke.
Collapse
Affiliation(s)
- L C M Bruurmijn
- UMC Utrecht Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - M Raemaekers
- UMC Utrecht Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - M P Branco
- UMC Utrecht Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - M J Vansteensel
- UMC Utrecht Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - N F Ramsey
- UMC Utrecht Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
4
|
De Sousa C, Gaillard C, Di Bello F, Ben Hadj Hassen S, Ben Hamed S. Behavioral validation of novel high resolution attention decoding method from multi-units & local field potentials. Neuroimage 2021; 231:117853. [PMID: 33582274 DOI: 10.1016/j.neuroimage.2021.117853] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2020] [Revised: 02/03/2021] [Accepted: 02/04/2021] [Indexed: 11/28/2022] Open
Abstract
The ability to access brain information in real-time is crucial both for a better understanding of cognitive functions and for the development of therapeutic applications based on brain-machine interfaces. Great success has been achieved in the field of neural motor prosthesis. Progress is still needed in the real-time decoding of higher-order cognitive processes such as covert attention. Recently, we showed that we can track the location of the attentional spotlight using classification methods applied to prefrontal multi-unit activity (MUA) in the non-human primates. Importantly, we demonstrated that the decoded (x,y) attentional spotlight parametrically correlates with the behavior of the monkeys thus validating our decoding of attention. We also demonstrate that this spotlight is extremely dynamic. Here, in order to get closer to non-invasive decoding applications, we extend our previous work to local field potential signals (LFP). Specifically, we achieve, for the first time, high decoding accuracy of the (x,y) location of the attentional spotlight from prefrontal LFP signals, to a degree comparable to that achieved from MUA signals, and we show that this LFP content is predictive of behavior. This LFP attention-related information is maximal in the gamma band (30-250 Hz), peaking between 60 to 120 Hz. In addition, we introduce a novel two-step decoding procedure based on the labelling of maximally attention-informative trials during the decoding procedure. This procedure strongly improves the correlation between our real-time MUA and LFP based decoding and behavioral performance, thus further refining the functional relevance of this real-time decoding of the (x,y) locus of attention. This improvement is more marked for LFP signals than for MUA signals. Overall, this study demonstrates that the attentional spotlight can be accessed from LFP frequency content, in real-time, and can be used to drive high-information content cognitive brain-machine interfaces for the development of new therapeutic strategies.
Collapse
Affiliation(s)
- Carine De Sousa
- Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Université Claude Bernard Lyon I, 67 Boulevard Pinel, 69675 Bron Cedex, France.
| | - C Gaillard
- Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Université Claude Bernard Lyon I, 67 Boulevard Pinel, 69675 Bron Cedex, France
| | - F Di Bello
- Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Université Claude Bernard Lyon I, 67 Boulevard Pinel, 69675 Bron Cedex, France
| | - S Ben Hadj Hassen
- Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Université Claude Bernard Lyon I, 67 Boulevard Pinel, 69675 Bron Cedex, France
| | - S Ben Hamed
- Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Université Claude Bernard Lyon I, 67 Boulevard Pinel, 69675 Bron Cedex, France.
| |
Collapse
|
5
|
Heunis S, Lamerichs R, Zinger S, Caballero‐Gaudes C, Jansen JFA, Aldenkamp B, Breeuwer M. Quality and denoising in real-time functional magnetic resonance imaging neurofeedback: A methods review. Hum Brain Mapp 2020; 41:3439-3467. [PMID: 32333624 PMCID: PMC7375116 DOI: 10.1002/hbm.25010] [Citation(s) in RCA: 22] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2019] [Revised: 02/13/2020] [Accepted: 04/03/2020] [Indexed: 01/31/2023] Open
Abstract
Neurofeedback training using real-time functional magnetic resonance imaging (rtfMRI-NF) allows subjects voluntary control of localised and distributed brain activity. It has sparked increased interest as a promising non-invasive treatment option in neuropsychiatric and neurocognitive disorders, although its efficacy and clinical significance are yet to be determined. In this work, we present the first extensive review of acquisition, processing and quality control methods available to improve the quality of the neurofeedback signal. Furthermore, we investigate the state of denoising and quality control practices in 128 recently published rtfMRI-NF studies. We found: (a) that less than a third of the studies reported implementing standard real-time fMRI denoising steps, (b) significant room for improvement with regards to methods reporting and (c) the need for methodological studies quantifying and comparing the contribution of denoising steps to the neurofeedback signal quality. Advances in rtfMRI-NF research depend on reproducibility of methods and results. Notably, a systematic effort is needed to build up evidence that disentangles the various mechanisms influencing neurofeedback effects. To this end, we recommend that future rtfMRI-NF studies: (a) report implementation of a set of standard real-time fMRI denoising steps according to a proposed COBIDAS-style checklist (https://osf.io/kjwhf/), (b) ensure the quality of the neurofeedback signal by calculating and reporting community-informed quality metrics and applying offline control checks and (c) strive to adopt transparent principles in the form of methods and data sharing and support of open-source rtfMRI-NF software. Code and data for reproducibility, as well as an interactive environment to explore the study data, can be accessed at https://github.com/jsheunis/quality-and-denoising-in-rtfmri-nf.
Collapse
Affiliation(s)
- Stephan Heunis
- Department of Electrical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
- Department of Research and DevelopmentEpilepsy Centre KempenhaegheHeezeThe Netherlands
| | - Rolf Lamerichs
- Department of Electrical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
- Department of Research and DevelopmentEpilepsy Centre KempenhaegheHeezeThe Netherlands
- Philips ResearchEindhovenThe Netherlands
| | - Svitlana Zinger
- Department of Electrical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
- Department of Research and DevelopmentEpilepsy Centre KempenhaegheHeezeThe Netherlands
| | | | - Jacobus F. A. Jansen
- Department of Electrical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
- Department of RadiologyMaastricht University Medical CentreMaastrichtThe Netherlands
- School for Mental Health and NeuroscienceMaastrichtThe Netherlands
| | - Bert Aldenkamp
- Department of Electrical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
- Department of Research and DevelopmentEpilepsy Centre KempenhaegheHeezeThe Netherlands
- School for Mental Health and NeuroscienceMaastrichtThe Netherlands
- Laboratory for Clinical and Experimental Neurophysiology, Neurobiology and NeuropsychologyGhent University HospitalGhentBelgium
- Department of NeurologyMaastricht University Medical CenterMaastrichtThe Netherlands
| | - Marcel Breeuwer
- Department of Biomedical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
- Philips HealthcareBestThe Netherlands
| |
Collapse
|
6
|
Abstract
Brain-computer interfaces (BCIs) based on functional magnetic resonance imaging (fMRI) provide an important complement to other noninvasive BCIs. While fMRI has several disadvantages (being nonportable, methodologically challenging, costly, and noisy), it is the only method providing high spatial resolution whole-brain coverage of brain activation. These properties allow relating mental activities to specific brain regions and networks providing a transparent scheme for BCI users to encode information and for real-time fMRI BCI systems to decode the intents of the user. Various mental activities have been used successfully in fMRI BCIs so far that can be classified into the four categories: (a) higher-order cognitive tasks (e.g., mental calculation), (b) covert language-related tasks (e.g., mental speech and mental singing), (c) imagery tasks (motor, visual, auditory, tactile, and emotion imagery), and (d) selective attention tasks (visual, auditory, and tactile attention). While the ultimate spatial and temporal resolution of fMRI BCIs is limited by the physiologic properties of the hemodynamic response, technical and analytical advances will likely lead to substantially improved fMRI BCIs in the future using, for example, decoding of imagined letter shapes at 7T as the basis for more "natural" communication BCIs.
Collapse
Affiliation(s)
- Bettina Sorger
- Department of Cognitive Neuroscience, Maastricht University, Maastricht, The Netherlands; Maastricht Brain Imaging Center (M-BIC), Maastricht, The Netherlands
| | - Rainer Goebel
- Department of Cognitive Neuroscience, Maastricht University, Maastricht, The Netherlands; Maastricht Brain Imaging Center (M-BIC), Maastricht, The Netherlands.
| |
Collapse
|
7
|
Abstract
Human brain function research has evolved dramatically in the last decades. In this chapter the role of modern methods of recording brain activity in understanding human brain function is explained. Current knowledge of brain function relevant to brain-computer interface (BCI) research is detailed, with an emphasis on the motor system which provides an exceptional level of detail to decoding of intended or attempted movements in paralyzed beneficiaries of BCI technology and translation to computer-mediated actions. BCI technologies that stand to benefit the most of the detailed organization of the human cortex are, and for the foreseeable future are likely to be, reliant on intracranial electrodes. These evolving technologies are expected to enable severely paralyzed people to regain the faculty of movement and speech in the coming decades.
Collapse
Affiliation(s)
- Nick F Ramsey
- Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands.
| |
Collapse
|
8
|
Andersson P, Ragni F, Lingnau A. Visual imagery during real-time fMRI neurofeedback from occipital and superior parietal cortex. Neuroimage 2019; 200:332-343. [DOI: 10.1016/j.neuroimage.2019.06.057] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2018] [Revised: 06/11/2019] [Accepted: 06/24/2019] [Indexed: 01/15/2023] Open
|
9
|
Kopel R, Sladky R, Laub P, Koush Y, Robineau F, Hutton C, Weiskopf N, Vuilleumier P, Van De Ville D, Scharnowski F. No time for drifting: Comparing performance and applicability of signal detrending algorithms for real-time fMRI. Neuroimage 2019; 191:421-429. [PMID: 30818024 PMCID: PMC6503944 DOI: 10.1016/j.neuroimage.2019.02.058] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2018] [Revised: 02/19/2019] [Accepted: 02/22/2019] [Indexed: 01/15/2023] Open
Abstract
As a consequence of recent technological advances in the field of functional magnetic resonance imaging (fMRI), results can now be made available in real-time. This allows for novel applications such as online quality assurance of the acquisition, intra-operative fMRI, brain-computer-interfaces, and neurofeedback. To that aim, signal processing algorithms for real-time fMRI must reliably correct signal contaminations due to physiological noise, head motion, and scanner drift. The aim of this study was to compare performance of the commonly used online detrending algorithms exponential moving average (EMA), incremental general linear model (iGLM) and sliding window iGLM (iGLMwindow). For comparison, we also included offline detrending algorithms (i.e., MATLAB's and SPM8's native detrending functions). Additionally, we optimized the EMA control parameter, by assessing the algorithm's performance on a simulated data set with an exhaustive set of realistic experimental design parameters. First, we optimized the free parameters of the online and offline detrending algorithms. Next, using simulated data, we systematically compared the performance of the algorithms with respect to varying levels of Gaussian and colored noise, linear and non-linear drifts, spikes, and step function artifacts. Additionally, using in vivo data from an actual rt-fMRI experiment, we validated our results in a post hoc offline comparison of the different detrending algorithms. Quantitative measures show that all algorithms perform well, even though they are differently affected by the different artifact types. The iGLM approach outperforms the other online algorithms and achieves online detrending performance that is as good as that of offline procedures. These results may guide developers and users of real-time fMRI analyses tools to best account for the problem of signal drifts in real-time fMRI.
Collapse
Affiliation(s)
- R Kopel
- Department of Radiology and Medical Informatics, CIBM, University of Geneva, Geneva, Switzerland; Institute of Bioengineering, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - R Sladky
- Department of Psychiatric, Psychotherapy and Psychosomatics, Psychiatric Hospital, University of Zürich, Zürich, Switzerland; Social, Cognitive and Affective Neuroscience Unit, Department of Basic Psychological Research and Research Methods, Faculty of Psychology, University of Vienna, Vienna, Austria.
| | - P Laub
- Institute of Bioengineering, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Y Koush
- Department of Radiology and Medical Informatics, CIBM, University of Geneva, Geneva, Switzerland; Institute of Bioengineering, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland; Department of Radiology and Medical Imaging, Yale University, New Haven, USA
| | - F Robineau
- Laboratory for Behavioral Neurology and Imaging of Cognition, Department of Neuroscience, University Medical Center, Geneva, Switzerland; Geneva Neuroscience Center, Geneva, Switzerland
| | - C Hutton
- Wellcome Trust Centre for Neuroimaging, UCL Institute of Neurology, University College London, London, UK
| | - N Weiskopf
- Geneva Neuroscience Center, Geneva, Switzerland; Department of Neurophysics, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - P Vuilleumier
- Laboratory for Behavioral Neurology and Imaging of Cognition, Department of Neuroscience, University Medical Center, Geneva, Switzerland; Geneva Neuroscience Center, Geneva, Switzerland
| | - D Van De Ville
- Department of Radiology and Medical Informatics, CIBM, University of Geneva, Geneva, Switzerland; Institute of Bioengineering, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - F Scharnowski
- Department of Radiology and Medical Informatics, CIBM, University of Geneva, Geneva, Switzerland; Institute of Bioengineering, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland; Department of Psychiatric, Psychotherapy and Psychosomatics, Psychiatric Hospital, University of Zürich, Zürich, Switzerland; Neuroscience Center Zürich, University of Zürich and Swiss Federal Institute of Technology, Winterthurerstr. 190, 8057, Zürich, Switzerland; Zürich Center for Integrative Human Physiology (ZIHP), University of Zürich, Winterthurerstr. 190, 8057, Zürich, Switzerland
| |
Collapse
|
10
|
Padfield N, Zabalza J, Zhao H, Masero V, Ren J. EEG-Based Brain-Computer Interfaces Using Motor-Imagery: Techniques and Challenges. SENSORS (BASEL, SWITZERLAND) 2019; 19:E1423. [PMID: 30909489 PMCID: PMC6471241 DOI: 10.3390/s19061423] [Citation(s) in RCA: 157] [Impact Index Per Article: 31.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/30/2019] [Revised: 03/10/2019] [Accepted: 03/19/2019] [Indexed: 12/11/2022]
Abstract
Electroencephalography (EEG)-based brain-computer interfaces (BCIs), particularly those using motor-imagery (MI) data, have the potential to become groundbreaking technologies in both clinical and entertainment settings. MI data is generated when a subject imagines the movement of a limb. This paper reviews state-of-the-art signal processing techniques for MI EEG-based BCIs, with a particular focus on the feature extraction, feature selection and classification techniques used. It also summarizes the main applications of EEG-based BCIs, particularly those based on MI data, and finally presents a detailed discussion of the most prevalent challenges impeding the development and commercialization of EEG-based BCIs.
Collapse
Affiliation(s)
- Natasha Padfield
- Centre for Signal and Image Processing, University of Strathclyde, Glasgow G1 1XW, UK.
| | - Jaime Zabalza
- Centre for Signal and Image Processing, University of Strathclyde, Glasgow G1 1XW, UK.
| | - Huimin Zhao
- School of Computer Sciences, Guangdong Polytechnic Normal University, Guangzhou 510665, China.
- The Guangzhou Key Laboratory of Digital Content Processing and Security Technologies, Guangzhou 510665, China.
| | - Valentin Masero
- Department of Computer Systems and Telematics Engineering, Universidad de Extremadura, 06007 Badajoz, Spain.
| | - Jinchang Ren
- Centre for Signal and Image Processing, University of Strathclyde, Glasgow G1 1XW, UK.
- School of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China.
| |
Collapse
|
11
|
Cinel C, Valeriani D, Poli R. Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects. Front Hum Neurosci 2019; 13:13. [PMID: 30766483 PMCID: PMC6365771 DOI: 10.3389/fnhum.2019.00013] [Citation(s) in RCA: 48] [Impact Index Per Article: 9.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2018] [Accepted: 01/10/2019] [Indexed: 01/10/2023] Open
Abstract
Recent advances in neuroscience have paved the way to innovative applications that cognitively augment and enhance humans in a variety of contexts. This paper aims at providing a snapshot of the current state of the art and a motivated forecast of the most likely developments in the next two decades. Firstly, we survey the main neuroscience technologies for both observing and influencing brain activity, which are necessary ingredients for human cognitive augmentation. We also compare and contrast such technologies, as their individual characteristics (e.g., spatio-temporal resolution, invasiveness, portability, energy requirements, and cost) influence their current and future role in human cognitive augmentation. Secondly, we chart the state of the art on neurotechnologies for human cognitive augmentation, keeping an eye both on the applications that already exist and those that are emerging or are likely to emerge in the next two decades. Particularly, we consider applications in the areas of communication, cognitive enhancement, memory, attention monitoring/enhancement, situation awareness and complex problem solving, and we look at what fraction of the population might benefit from such technologies and at the demands they impose in terms of user training. Thirdly, we briefly review the ethical issues associated with current neuroscience technologies. These are important because they may differentially influence both present and future research on (and adoption of) neurotechnologies for human cognitive augmentation: an inferior technology with no significant ethical issues may thrive while a superior technology causing widespread ethical concerns may end up being outlawed. Finally, based on the lessons learned in our analysis, using past trends and considering other related forecasts, we attempt to forecast the most likely future developments of neuroscience technology for human cognitive augmentation and provide informed recommendations for promising future research and exploitation avenues.
Collapse
Affiliation(s)
- Caterina Cinel
- Brain Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Colchester, United Kingdom
| | - Davide Valeriani
- Brain Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Colchester, United Kingdom
- Department of Otolaryngology, Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, United States
| | - Riccardo Poli
- Brain Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Colchester, United Kingdom
| |
Collapse
|
12
|
Van den Boom MA, Jansma JM, Ramsey NF. Rapid acquisition of dynamic control over DLPFC using real-time fMRI feedback. Eur Neuropsychopharmacol 2018; 28:1194-1205. [PMID: 30217551 PMCID: PMC6420021 DOI: 10.1016/j.euroneuro.2018.08.508] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/20/2017] [Revised: 08/08/2018] [Accepted: 08/14/2018] [Indexed: 12/20/2022]
Abstract
It has been postulated that gaining control over activity in the dorsolateral prefrontal cortex (DLPFC), a key region of the working memory brain network, may be beneficial for cognitive performance and treatment of certain psychiatric disorders. Several studies have reported that, with neurofeedback training, subjects can learn to increase DLPFC activity. However, improvement of dynamic control in terms of switching between low and high activity in DLPFC brain states may potentially constitute more effective self-regulation. Here, we report on feasibility of obtaining dynamic control over DLPFC, meaning the ability to both in- and decrease activity at will, within a single functional MRI scan session. Two groups of healthy volunteers (N = 24) were asked to increase and decrease activity in the left DLPFC as often as possible during fMRI scans (at 7 Tesla), while receiving real-time visual feedback. The experimental group practiced with real-time feedback, whereas the control group received sham feedback. The experimental group significantly increased the speed of intentionally alternating DLPFC activity, while performance of the control group did not change. Analysis of the characteristics of the BOLD signal during successful trials revealed that training with neurofeedback predominantly reduced the time for the DLPFC to return to baseline after activation. These results provide a preliminary indication that people may be able to learn to dynamically down-regulate the level of physiological activity in the DLPFC, and may have implications for psychiatric disorders where DLPFC plays a role.
Collapse
Affiliation(s)
- Max Alexander Van den Boom
- Department of Neurology and Neurosurgery, Brain Center Rudolf Magnus, University Medical Center Utrecht, P.O. Box 85500, Utrecht, The Netherlands.
| | - Johan Martijn Jansma
- Department of Neurosurgery, Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands
| | - Nick Franciscus Ramsey
- Department of Neurology and Neurosurgery, Brain Center Rudolf Magnus, University Medical Center Utrecht, P.O. Box 85500, Utrecht, The Netherlands.
| |
Collapse
|
13
|
Ekanayake J, Hutton C, Ridgway G, Scharnowski F, Weiskopf N, Rees G. Real-time decoding of covert attention in higher-order visual areas. Neuroimage 2018; 169:462-472. [PMID: 29247807 PMCID: PMC5864512 DOI: 10.1016/j.neuroimage.2017.12.019] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2017] [Revised: 12/06/2017] [Accepted: 12/09/2017] [Indexed: 12/21/2022] Open
Abstract
Brain-computer-interfaces (BCI) provide a means of using human brain activations to control devices for communication. Until now this has only been demonstrated in primary motor and sensory brain regions, using surgical implants or non-invasive neuroimaging techniques. Here, we provide proof-of-principle for the use of higher-order brain regions involved in complex cognitive processes such as attention. Using realtime fMRI, we implemented an online 'winner-takes-all approach' with quadrant-specific parameter estimates, to achieve single-block classification of brain activations. These were linked to the covert allocation of attention to real-world images presented at 4-quadrant locations. Accuracies in three target regions were significantly above chance, with individual decoding accuracies reaching upto 70%. By utilising higher order mental processes, 'cognitive BCIs' access varied and therefore more versatile information, potentially providing a platform for communication in patients who are unable to speak or move due to brain injury.
Collapse
Affiliation(s)
- Jinendra Ekanayake
- Wellcome Trust Centre for Interventional and Surgical Sciences, University College London, London, United Kingdom; Wellcome Trust Centre for Neuroimaging, University College London, London, United Kingdom; Institute of Cognitive Neuroscience, University College London, London, United Kingdom.
| | - Chloe Hutton
- Siemens Molecular Imaging, Oxford, United Kingdom
| | | | - Frank Scharnowski
- Psychiatric University Hospital, University of Zürich, Lenggstrasse 31, 8032 Zürich, Switzerland; Neuroscience Center Zürich, University of Zürich and Swiss Federal Institute of Technology, Winterthurerstr. 190, 8057 Zürich, Switzerland; Zürich Center for Integrative Human Physiology (ZIHP), University of Zürich, Winterthurerstr. 190, 8057 Zürich, Switzerland
| | - Nikolaus Weiskopf
- Wellcome Trust Centre for Neuroimaging, University College London, London, United Kingdom; Department of Neurophysics, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Geraint Rees
- Wellcome Trust Centre for Neuroimaging, University College London, London, United Kingdom; Institute of Cognitive Neuroscience, University College London, London, United Kingdom
| |
Collapse
|
14
|
Bruurmijn MLCM, Pereboom IPL, Vansteensel MJ, Raemaekers MAH, Ramsey NF. Preservation of hand movement representation in the sensorimotor areas of amputees. Brain 2017; 140:3166-3178. [PMID: 29088322 DOI: 10.1093/brain/awx274] [Citation(s) in RCA: 39] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2017] [Accepted: 08/25/2017] [Indexed: 11/13/2022] Open
Abstract
Denervation due to amputation is known to induce cortical reorganization in the sensorimotor cortex. Although there is evidence that reorganization does not lead to a complete loss of the representation of the phantom limb, it is unclear to what extent detailed, finger-specific activation patterns are preserved in motor cortex, an issue that is also relevant for development of brain-computer interface solutions for paralysed people. We applied machine learning to obtain a quantitative measure for the functional organization within the motor and adjacent cortices in amputees, using high resolution functional MRI and attempted hand gestures. Subjects with above-elbow arm amputation (n = 8) and non-amputated controls (n = 9) made several gestures with either their right or left hand. Amputees attempted to make gestures with their amputated hand. Images were acquired using 7 T functional MRI. The sensorimotor cortex was divided into four regions, and activity patterns were classified in individual subjects using a support vector machine. Classification scores were significantly above chance for all subjects and all hands, and were highly similar between amputees and controls in most regions. Decodability of phantom movements from primary motor cortex reached the levels of right hand movements in controls. Attempted movements were successfully decoded from primary sensory cortex in amputees, albeit lower than in controls but well above chance level despite absence of somatosensory feedback. There was no significant correlation between decodability and years since amputation, or age. The ability to decode attempted gestures demonstrates that the detailed hand representation is preserved in motor cortex and adjacent regions after denervation. This encourages targeting sensorimotor activity patterns for development of brain-computer interfaces.
Collapse
Affiliation(s)
- Mark L C M Bruurmijn
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Isabelle P L Pereboom
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Mariska J Vansteensel
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Mathijs A H Raemaekers
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Nick F Ramsey
- Brain Center Rudolf Magnus, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
15
|
Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids. Neuroimage 2017; 180:301-311. [PMID: 28993231 DOI: 10.1016/j.neuroimage.2017.10.011] [Citation(s) in RCA: 58] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2017] [Revised: 10/04/2017] [Accepted: 10/06/2017] [Indexed: 12/19/2022] Open
Abstract
For people who cannot communicate due to severe paralysis or involuntary movements, technology that decodes intended speech from the brain may offer an alternative means of communication. If decoding proves to be feasible, intracranial Brain-Computer Interface systems can be developed which are designed to translate decoded speech into computer generated speech or to instructions for controlling assistive devices. Recent advances suggest that such decoding may be feasible from sensorimotor cortex, but it is not clear how this challenge can be approached best. One approach is to identify and discriminate elements of spoken language, such as phonemes. We investigated feasibility of decoding four spoken phonemes from the sensorimotor face area, using electrocorticographic signals obtained with high-density electrode grids. Several decoding algorithms including spatiotemporal matched filters, spatial matched filters and support vector machines were compared. Phonemes could be classified correctly at a level of over 75% with spatiotemporal matched filters. Support Vector machine analysis reached a similar level, but spatial matched filters yielded significantly lower scores. The most informative electrodes were clustered along the central sulcus. Highest scores were achieved from time windows centered around voice onset time, but a 500 ms window before onset time could also be classified significantly. The results suggest that phoneme production involves a sequence of robust and reproducible activity patterns on the cortical surface. Importantly, decoding requires inclusion of temporal information to capture the rapid shifts of robust patterns associated with articulator muscle group contraction during production of a phoneme. The high classification scores are likely to be enabled by the use of high density grids, and by the use of discrete phonemes. Implications for use in Brain-Computer Interfaces are discussed.
Collapse
|
16
|
Bassett DS, Khambhati AN. A network engineering perspective on probing and perturbing cognition with neurofeedback. Ann N Y Acad Sci 2017; 1396:126-143. [PMID: 28445589 PMCID: PMC5446287 DOI: 10.1111/nyas.13338] [Citation(s) in RCA: 46] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
Abstract
Network science and engineering provide a flexible and generalizable tool set to describe and manipulate complex systems characterized by heterogeneous interaction patterns among component parts. While classically applied to social systems, these tools have recently proven to be particularly useful in the study of the brain. In this review, we describe the nascent use of these tools to understand human cognition, and we discuss their utility in informing the meaningful and predictable perturbation of cognition in combination with the emerging capabilities of neurofeedback. To blend these disparate strands of research, we build on emerging conceptualizations of how the brain functions (as a complex network) and how we can develop and target interventions or modulations (as a form of network control). We close with an outline of current frontiers that bridge neurofeedback, connectomics, and network control theory to better understand human cognition.
Collapse
Affiliation(s)
- Danielle S. Bassett
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvania
- Department of Electrical and Systems EngineeringUniversity of PennsylvaniaPhiladelphiaPennsylvania
| | - Ankit N. Khambhati
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvania
| |
Collapse
|
17
|
De Martino F, Yacoub E, Kemper V, Moerel M, Uludağ K, De Weerd P, Ugurbil K, Goebel R, Formisano E. The impact of ultra-high field MRI on cognitive and computational neuroimaging. Neuroimage 2017; 168:366-382. [PMID: 28396293 DOI: 10.1016/j.neuroimage.2017.03.060] [Citation(s) in RCA: 73] [Impact Index Per Article: 10.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2016] [Revised: 03/20/2017] [Accepted: 03/29/2017] [Indexed: 01/14/2023] Open
Abstract
The ability to measure functional brain responses non-invasively with ultra high field MRI (7 T and above) represents a unique opportunity in advancing our understanding of the human brain. Compared to lower fields (3 T and below), ultra high field MRI has an increased sensitivity, which can be used to acquire functional images with greater spatial resolution, and greater specificity of the blood oxygen level dependent (BOLD) signal to the underlying neuronal responses. Together, increased resolution and specificity enable investigating brain functions at a submillimeter scale, which so far could only be done with invasive techniques. At this mesoscopic spatial scale, perception, cognition and behavior can be probed at the level of fundamental units of neural computations, such as cortical columns, cortical layers, and subcortical nuclei. This represents a unique and distinctive advantage that differentiates ultra high from lower field imaging and that can foster a tighter link between fMRI and computational modeling of neural networks. So far, functional brain mapping at submillimeter scale has focused on the processing of sensory information and on well-known systems for which extensive information is available from invasive recordings in animals. It remains an open challenge to extend this methodology to uniquely human functions and, more generally, to systems for which animal models may be problematic. To succeed, the possibility to acquire high-resolution functional data with large spatial coverage, the availability of computational models of neural processing as well as accurate biophysical modeling of neurovascular coupling at mesoscopic scale all appear necessary.
Collapse
Affiliation(s)
- Federico De Martino
- Department of Cognitive Neurosciences, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 ER Maastricht, The Netherlands; Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota, 2021 sixth street SE, 55455 Minneapolis, MN, USA.
| | - Essa Yacoub
- Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota, 2021 sixth street SE, 55455 Minneapolis, MN, USA
| | - Valentin Kemper
- Department of Cognitive Neurosciences, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 ER Maastricht, The Netherlands
| | - Michelle Moerel
- Department of Cognitive Neurosciences, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 ER Maastricht, The Netherlands; Maastricht Center for System Biology, Maastricht University, Universiteitssingel 60, 6229 ER Maastricht, The Netherlands
| | - Kâmil Uludağ
- Department of Cognitive Neurosciences, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 ER Maastricht, The Netherlands
| | - Peter De Weerd
- Department of Cognitive Neurosciences, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 ER Maastricht, The Netherlands
| | - Kamil Ugurbil
- Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota, 2021 sixth street SE, 55455 Minneapolis, MN, USA
| | - Rainer Goebel
- Department of Cognitive Neurosciences, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 ER Maastricht, The Netherlands
| | - Elia Formisano
- Department of Cognitive Neurosciences, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 ER Maastricht, The Netherlands; Maastricht Center for System Biology, Maastricht University, Universiteitssingel 60, 6229 ER Maastricht, The Netherlands
| |
Collapse
|
18
|
Direct Two-Dimensional Access to the Spatial Location of Covert Attention in Macaque Prefrontal Cortex. Curr Biol 2016; 26:1699-1704. [DOI: 10.1016/j.cub.2016.04.054] [Citation(s) in RCA: 30] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2016] [Revised: 03/30/2016] [Accepted: 04/20/2016] [Indexed: 11/18/2022]
|
19
|
Astrand E, Wardak C, Ben Hamed S. Selective visual attention to drive cognitive brain-machine interfaces: from concepts to neurofeedback and rehabilitation applications. Front Syst Neurosci 2014; 8:144. [PMID: 25161613 PMCID: PMC4130369 DOI: 10.3389/fnsys.2014.00144] [Citation(s) in RCA: 46] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2014] [Accepted: 07/23/2014] [Indexed: 02/02/2023] Open
Abstract
Brain–machine interfaces (BMIs) using motor cortical activity to drive an external effector like a screen cursor or a robotic arm have seen enormous success and proven their great rehabilitation potential. An emerging parallel effort is now directed to BMIs controlled by endogenous cognitive activity, also called cognitive BMIs. While more challenging, this approach opens new dimensions to the rehabilitation of cognitive disorders. In the present work, we focus on BMIs driven by visuospatial attention signals and we provide a critical review of these studies in the light of the accumulated knowledge about the psychophysics, anatomy, and neurophysiology of visual spatial attention. Importantly, we provide a unique comparative overview of the several studies, ranging from non-invasive to invasive human and non-human primates studies, that decode attention-related information from ongoing neuronal activity. We discuss these studies in the light of the challenges attention-driven cognitive BMIs have to face. In a second part of the review, we discuss past and current attention-based neurofeedback studies, describing both the covert effects of neurofeedback onto neuronal activity and its overt behavioral effects. Importantly, we compare neurofeedback studies based on the amplitude of cortical activity to studies based on the enhancement of cortical information content. Last, we discuss several lines of future research and applications for attention-driven cognitive brain-computer interfaces (BCIs), including the rehabilitation of cognitive deficits, restored communication in locked-in patients, and open-field applications for enhanced cognition in normal subjects. The core motivation of this work is the key idea that the improvement of current cognitive BMIs for therapeutic and open field applications needs to be grounded in a proper interdisciplinary understanding of the physiology of the cognitive function of interest, be it spatial attention, working memory or any other cognitive signal.
Collapse
Affiliation(s)
- Elaine Astrand
- CNRS, Cognitive Neuroscience Center, UMR 5229, University of Lyon 1 Bron Cedex, France
| | - Claire Wardak
- CNRS, Cognitive Neuroscience Center, UMR 5229, University of Lyon 1 Bron Cedex, France
| | - Suliann Ben Hamed
- CNRS, Cognitive Neuroscience Center, UMR 5229, University of Lyon 1 Bron Cedex, France
| |
Collapse
|
20
|
Upregulation of the rostral anterior cingulate cortex can alter the perception of emotions: fMRI-based neurofeedback at 3 and 7 T. Brain Topogr 2014; 28:197-207. [PMID: 25087073 DOI: 10.1007/s10548-014-0384-4] [Citation(s) in RCA: 25] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2014] [Accepted: 07/14/2014] [Indexed: 10/24/2022]
Abstract
Recent advances in real-time functional magnetic resonance imaging (rt-fMRI) techniques enable online feedback about momentary brain activity from a localized region of interest. The anterior cingulate cortex (ACC) as a central hub for cognitive and emotional networks and its modulation has been suggested to elicit mood changes. In the presented real-time fMRI neurofeedback experiment at a 3 and a 7 T scanner we enabled participants to regulate ACC activity within one training session. The session consisted of three training runs of 8.5 min where subjects received online feedback about their current ACC activity. Before and after each run we presented emotional prosody. Subjects rated these stimuli according to their emotional valence and arousal, which served as an implicit mood measure. We found increases in ACC activation at 3 T (n = 15) and at 7 T (n = 9) with a higher activation success for the 3 T group. FMRI signal control of the rostral ACC depended on signal quality and predicted a valence bias in the rating of emotional prosody. Real-time fMRI neurofeedback of the ACC is feasible at different magnetic field strengths and can modulate localized ACC activity and emotion perception. It promises non-invasive therapeutic approaches for different psychiatric disorders characterized by impaired self-regulation.
Collapse
|
21
|
A hybrid method for the decoding of spatial attention using the MEG brain signals. Biomed Signal Process Control 2014. [DOI: 10.1016/j.bspc.2012.12.005] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
22
|
Morioka H, Kanemura A, Morimoto S, Yoshioka T, Oba S, Kawanabe M, Ishii S. Decoding spatial attention by using cortical currents estimated from electroencephalography with near-infrared spectroscopy prior information. Neuroimage 2013; 90:128-39. [PMID: 24374077 DOI: 10.1016/j.neuroimage.2013.12.035] [Citation(s) in RCA: 40] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2013] [Accepted: 12/18/2013] [Indexed: 11/19/2022] Open
Abstract
For practical brain-machine interfaces (BMIs), electroencephalography (EEG) and near-infrared spectroscopy (NIRS) are the only current methods that are non-invasive and available in non-laboratory environments. However, the use of EEG and NIRS involves certain inherent problems. EEG signals are generally a mixture of neural activity from broad areas, some of which may not be related to the task targeted by BMI, hence impairing BMI performance. NIRS has an inherent time delay as it measures blood flow, which therefore detracts from practical real-time BMI utility. To try to improve real environment EEG-NIRS-based BMIs, we propose here a novel methodology in which the subjects' mental states are decoded from cortical currents estimated from EEG, with the help of information from NIRS. Using a Variational Bayesian Multimodal EncephaloGraphy (VBMEG) methodology, we incorporated a novel form of NIRS-based prior to capture event related desynchronization from isolated current sources on the cortical surface. Then, we applied a Bayesian logistic regression technique to decode subjects' mental states from further sparsified current sources. Applying our methodology to a spatial attention task, we found our EEG-NIRS-based decoder exhibited significant performance improvement over decoding methods based on EEG sensor signals alone. The advancement of our methodology, decoding from current sources sparsely isolated on the cortex, was also supported by neuroscientific considerations; intraparietal sulcus, a region known to be involved in spatial attention, was a key responsible region in our task. These results suggest that our methodology is not only a practical option for EEG-NIRS-based BMI applications, but also a potential tool to investigate brain activity in non-laboratory and naturalistic environments.
Collapse
Affiliation(s)
- Hiroshi Morioka
- ATR Neural Information Analysis Laboratories, Kyoto 619-0288, Japan; Graduate School of Informatics, Kyoto University, Kyoto 611-0011, Japan
| | | | - Satoshi Morimoto
- ATR Neural Information Analysis Laboratories, Kyoto 619-0288, Japan
| | - Taku Yoshioka
- ATR Neural Information Analysis Laboratories, Kyoto 619-0288, Japan
| | - Shigeyuki Oba
- Graduate School of Informatics, Kyoto University, Kyoto 611-0011, Japan
| | - Motoaki Kawanabe
- ATR Neural Information Analysis Laboratories, Kyoto 619-0288, Japan
| | - Shin Ishii
- ATR Neural Information Analysis Laboratories, Kyoto 619-0288, Japan; Graduate School of Informatics, Kyoto University, Kyoto 611-0011, Japan.
| |
Collapse
|
23
|
Bleichner MG, Jansma JM, Sellmeijer J, Raemaekers M, Ramsey NF. Give Me a Sign: Decoding Complex Coordinated Hand Movements Using High-Field fMRI. Brain Topogr 2013; 27:248-57. [DOI: 10.1007/s10548-013-0322-x] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2013] [Accepted: 10/01/2013] [Indexed: 10/26/2022]
|
24
|
Andersson P, Ramsey NF, Viergever MA, Pluim JPW. 7T fMRI reveals feasibility of covert visual attention-based brain-computer interfacing with signals obtained solely from cortical grey matter accessible by subdural surface electrodes. Clin Neurophysiol 2013; 124:2191-7. [PMID: 23773675 DOI: 10.1016/j.clinph.2013.05.009] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2012] [Revised: 05/11/2013] [Accepted: 05/14/2013] [Indexed: 11/30/2022]
Abstract
OBJECTIVE There is a growing interest in brain-computer interfaces (BCI) based on invasive technologies. fMRI is exceptionally suited for selecting implant sites since BOLD signals has been shown to correlate well spatially with electric potentials recorded from the brain surface. Previous studies show that it is possible to decode covertly directed visuospatial attention using fMRI. In the present study we increase the relevance of the fMRI analysis for surface-electrodes by only allowing voxels at the surface of the brain. METHODS We classify visuospatial attention directed to four different directions (left, right, up and down) using a support vector machine while enforcing several spatial restrictions on the voxels available for the classifier. All the spatial restrictions applied are based on how accessible the brain areas are for implanted surface electrodes. RESULTS The results show that fMRI signals from only the surface of the brain are sufficient for a good classification. Data also show that the topographical pattern is quite variable across subjects. CONCLUSIONS A good control of BCI systems based on signals from surface electrodes can be achieved using visuospatial attention. Due to the large spatial variations in brain topography, individual mapping with fMRI to locate the optimal electrode implant sites is essential. SIGNIFICANCE Visuospatial attention promises to be an effective target for implanted BCI systems.
Collapse
Affiliation(s)
- Patrik Andersson
- Image Sciences Institute, University Medical Center Utrecht, The Netherlands
| | | | | | | |
Collapse
|
25
|
Andersson P, Pluim JPW, Viergever MA, Ramsey NF. Navigation of a telepresence robot via covert visuospatial attention and real-time fMRI. Brain Topogr 2012; 26:177-85. [PMID: 22965825 PMCID: PMC3536975 DOI: 10.1007/s10548-012-0252-z] [Citation(s) in RCA: 27] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2012] [Accepted: 08/17/2012] [Indexed: 11/27/2022]
Abstract
Brain-computer interfaces (BCIs) allow people with severe neurological impairment and without ability to control their muscles to regain some control over their environment. The BCI user performs a mental task to regulate brain activity, which is measured and translated into commands controlling some external device. We here show that healthy participants are capable of navigating a robot by covertly shifting their visuospatial attention. Covert Visuospatial Attention (COVISA) constitutes a very intuitive brain function for spatial navigation and does not depend on presented stimuli or on eye movements. Our robot is equipped with motors and a camera that sends visual feedback to the user who can navigate it from a remote location. We used an ultrahigh field MRI scanner (7 Tesla) to obtain fMRI signals that were decoded in real time using a support vector machine. Four healthy subjects with virtually no training succeeded in navigating the robot to at least three of four target locations. Our results thus show that with COVISA BCI, realtime robot navigation can be achieved. Since the magnitude of the fMRI signal has been shown to correlate well with the magnitude of spectral power changes in the gamma frequency band in signals measured by intracranial electrodes, the COVISA concept may in future translate to intracranial application in severely paralyzed people.
Collapse
Affiliation(s)
- Patrik Andersson
- Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands.
| | | | | | | |
Collapse
|
26
|
Andersson P, Ramsey NF, Raemaekers M, Viergever MA, Pluim JPW. Real-time decoding of the direction of covert visuospatial attention. J Neural Eng 2012; 9:045004. [DOI: 10.1088/1741-2560/9/4/045004] [Citation(s) in RCA: 30] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
|