1
|
Mijani A, Cherloo MN, Tang H, Zhan L. Spectrum-Enhanced TRCA (SE-TRCA): A novel approach for direction detection in SSVEP-based BCI. Comput Biol Med 2023; 166:107488. [PMID: 37778215 DOI: 10.1016/j.compbiomed.2023.107488] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Revised: 08/26/2023] [Accepted: 09/15/2023] [Indexed: 10/03/2023]
Abstract
The Steady State Visual Evoked Potential (SSVEP) is a widely used component in BCIs due to its high noise resistance and low equipment requirements. Recently, a novel SSVEP-based paradigm has been introduced for direction detection, in which, unlike the common SSVEP paradigms that use several frequency stimuli, only one flickering stimulus is used and it makes direction detection very challenging. So far, only the CCA method has been used for direction detection using SSVEP component analysis. Since Canonical Correlation Analysis (CCA) has some limitations, a Task-Related Component Analysis (TRCA) based method has been introduced for feature extraction to improve the direction detection performance. Although these methods have been proven efficient, they do not utilize the latent frequency information in the EEG signal. Therefore, the performance of direction detection using SSVEP component analysis is still suboptimal. For further improvement, the TRCA-based algorithm is enhanced by incorporating frequency information and introducing Spectrum-Enhanced TRCA (SE-TRCA). SE-TRCA method can utilize frequency information in conjunction with spatial information by concatenating the EEG signal and its shifted version. Accordingly, the obtained spatio-spectral filters perform as a Finite Impulse Response (FIR) filter. To evaluate the proposed SE-TRCA method, two different sorts of datasets (1) a hybrid BCI dataset (including SSVEP component for direction detection) and (2) a pure benchmark SSVEP dataset (including SSVEP component for frequency detection) have been used. Our experiments showed that the accuracy of direction detection using the proposed SE-TRCA and TRCA approaches compared to CCA-based approach have been increased by 23.35% and 28.24%, respectively. Furthermore, the accuracy of character recognition obtained from integrating P300 and SSVEP components in CCA, TRCA, and SETRCA approaches are 54.01%, 56.02%, and 58.56%, on the hybrid dataset, respectively. The evaluation of the SE-TRCA method on the benchmark SSVEP dataset demonstrates that the SE-TRCA method outperforms both CCA and TRCA, particularly regarding frequency detection accuracy. In this specific dataset, the SE-TRCA method achieved an impressive frequency detection accuracy of 98.19% for a 3-s signal, surpassing the accuracies of TRCA and CCA, which were 97.91% and 90.47%, respectively. These results demonstrated that the TRCA-based approach is more efficient than the CCA approach to extracting spatial filters. Moreover, SE-TRCA, extracting both Spectral and spatial information from the EEG signal, can capture more discriminative features from the SSVEP component and increase the accuracy of classification. The results of this study emphasize the effectiveness of the proposed SE-TRCA approach across different SSVEP paradigms and tasks. These findings provide strong evidence for the method's ability to generalize well in SSVEP analysis.
Collapse
Affiliation(s)
- AmirMohammad Mijani
- Department of Electrical and Computer Engineering, University of Pittsburgh, 3700 O'Hara Street, Pittsburgh, 15261, PA, USA.
| | - Mohammad Norizadeh Cherloo
- Department of Biomedical Engineering, University of Science and Technology (IUST), Narmak, Tehran, 16846-13114, Tehran, Iran.
| | - Haoteng Tang
- Department of Computer Science, University of Texas Rio Grande Valley, 1201 W University Dr, Edinburg, 78539, TX, USA.
| | - Liang Zhan
- Department of Electrical and Computer Engineering, University of Pittsburgh, 3700 O'Hara Street, Pittsburgh, 15261, PA, USA.
| |
Collapse
|
2
|
Bai X, Li M, Qi S, Ng ACM, Ng T, Qian W. A hybrid P300-SSVEP brain-computer interface speller with a frequency enhanced row and column paradigm. Front Neurosci 2023; 17:1133933. [PMID: 37008204 PMCID: PMC10050351 DOI: 10.3389/fnins.2023.1133933] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2022] [Accepted: 02/27/2023] [Indexed: 03/17/2023] Open
Abstract
ObjectiveThis study proposes a new hybrid brain-computer interface (BCI) system to improve spelling accuracy and speed by stimulating P300 and steady-state visually evoked potential (SSVEP) in electroencephalography (EEG) signals.MethodsA frequency enhanced row and column (FERC) paradigm is proposed to incorporate the frequency coding into the row and column (RC) paradigm so that the P300 and SSVEP signals can be evoked simultaneously. A flicker (white-black) with a specific frequency from 6.0 to 11.5 Hz with an interval of 0.5 Hz is assigned to one row or column of a 6 × 6 layout, and the row/column flashes are carried out in a pseudorandom sequence. A wavelet and support vector machine (SVM) combination is adopted for P300 detection, an ensemble task-related component analysis (TRCA) method is used for SSVEP detection, and the two detection possibilities are fused using a weight control approach.ResultsThe implemented BCI speller achieved an accuracy of 94.29% and an information transfer rate (ITR) of 28.64 bit/min averaged across 10 subjects during the online tests. An accuracy of 96.86% is obtained during the offline calibration tests, higher than that of only using P300 (75.29%) or SSVEP (89.13%). The SVM in P300 outperformed the previous linear discrimination classifier and its variants (61.90–72.22%), and the ensemble TRCA in SSVEP outperformed the canonical correlation analysis method (73.33%).ConclusionThe proposed hybrid FERC stimulus paradigm can improve the performance of the speller compared with the classical single stimulus paradigm. The implemented speller can achieve comparable accuracy and ITR to its state-of-the-art counterparts with advanced detection algorithms.
Collapse
Affiliation(s)
- Xin Bai
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China
| | - Minglun Li
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, China
| | - Shouliang Qi
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China
- *Correspondence: Shouliang Qi,
| | | | - Tit Ng
- Shenzhen Jingmei Health Technology Co., Ltd., Shenzhen, China
| | - Wei Qian
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| |
Collapse
|
3
|
Han J, Xu M, Xiao X, Yi W, Jung TP, Ming D. A high-speed hybrid brain-computer interface with more than 200 targets. J Neural Eng 2023; 20:016025. [PMID: 36608342 DOI: 10.1088/1741-2552/acb105] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Accepted: 01/06/2023] [Indexed: 01/07/2023]
Abstract
Objective. Brain-computer interfaces (BCIs) have recently made significant strides in expanding their instruction set, which has attracted wide attention from researchers. The number of targets and commands is a key indicator of how well BCIs can decode the brain's intentions. No studies have reported a BCI system with over 200 targets.Approach. This study developed the first high-speed BCI system with up to 216 targets that were encoded by a combination of electroencephalography features, including P300, motion visual evoked potential (mVEP), and steady-state visual evoked potential (SSVEP). Specifically, the hybrid BCI paradigm used the time-frequency division multiple access strategy to elaborately tag targets with P300 and mVEP of different time windows, along with SSVEP of different frequencies. The hybrid features were then decoded by task-discriminant component analysis and linear discriminant analysis. Ten subjects participated in the offline and online cued-guided spelling experiments. Other ten subjects took part in online free-spelling experiments.Main results.The offline results showed that the mVEP and P300 components were prominent in the central, parietal, and occipital regions, while the most distinct SSVEP feature was in the occipital region. The online cued-guided spelling and free-spelling results showed that the proposed BCI system achieved an average accuracy of 85.37% ± 7.49% and 86.00% ± 5.98% for the 216-target classification, resulting in an average information transfer rate (ITR) of 302.83 ± 39.20 bits min-1and 204.47 ± 37.56 bits min-1, respectively. Notably, the peak ITR could reach up to 367.83 bits min-1.Significance.This study developed the first high-speed BCI system with more than 200 targets, which holds promise for extending BCI's application scenarios.
Collapse
Affiliation(s)
- Jin Han
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
| | - Minpeng Xu
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| | - Xiaolin Xiao
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| | - Weibo Yi
- Beijing Machine and Equipment Institute, Beijing 100854, People's Republic of China
| | - Tzyy-Ping Jung
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Swartz Centre for Computational Neuroscience, University of California, San Diego, CA, United States of America
| | - Dong Ming
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| |
Collapse
|
4
|
Yue Z, Wu Q, Ren SY, Li M, Shi B, Pan Y, Wang J. A novel multiple time-frequency sequential coding strategy for hybrid brain-computer interface. Front Hum Neurosci 2022; 16:859259. [PMID: 35966991 PMCID: PMC9372511 DOI: 10.3389/fnhum.2022.859259] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2022] [Accepted: 06/28/2022] [Indexed: 11/18/2022] Open
Abstract
Background For brain-computer interface (BCI) communication, electroencephalography provides a preferable choice due to its high temporal resolution and portability over other neural recording techniques. However, current BCIs are unable to sufficiently use the information from time and frequency domains simultaneously. Thus, we proposed a novel hybrid time-frequency paradigm to investigate better ways of using the time and frequency information. Method We adopt multiple omitted stimulus potential (OSP) and steady-state motion visual evoked potential (SSMVEP) to design the hybrid paradigm. A series of pre-experiments were undertaken to study factors that would influence the feasibility of the hybrid paradigm and the interaction between multiple features. After that, a novel Multiple Time-Frequencies Sequential Coding (MTFSC) strategy was introduced and explored in experiments. Results Omissions with multiple short and long durations could effectively elicit time and frequency features, including the multi-OSP, ERP, and SSVEP in this hybrid paradigm. The MTFSC was feasible and efficient. The preliminary online analysis showed that the accuracy and the ITR of the nine-target stimulator over thirteen subjects were 89.04% and 36.37 bits/min. Significance This study first combined the SSMVEP and multi-OSP in a hybrid paradigm to produce robust and abundant time features for coding BCI. Meanwhile, the MTFSC proved feasible and showed great potential in improving performance, such as expanding the number of BCI targets by better using time information in specific stimulated frequencies. This study holds promise for designing better BCI systems with a novel coding method.
Collapse
Affiliation(s)
- Zan Yue
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Qiong Wu
- Beijing Tsinghua Changgeng Hospital, Tsinghua University, Beijing, China
| | - Shi-Yuan Ren
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Man Li
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Bin Shi
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Yu Pan
- Beijing Tsinghua Changgeng Hospital, Tsinghua University, Beijing, China
- *Correspondence: Yu Pan
| | - Jing Wang
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
- Jing Wang
| |
Collapse
|
5
|
Hong J, Qin X. Signal processing algorithms for SSVEP-based brain computer interface: State-of-the-art and recent developments. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2021. [DOI: 10.3233/jifs-201280] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
Abstract
Over past two decades, steady-state evoked potentials (SSVEP)-based brain computer interface (BCI) systems have been extensively developed. As we all know, signal processing algorithms play an important role in this BCI. However, there is no comprehensive review of the latest development of signal processing algorithms for SSVEP-based BCI. By analyzing the papers published in authoritative journals in nearly five years, signal processing algorithms of preprocessing, feature extraction and classification modules are discussed in detail. In addition, other aspects existed in this BCI are mentioned. The following key problems are solved. (1) In recent years, which signal processing algorithms are frequently used in each module? (2) Which signal processing algorithms attract more attention in recent years? (3) Which modules are the key to signal processing in BCI field? This information is very important for choosing the appropriate algorithms, and can also be considered as a reference for further research. Simultaneously, we hope that this work can provide relevant BCI researchers with valuable information about the latest trends of signal processing algorithms for SSVEP-based BCI systems.
Collapse
Affiliation(s)
- Jie Hong
- School of Mechanical Engineering, Northwestern Polytechnical University, Xi’an, Shaanxi, China
| | - Xiansheng Qin
- School of Mechanical Engineering, Northwestern Polytechnical University, Xi’an, Shaanxi, China
| |
Collapse
|
6
|
Li M, He D, Li C, Qi S. Brain-Computer Interface Speller Based on Steady-State Visual Evoked Potential: A Review Focusing on the Stimulus Paradigm and Performance. Brain Sci 2021; 11:450. [PMID: 33916189 PMCID: PMC8065759 DOI: 10.3390/brainsci11040450] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2021] [Revised: 03/25/2021] [Accepted: 03/29/2021] [Indexed: 11/17/2022] Open
Abstract
The steady-state visual evoked potential (SSVEP), measured by the electroencephalograph (EEG), has high rates of information transfer and signal-to-noise ratio, and has been used to construct brain-computer interface (BCI) spellers. In BCI spellers, the targets of alphanumeric characters are assigned different visual stimuli and the fixation of each target generates a unique SSVEP. Matching the SSVEP to the stimulus allows users to select target letters and numbers. Many BCI spellers that harness the SSVEP have been proposed over the past two decades. Various paradigms of visual stimuli, including the procedure of target selection, layout of targets, stimulus encoding, and the combination with other triggering methods are used and considered to influence on the BCI speller performance significantly. This paper reviews these stimulus paradigms and analyzes factors influencing their performance. The fundamentals of BCI spellers are first briefly described. SSVEP-based BCI spellers, where only the SSVEP is used, are classified by stimulus paradigms and described in chronological order. Furthermore, hybrid spellers that involve the use of the SSVEP are presented in parallel. Factors influencing the performance and visual fatigue of BCI spellers are provided. Finally, prevailing challenges and prospective research directions are discussed to promote the development of BCI spellers.
Collapse
Affiliation(s)
- Minglun Li
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China; (M.L.); (D.H.); (C.L.)
| | - Dianning He
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China; (M.L.); (D.H.); (C.L.)
| | - Chen Li
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China; (M.L.); (D.H.); (C.L.)
| | - Shouliang Qi
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China; (M.L.); (D.H.); (C.L.)
- Engineering Research Center of Medical Imaging and Intelligent Analysis, Ministry of Education, Northeastern University, Shenyang 110169, China
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang 110169, China
| |
Collapse
|
7
|
Katyal EA, Singla R. EEG-based hybrid QWERTY mental speller with high information transfer rate. Med Biol Eng Comput 2021; 59:633-661. [PMID: 33594631 DOI: 10.1007/s11517-020-02310-w] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2020] [Accepted: 12/30/2020] [Indexed: 11/27/2022]
Abstract
BACKGROUND Brain-computer interface (BCI) spellers detect variations in brain waves to help subjects communicate with the world. This study introduces a P300-SSVEP hybrid BCI-based QWERTY speller. METHODS The proposed hybrid speller, combines SSVEP and P300 features using a hybrid paradigm. P300 was used as time division multiplexing index which results in the use of lesser number of assumed frequencies for SSVEP elicitation. Each flickering frequency was also assigned a unique colour, to enhance system accuracy. RESULTS On the basis of 20 subjects, an average accuracy of classification of 96.42% and a mean information transfer rate (ITR) of 131.0 bits per min. (BPM) was achieved during the free spelling trial (trial-F). COMPARISON The t test results revealed that the hybrid QWERTY speller performed significantly better (on the basis of mean classification accuracy and ITR) as compared to the traditional P300 speller) and the QWERTY SSVEP speller. Also, the amount of time taken to spell a word was significantly lesser in the case of hybrid QWERTY speller in contrast to traditional P300 speller while it was almost the same as compared to QWERTY SSVEP speller. CONCLUSION QWERTY speller outperformed the stereotypical P300 speller as well as QWERTY SSVEP speller.
Collapse
Affiliation(s)
- Er Akshay Katyal
- ICE Department, Dr B.R. Ambedkar N.I.T. Jalandhar, GT Road Bye-Pass, Jalandhar, Punjab, 144011, India.
| | - Rajesh Singla
- ICE Department, Dr B.R. Ambedkar N.I.T. Jalandhar, GT Road Bye-Pass, Jalandhar, Punjab, 144011, India
| |
Collapse
|
8
|
Katyal A, Singla R. Synchronized Detection of Evoked Potentials to Drive a High Information Transfer Rate Hybrid Brain-Computer Interface Application. ADVANCED BIOMEDICAL ENGINEERING 2021. [DOI: 10.14326/abe.10.58] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Affiliation(s)
- Akshay Katyal
- Department of Instrumentation and Control Engineering, Dr BR Ambedkar National Institute of Technology Jalandhar
| | - Rajesh Singla
- Department of Instrumentation and Control Engineering, Dr BR Ambedkar National Institute of Technology Jalandhar
| |
Collapse
|
9
|
Xu M, Han J, Wang Y, Jung TP, Ming D. Implementing Over 100 Command Codes for a High-Speed Hybrid Brain-Computer Interface Using Concurrent P300 and SSVEP Features. IEEE Trans Biomed Eng 2020; 67:3073-3082. [DOI: 10.1109/tbme.2020.2975614] [Citation(s) in RCA: 62] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
10
|
Mouli S, Palaniappan R. DIY hybrid SSVEP-P300 LED stimuli for BCI platform using EMOTIV EEG headset. HARDWAREX 2020; 8:e00113. [PMID: 35498243 PMCID: PMC9041272 DOI: 10.1016/j.ohx.2020.e00113] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/26/2019] [Revised: 05/19/2020] [Accepted: 05/20/2020] [Indexed: 06/14/2023]
Abstract
A fully customisable chip-on board (COB) LED design to evoke two brain responses simultaneously (steady state visual evoked potential (SSVEP) and transient evoked potential, P300) is discussed in this paper. Considering different possible modalities in brain-computer interfacing (BCI), SSVEP is widely accepted as it requires a lesser number of electroencephalogram (EEG) electrodes and minimal training time. The aim of this work was to produce a hybrid BCI hardware platform to evoke SSVEP and P300 precisely with reduced fatigue and improved classification performance. The system comprises of four independent radial green visual stimuli controlled individually by a 32-bit microcontroller platform to evoke SSVEP and four red LEDs flashing at random intervals to generate P300 events. The system can also record the P300 event timestamps that can be used in classification, to improve the accuracy and reliability. The hybrid stimulus was tested for real-time classification accuracy by controlling a LEGO robot to move in four directions.
Collapse
Affiliation(s)
- Surej Mouli
- Data Science Research Group, School of Computing, University of Kent
| | | |
Collapse
|
11
|
Shao X, Lin M. Filter bank temporally local canonical correlation analysis for short time window SSVEPs classification. Cogn Neurodyn 2020; 14:689-696. [PMID: 33014181 PMCID: PMC7501359 DOI: 10.1007/s11571-020-09620-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2020] [Revised: 06/21/2020] [Accepted: 07/19/2020] [Indexed: 11/26/2022] Open
Abstract
Canonical correlation analysis (CCA) method and its extended methods have been widely and successfully applied to the frequency recognition in SSVEP-based BCI systems. As a state-of-the-art extended method, filter bank canonical correlation analysis has higher accuracy and information transmission rate (ITR) than CCA. However, in the CCA method, the temporally local structure of samples has not been well considered. In this correspondence, we proposed termed temporally local canonical correlation analysis (TCCA). In this new method, the original covariance matrix was replaced by the temporally local covariance matrix. Furthermore, we proposed an improved frequency identification method of filter bank based on TCCA, named filter bank temporally local canonical correlation analysis (FBTCCA). In the offline environment, we used a leave-one-subject-out validation strategy on datasets of ten testees to optimize the parameters of TCCA and FBTCCA and evaluate the two algorithms. The experimental results affirm that TCCA markedly outperformed CCA, and FBTCCA obtained the highest accuracy among the four methods. This study corroborates that TCCA-based approaches have great potential for implementing short time window SSVEP-based BCI systems.
Collapse
Affiliation(s)
- Xinghan Shao
- School of Mechanical Engineering, Shandong University, Jinan, 250000 China
| | - Mingxing Lin
- School of Mechanical Engineering, Shandong University, Jinan, 250000 China
| |
Collapse
|
12
|
Liang L, Lin J, Yang C, Wang Y, Chen X, Gao S, Gao X. Optimizing a dual-frequency and phase modulation method for SSVEP-based BCIs. J Neural Eng 2020; 17:046026. [PMID: 32726763 DOI: 10.1088/1741-2552/abaa9b] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE The design of the stimulation paradigm plays an important role in steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) studies. Among various stimulation designs, the dual-frequency paradigm in which two frequencies are used to encode one target is of importance and interest. However, because the number of possible frequency combinations is huge, the existing dual-frequency modulation paradigms failed to optimize the encoding towards the best combinations. Thus, this work aiming at designing a new dual-frequency and phase modulation paradigm with the best combinations stimuli. APPROACH This study proposed a dual-frequency and phase modulation method, which can achieve a large number of targets by making different combinations of two frequencies and an initial phase. This study also designed a set of methods for quickly optimizing the stimulation codes for the dual-frequency and phase modulation method. MAIN RESULTS An online 40-class BCI experiment with 12 subjects obtained an accuracy of 96.06[Formula: see text]4.00% and an averaged information transfer rate (ITR) of 196.09[Formula: see text]15.25 bits min-1, which were much higher than the existing dual-frequency modulation paradigms. Moreover, an offline simulation with a public dataset showed that the optimization method was also effective for optimizing the single-frequency and phase modulation paradigm. SIGNIFICANCE These results demonstrate the high performance of the proposed dual-frequency and phase modulation method and the high efficiency of the optimization method for designing SSVEP stimulation paradigms. In addition, the coding efficiency of the optimized dual-frequency and phase modulation paradigm is higher than that of the single-frequency and phase modulation paradigm, and it is expected to further realize the BCI paradigm with a large amount of targets.
Collapse
Affiliation(s)
- Liyan Liang
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, People's Republic of China
| | | | | | | | | | | | | |
Collapse
|
13
|
Rashid M, Sulaiman N, P P Abdul Majeed A, Musa RM, Ab Nasir AF, Bari BS, Khatun S. Current Status, Challenges, and Possible Solutions of EEG-Based Brain-Computer Interface: A Comprehensive Review. Front Neurorobot 2020; 14:25. [PMID: 32581758 PMCID: PMC7283463 DOI: 10.3389/fnbot.2020.00025] [Citation(s) in RCA: 99] [Impact Index Per Article: 24.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2019] [Accepted: 04/08/2020] [Indexed: 12/12/2022] Open
Abstract
Brain-Computer Interface (BCI), in essence, aims at controlling different assistive devices through the utilization of brain waves. It is worth noting that the application of BCI is not limited to medical applications, and hence, the research in this field has gained due attention. Moreover, the significant number of related publications over the past two decades further indicates the consistent improvements and breakthroughs that have been made in this particular field. Nonetheless, it is also worth mentioning that with these improvements, new challenges are constantly discovered. This article provides a comprehensive review of the state-of-the-art of a complete BCI system. First, a brief overview of electroencephalogram (EEG)-based BCI systems is given. Secondly, a considerable number of popular BCI applications are reviewed in terms of electrophysiological control signals, feature extraction, classification algorithms, and performance evaluation metrics. Finally, the challenges to the recent BCI systems are discussed, and possible solutions to mitigate the issues are recommended.
Collapse
Affiliation(s)
- Mamunur Rashid
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Norizam Sulaiman
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Anwar P P Abdul Majeed
- Innovative Manufacturing, Mechatronics and Sports Laboratory, Faculty of Manufacturing and Mechatronic Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Rabiu Muazu Musa
- Centre for Fundamental and Continuing Education, Universiti Malaysia Terengganu, Kuala Nerus, Malaysia
| | - Ahmad Fakhri Ab Nasir
- Innovative Manufacturing, Mechatronics and Sports Laboratory, Faculty of Manufacturing and Mechatronic Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Bifta Sama Bari
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Sabira Khatun
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| |
Collapse
|
14
|
Katyal A, Singla R. A novel hybrid paradigm based on steady state visually evoked potential & P300 to enhance information transfer rate. Biomed Signal Process Control 2020. [DOI: 10.1016/j.bspc.2020.101884] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
|
15
|
|
16
|
Jalilpour S, Hajipour Sardouie S, Mijani A. A novel hybrid BCI speller based on RSVP and SSVEP paradigm. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2020; 187:105326. [PMID: 31980276 DOI: 10.1016/j.cmpb.2020.105326] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/24/2019] [Revised: 11/30/2019] [Accepted: 01/08/2020] [Indexed: 06/10/2023]
Abstract
BACKGROUND AND OBJECTIVE Steady-state visual evoked potential (SSVEP) and rapid serial visual presentation (RSVP) are useful methods in the brain-computer interface (BCI) systems. Hybrid BCI systems that combine these two approaches can enhance the proficiency of the P300 spellers. METHODS In this study, a new hybrid RSVP/SSVEP BCI is proposed to increase the classification accuracy and information transfer rate (ITR) as compared with the other RSVP speller paradigms. In this paradigm, RSVP (eliciting a P300 response) and SSVEP stimulations are presented in such a way that the target group of characters is identified by RSVP stimuli, and the target character is recognized by SSVEP stimuli. RESULTS The proposed paradigm achieved accuracy of 93.06%, and ITR of 23.41 bit/min averaged across six subjects. CONCLUSIONS The new hybrid system demonstrates that by using SSVEP stimulation in Triple RSVP speller paradigm, we could enhance the performance of the system as compared with the traditional Triple RSVP paradigm. Our work is the first hybrid paradigm in RSVP spellers that could obtain the higher classification accuracy and information transfer rate in comparison with the previous RSVP spellers.
Collapse
Affiliation(s)
- Shayan Jalilpour
- Human-Machine Interfaces Laboratory (HMIL), Sharif University of Technology, Tehran, Iran
| | | | - Amirmohammad Mijani
- Human-Machine Interfaces Laboratory (HMIL), Sharif University of Technology, Tehran, Iran
| |
Collapse
|
17
|
Character encoding based on occurrence probability enhances the performance of SSVEP-based BCI spellers. Biomed Signal Process Control 2020. [DOI: 10.1016/j.bspc.2020.101888] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
|
18
|
Mannan MMN, Kamran MA, Kang S, Choi HS, Jeong MY. A Hybrid Speller Design Using Eye Tracking and SSVEP Brain-Computer Interface. SENSORS 2020; 20:s20030891. [PMID: 32046131 PMCID: PMC7039291 DOI: 10.3390/s20030891] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/31/2019] [Revised: 01/27/2020] [Accepted: 02/05/2020] [Indexed: 12/14/2022]
Abstract
Steady-state visual evoked potentials (SSVEPs) have been extensively utilized to develop brain-computer interfaces (BCIs) due to the advantages of robustness, large number of commands, high classification accuracies, and information transfer rates (ITRs). However, the use of several simultaneous flickering stimuli often causes high levels of user discomfort, tiredness, annoyingness, and fatigue. Here we propose to design a stimuli-responsive hybrid speller by using electroencephalography (EEG) and video-based eye-tracking to increase user comfortability levels when presented with large numbers of simultaneously flickering stimuli. Interestingly, a canonical correlation analysis (CCA)-based framework was useful to identify target frequency with a 1 s duration of flickering signal. Our proposed BCI-speller uses only six frequencies to classify forty-eight targets, thus achieve greatly increased ITR, whereas basic SSVEP BCI-spellers use an equal number of frequencies to the number of targets. Using this speller, we obtained an average classification accuracy of 90.35 ± 3.597% with an average ITR of 184.06 ± 12.761 bits per minute in a cued-spelling task and an ITR of 190.73 ± 17.849 bits per minute in a free-spelling task. Consequently, our proposed speller is superior to the other spellers in terms of targets classified, classification accuracy, and ITR, while producing less fatigue, annoyingness, tiredness and discomfort. Together, our proposed hybrid eye tracking and SSVEP BCI-based system will ultimately enable a truly high-speed communication channel.
Collapse
Affiliation(s)
- Malik M. Naeem Mannan
- Department of Cogno-Mechatronics Engineering, Pusan National University, 2 Busandaehak-ro, 63 Beon-gil, Geumjeong-gu, Busan 609-735, Korea; (M.M.N.M.); (M.A.K.); (H.S.C.)
| | - M. Ahmad Kamran
- Department of Cogno-Mechatronics Engineering, Pusan National University, 2 Busandaehak-ro, 63 Beon-gil, Geumjeong-gu, Busan 609-735, Korea; (M.M.N.M.); (M.A.K.); (H.S.C.)
| | - Shinil Kang
- National Center for Optically-Assisted Ultrahigh-Precision Mechanical Systems, Yonsei University, Seoul 03722, Korea;
- School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea
| | - Hak Soo Choi
- Department of Cogno-Mechatronics Engineering, Pusan National University, 2 Busandaehak-ro, 63 Beon-gil, Geumjeong-gu, Busan 609-735, Korea; (M.M.N.M.); (M.A.K.); (H.S.C.)
- Division of Hematology/Oncology, Department of Medicine, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, MA 02115, USA
| | - Myung Yung Jeong
- Department of Cogno-Mechatronics Engineering, Pusan National University, 2 Busandaehak-ro, 63 Beon-gil, Geumjeong-gu, Busan 609-735, Korea; (M.M.N.M.); (M.A.K.); (H.S.C.)
- Correspondence:
| |
Collapse
|
19
|
de Neeling M, Van Hulle MM. Single-paradigm and hybrid brain computing interfaces and their use by disabled patients. J Neural Eng 2019; 16:061001. [DOI: 10.1088/1741-2552/ab2706] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|
20
|
Sadeghi S, Maleki A. Accurate estimation of information transfer rate based on symbol occurrence probability in brain-computer interfaces. Biomed Signal Process Control 2019. [DOI: 10.1016/j.bspc.2019.101607] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
21
|
Sadeghi S, Maleki A. Recent Advances in Hybrid Brain-Computer Interface Systems: A Technological and Quantitative Review. Basic Clin Neurosci 2019; 9:373-388. [PMID: 30719252 PMCID: PMC6360492 DOI: 10.32598/bcn.9.5.373] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2017] [Revised: 07/10/2017] [Accepted: 05/29/2018] [Indexed: 12/03/2022] Open
Abstract
Brain-Computer Interface (BCI) is a system that enables users to transmit commands to the computer using their brain activity recorded by electroencephalography. In a Hybrid Brain-Computer Interface (HBCI), a BCI control signal combines with one or more BCI control signals or with Human-Machine Interface (HMI) biosignals to increase classification accuracy, boost system speed, and improve user’s satisfaction. HBCI systems are categorized according to the type of combined signals and the combination technique (simultaneous or sequential). They have been used in several applications such as cursor control, target selection, and spellers. Increasing the number of articles published in this field indicates the significance of these systems. In this paper, different HBCI combinations, their important features, and potential applications are discussed. In most cases, the combination of a BCI control signal with a HMI biosignal yields higher information transfer rate than two BCI control signals.
Collapse
Affiliation(s)
- Sahar Sadeghi
- Department of Biomedical Engineering, Faculty of New Sciences and Technologies, Semnan University, Semnan, Iran
| | - Ali Maleki
- Department of Biomedical Engineering, Faculty of New Sciences and Technologies, Semnan University, Semnan, Iran
| |
Collapse
|
22
|
Khatri TK, Farooq H, Alam MT, Khalid MN, Rasheed K. Emergency Feedback System Based on SSVEP Brain Computing Interface. COMMUNICATIONS IN COMPUTER AND INFORMATION SCIENCE 2019:668-678. [DOI: 10.1007/978-981-13-6052-7_57] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/01/2023]
|
23
|
Keihani A, Shirzhiyan Z, Farahi M, Shamsi E, Mahnam A, Makkiabadi B, Haidari MR, Jafari AH. Use of Sine Shaped High-Frequency Rhythmic Visual Stimuli Patterns for SSVEP Response Analysis and Fatigue Rate Evaluation in Normal Subjects. Front Hum Neurosci 2018; 12:201. [PMID: 29892219 PMCID: PMC5985331 DOI: 10.3389/fnhum.2018.00201] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2017] [Accepted: 04/30/2018] [Indexed: 11/13/2022] Open
Abstract
Background: Recent EEG-SSVEP signal based BCI studies have used high frequency square pulse visual stimuli to reduce subjective fatigue. However, the effect of total harmonic distortion (THD) has not been considered. Compared to CRT and LCD monitors, LED screen displays high-frequency wave with better refresh rate. In this study, we present high frequency sine wave simple and rhythmic patterns with low THD rate by LED to analyze SSVEP responses and evaluate subjective fatigue in normal subjects. Materials and Methods: We used patterns of 3-sequence high-frequency sine waves (25, 30, and 35 Hz) to design our visual stimuli. Nine stimuli patterns, 3 simple (repetition of each of above 3 frequencies e.g., P25-25-25) and 6 rhythmic (all of the frequencies in 6 different sequences e.g., P25-30-35) were chosen. A hardware setup with low THD rate (<0.1%) was designed to present these patterns on LED. Twenty two normal subjects (aged 23-30 (25 ± 2.1) yrs) were enrolled. Visual analog scale (VAS) was used for subjective fatigue evaluation after presentation of each stimulus pattern. PSD, CCA, and LASSO methods were employed to analyze SSVEP responses. The data including SSVEP features and fatigue rate for different visual stimuli patterns were statistically evaluated. Results: All 9 visual stimuli patterns elicited SSVEP responses. Overall, obtained accuracy rates were 88.35% for PSD and > 90% for CCA and LASSO (for TWs > 1 s). High frequency rhythmic patterns group with low THD rate showed higher accuracy rate (99.24%) than simple patterns group (98.48%). Repeated measure ANOVA showed significant difference between rhythmic pattern features (P < 0.0005). Overall, there was no significant difference between the VAS of rhythmic [3.85 ± 2.13] compared to the simple patterns group [3.96 ± 2.21], (P = 0.63). Rhythmic group had lower within group VAS variation (min = P25-30-35 [2.90 ± 2.45], max = P35-25-30 [4.81 ± 2.65]) as well as least individual pattern VAS (P25-30-35). Discussion and Conclusion: Overall, rhythmic and simple pattern groups had higher and similar accuracy rates. Rhythmic stimuli patterns showed insignificantly lower fatigue rate than simple patterns. We conclude that both rhythmic and simple visual high frequency sine wave stimuli require further research for human subject SSVEP-BCI studies.
Collapse
Affiliation(s)
- Ahmadreza Keihani
- Department of Medical Physics and Biomedical Engineering, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran.,Research Center for Biomedical Technologies and Robotics, Tehran University of Medical Sciences, Tehran, Iran
| | - Zahra Shirzhiyan
- Department of Medical Physics and Biomedical Engineering, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran.,Research Center for Biomedical Technologies and Robotics, Tehran University of Medical Sciences, Tehran, Iran
| | - Morteza Farahi
- Department of Medical Physics and Biomedical Engineering, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran.,Research Center for Biomedical Technologies and Robotics, Tehran University of Medical Sciences, Tehran, Iran
| | - Elham Shamsi
- Department of Medical Physics and Biomedical Engineering, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran.,Research Center for Biomedical Technologies and Robotics, Tehran University of Medical Sciences, Tehran, Iran
| | - Amin Mahnam
- Department of Biomedical Engineering, Faculty of Engineering, University of Isfahan, Isfahan, Iran
| | - Bahador Makkiabadi
- Department of Medical Physics and Biomedical Engineering, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran.,Research Center for Biomedical Technologies and Robotics, Tehran University of Medical Sciences, Tehran, Iran
| | - Mohsen R Haidari
- Section of Neuroscience, Department of Neurology, Faculty of Medicine, Baqiyatallah University of Medical Sciences, Tehran, Iran
| | - Amir H Jafari
- Department of Medical Physics and Biomedical Engineering, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran.,Research Center for Biomedical Technologies and Robotics, Tehran University of Medical Sciences, Tehran, Iran
| |
Collapse
|
24
|
Rezeika A, Benda M, Stawicki P, Gembler F, Saboor A, Volosyak I. Brain-Computer Interface Spellers: A Review. Brain Sci 2018; 8:brainsci8040057. [PMID: 29601538 PMCID: PMC5924393 DOI: 10.3390/brainsci8040057] [Citation(s) in RCA: 169] [Impact Index Per Article: 28.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2018] [Revised: 03/16/2018] [Accepted: 03/27/2018] [Indexed: 12/14/2022] Open
Abstract
A Brain-Computer Interface (BCI) provides a novel non-muscular communication method via brain signals. A BCI-speller can be considered as one of the first published BCI applications and has opened the gate for many advances in the field. Although many BCI-spellers have been developed during the last few decades, to our knowledge, no reviews have described the different spellers proposed and studied in this vital field. The presented speller systems are categorized according to major BCI paradigms: P300, steady-state visual evoked potential (SSVEP), and motor imagery (MI). Different BCI paradigms require specific electroencephalogram (EEG) signal features and lead to the development of appropriate Graphical User Interfaces (GUIs). The purpose of this review is to consolidate the most successful BCI-spellers published since 2010, while mentioning some other older systems which were built explicitly for spelling purposes. We aim to assist researchers and concerned individuals in the field by illustrating the highlights of different spellers and presenting them in one review. It is almost impossible to carry out an objective comparison between different spellers, as each has its variables, parameters, and conditions. However, the gathered information and the provided taxonomy about different BCI-spellers can be helpful, as it could identify suitable systems for first-hand users, as well as opportunities of development and learning from previous studies for BCI researchers.
Collapse
Affiliation(s)
- Aya Rezeika
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Mihaly Benda
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Piotr Stawicki
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Felix Gembler
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Abdul Saboor
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Ivan Volosyak
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| |
Collapse
|
25
|
Zhang X, Xu G, Xie J, Zhang X. Brain response to luminance-based and motion-based stimulation using inter-modulation frequencies. PLoS One 2017; 12:e0188073. [PMID: 29141030 PMCID: PMC5687765 DOI: 10.1371/journal.pone.0188073] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2017] [Accepted: 10/31/2017] [Indexed: 12/02/2022] Open
Abstract
Steady state visual evoked potential (SSVEP)-based brain computer interface (BCI) has advantages of high information transfer rate (ITR), less electrodes and little training. So it has been widely investigated. However, the available stimulus frequencies are limited by brain responses. Simultaneous modulation of stimulus luminance is a novel method to resolve this problem. In this study, three experiments were devised to gain a deeper understanding of the brain response to the stimulation using inter-modulation frequencies. First, luminance-based stimulation using one to five inter-modulation frequencies was analyzed for the first time. The characteristics of the brain responses to the proposed stimulation were reported. Second, the motion-based stimulation with equal luminance using inter-modulation frequencies was also proposed for the first time. The response of the brain under these conditions were similar to that of luminance-based stimulation which can induce combination frequencies. And an elementary analysis was conducted to explain the reason of the occurrence of combination frequencies. Finally, the online test demonstrated the efficacy of our proposed two stimulation methods for BCI. The average ITRs reached 34.7836 bits/min and 39.2856 bits/min for luminance-based and motion-based stimulation respectively. This study demonstrated that the simultaneous modulation of stimulus luminance could extend to at least five frequencies to induce SSVEP and the brain response to the stimulus still maintained a certain positive correlation with luminance. And not only luminance-based stimulation, but also motion-based stimulation with equal luminance can elicit inter-modulation frequencies to effectively increase the number of targets for multi-class SSVEP.
Collapse
Affiliation(s)
- Xin Zhang
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, China
| | - Guanghua Xu
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, China
- State Key Laboratory for Manufacturing Systems Engineering, Xi'an Jiaotong University, Xi'an, China
- * E-mail:
| | - Jun Xie
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, China
| | - Xun Zhang
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, China
| |
Collapse
|
26
|
Zhang Z, Huang Y, Chen S, Qu J, Pan X, Yu T, Li Y. An Intention-Driven Semi-autonomous Intelligent Robotic System for Drinking. Front Neurorobot 2017; 11:48. [PMID: 28943849 PMCID: PMC5596089 DOI: 10.3389/fnbot.2017.00048] [Citation(s) in RCA: 28] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2017] [Accepted: 08/18/2017] [Indexed: 11/13/2022] Open
Abstract
In this study, an intention-driven semi-autonomous intelligent robotic (ID-SIR) system is designed and developed to assist the severely disabled patients to live independently. The system mainly consists of a non-invasive brain–machine interface (BMI) subsystem, a robot manipulator and a visual detection and localization subsystem. Different from most of the existing systems remotely controlled by joystick, head- or eye tracking, the proposed ID-SIR system directly acquires the intention from users’ brain. Compared with the state-of-art system only working for a specific object in a fixed place, the designed ID-SIR system can grasp any desired object in a random place chosen by a user and deliver it to his/her mouth automatically. As one of the main advantages of the ID-SIR system, the patient is only required to send one intention command for one drinking task and the autonomous robot would finish the rest of specific controlling tasks, which greatly eases the burden on patients. Eight healthy subjects attended our experiment, which contained 10 tasks for each subject. In each task, the proposed ID-SIR system delivered the desired beverage container to the mouth of the subject and then put it back to the original position. The mean accuracy of the eight subjects was 97.5%, which demonstrated the effectiveness of the ID-SIR system.
Collapse
Affiliation(s)
- Zhijun Zhang
- School of Automation Science and Engineering, South China University of Technology, Guangzhou, China
| | - Yongqian Huang
- School of Automation Science and Engineering, South China University of Technology, Guangzhou, China
| | - Siyuan Chen
- School of Automation Science and Engineering, South China University of Technology, Guangzhou, China
| | - Jun Qu
- School of Automation Science and Engineering, South China University of Technology, Guangzhou, China
| | - Xin Pan
- School of Automation Science and Engineering, South China University of Technology, Guangzhou, China
| | - Tianyou Yu
- School of Automation Science and Engineering, South China University of Technology, Guangzhou, China
| | - Yuanqing Li
- School of Automation Science and Engineering, South China University of Technology, Guangzhou, China
| |
Collapse
|
27
|
Hong KS, Khan MJ. Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review. Front Neurorobot 2017; 11:35. [PMID: 28790910 PMCID: PMC5522881 DOI: 10.3389/fnbot.2017.00035] [Citation(s) in RCA: 116] [Impact Index Per Article: 16.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2017] [Accepted: 07/03/2017] [Indexed: 12/11/2022] Open
Abstract
In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.
Collapse
Affiliation(s)
- Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea.,Department of Cogno-Mechatronics Engineering, Pusan National University, Busan, South Korea
| | - Muhammad Jawad Khan
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| |
Collapse
|
28
|
Abstract
Amyotrophic lateral sclerosis (ALS) patients whose voluntary muscles are paralyzed commonly communicate with the outside world using eye movement. There have been many efforts to support this method of communication by tracking or detecting eye movement. An electrooculogram (EOG), an electro-physiological signal, is generated by eye movements and can be measured with electrodes placed around the eye. In this study, we proposed a new practical electrode position on the forehead to measure EOG signals, and we developed a wearable forehead EOG measurement system for use in Human Computer/Machine interfaces (HCIs/HMIs). Four electrodes, including the ground electrode, were placed on the forehead. The two channels were arranged vertically and horizontally, sharing a positive electrode. Additionally, a real-time eye movement classification algorithm was developed based on the characteristics of the forehead EOG. Three applications were employed to evaluate the proposed system: a virtual keyboard using a modified Bremen BCI speller and an automatic sequential row-column scanner, and a drivable power wheelchair. The mean typing speeds of the modified Bremen brain–computer interface (BCI) speller and automatic row-column scanner were 10.81 and 7.74 letters per minute, and the mean classification accuracies were 91.25% and 95.12%, respectively. In the power wheelchair demonstration, the user drove the wheelchair through an 8-shape course without collision with obstacles.
Collapse
Affiliation(s)
- Jeong Heo
- Interdisciplinary Program of Bioengineering, Seoul National University, Seoul 03080, Korea.
| | - Heenam Yoon
- Interdisciplinary Program of Bioengineering, Seoul National University, Seoul 03080, Korea.
| | - Kwang Suk Park
- Department of Biomedical Engineering, College of Medicine, Seoul National University, Seoul 03080, Korea.
| |
Collapse
|
29
|
Choi I, Rhiu I, Lee Y, Yun MH, Nam CS. A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives. PLoS One 2017; 12:e0176674. [PMID: 28453547 PMCID: PMC5409179 DOI: 10.1371/journal.pone.0176674] [Citation(s) in RCA: 46] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
A new Brain-Computer Interface (BCI) technique, which is called a hybrid BCI, has recently been proposed to address the limitations of conventional single BCI system. Although some hybrid BCI studies have shown promising results, the field of hybrid BCI is still in its infancy and there is much to be done. Especially, since the hybrid BCI systems are so complicated and complex, it is difficult to understand the constituent and role of a hybrid BCI system at a glance. Also, the complicated and complex systems make it difficult to evaluate the usability of the systems. We systematically reviewed and analyzed the current state-of-the-art hybrid BCI studies, and proposed a systematic taxonomy for classifying the types of hybrid BCIs with multiple taxonomic criteria. After reviewing 74 journal articles, hybrid BCIs could be categorized with respect to 1) the source of brain signals, 2) the characteristics of the brain signal, and 3) the characteristics of operation in each system. In addition, we exhaustively reviewed recent literature on usability of BCIs. To identify the key evaluation dimensions of usability, we focused on task and measurement characteristics of BCI usability. We classified and summarized 31 BCI usability journal articles according to task characteristics (type and description of task) and measurement characteristics (subjective and objective measures). Afterwards, we proposed usability dimensions for BCI and hybrid BCI systems according to three core-constructs: Satisfaction, effectiveness, and efficiency with recommendations for further research. This paper can help BCI researchers, even those who are new to the field, can easily understand the complex structure of the hybrid systems at a glance. Recommendations for future research can also be helpful in establishing research directions and gaining insight in how to solve ergonomics and HCI design issues surrounding BCI and hybrid BCI systems by usability evaluation.
Collapse
Affiliation(s)
- Inchul Choi
- Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, North Carolina, United States of America
| | - Ilsun Rhiu
- Division of Global Management Engineering, Hoseo University, Asan, Korea
| | - Yushin Lee
- Department of Industrial Engineering, Seoul National University, Seoul, Korea
| | - Myung Hwan Yun
- Department of Industrial Engineering, Seoul National University, Seoul, Korea
| | - Chang S. Nam
- Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, North Carolina, United States of America
- * E-mail:
| |
Collapse
|
30
|
Zhang Y, Guo D, Xu P, Zhang Y, Yao D. Robust frequency recognition for SSVEP-based BCI with temporally local multivariate synchronization index. Cogn Neurodyn 2016; 10:505-511. [PMID: 27891199 PMCID: PMC5106453 DOI: 10.1007/s11571-016-9398-9] [Citation(s) in RCA: 41] [Impact Index Per Article: 5.1] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2016] [Revised: 06/20/2016] [Accepted: 07/13/2016] [Indexed: 01/12/2023] Open
Abstract
Multivariate synchronization index (MSI) has been proved to be an efficient method for frequency recognition in SSVEP-BCI systems. It measures the correlation according to the entropy of the normalized eigenvalues of the covariance matrix of multichannel signals. In the MSI method, the estimation of covariance matrix omits the temporally local structure of samples. In this study, a new spatio-temporal method, termed temporally local MSI (TMSI), was presented. This new method explicitly exploits temporally local information in modelling the covariance matrix. In order to evaluate the performance of the TMSI, we performs a comparison between the two methods on the real SSVEP datasets from eleven subjects. The results show that the TMSI outperforms the standard MSI. TMSI benefits from exploiting the temporally local structure of EEG signals, and could be a potential method for robust performance of SSVEP-based BCI.
Collapse
Affiliation(s)
- Yangsong Zhang
- School of Computer Science and Technology, Southwest University of Science and Technology, Mianyang, 621010 China
| | - Daqing Guo
- Key Laboratory for NeuroInformation of Ministry of Education, Center for Information in Medicine, University of Electronic Science and Technology of China, Chengdu, 610054 China
| | - Peng Xu
- Key Laboratory for NeuroInformation of Ministry of Education, Center for Information in Medicine, University of Electronic Science and Technology of China, Chengdu, 610054 China
| | - Yu Zhang
- Key Laboratory for Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai, 200237 China
| | - Dezhong Yao
- Key Laboratory for NeuroInformation of Ministry of Education, Center for Information in Medicine, University of Electronic Science and Technology of China, Chengdu, 610054 China
| |
Collapse
|
31
|
A Prototype SSVEP Based Real Time BCI Gaming System. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2016; 2016:3861425. [PMID: 27051414 PMCID: PMC4804071 DOI: 10.1155/2016/3861425] [Citation(s) in RCA: 33] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/15/2015] [Revised: 01/06/2016] [Accepted: 01/10/2016] [Indexed: 11/17/2022]
Abstract
Although brain-computer interface technology is mainly designed with disabled people in mind, it can also be beneficial to healthy subjects, for example, in gaming or virtual reality systems. In this paper we discuss the typical architecture, paradigms, requirements, and limitations of electroencephalogram-based gaming systems. We have developed a prototype three-class brain-computer interface system, based on the steady state visually evoked potentials paradigm and the Emotiv EPOC headset. An online target shooting game, implemented in the OpenViBE environment, has been used for user feedback. The system utilizes wave atom transform for feature extraction, achieving an average accuracy of 78.2% using linear discriminant analysis classifier, 79.3% using support vector machine classifier with a linear kernel, and 80.5% using a support vector machine classifier with a radial basis function kernel.
Collapse
|