A practical, intuitive brain-computer interface for communicating 'yes' or 'no' by listening

The challenge of controlling an auditory BCI in the case of severe motor disability

BACKGROUND

The locked-in syndrome (LIS), due to a lesion in the pons, impedes communication. This situation can also be met after some severe brain injury or in advanced Amyotrophic Lateral Sclerosis (ALS). In the most severe condition, the persons cannot communicate at all because of a complete oculomotor paralysis (Complete LIS or CLIS). This even prevents the detection of consciousness. Some studies suggest that auditory brain-computer interface (BCI) could restore a communication through a « yes-no» code.

METHODS

We developed an auditory EEG-based interface which makes use of voluntary modulations of attention, to restore a yes-no communication code in non-responding persons. This binary BCI uses repeated speech sounds (alternating "yes" on the right ear and "no" on the left ear) corresponding to either frequent (short) or rare (long) stimuli. Users are instructed to pay attention to the relevant stimuli only. We tested this BCI with 18 healthy subjects, and 7 people with severe motor disability (3 "classical" persons with locked-in syndrome and 4 persons with ALS).

RESULTS

We report online BCI performance and offline event-related potential analysis. On average in healthy subjects, online BCI accuracy reached 86% based on 50 questions. Only one out of 18 subjects could not perform above chance level. Ten subjects had an accuracy above 90%. However, most patients could not produce online performance above chance level, except for two people with ALS who obtained 100% accuracy. We report individual event-related potentials and their modulation by attention. In addition to the classical P3b, we observed a signature of sustained attention on responses to frequent sounds, but in healthy subjects and patients with good BCI control only.

CONCLUSIONS

Auditory BCI can be very well controlled by healthy subjects, but it is not a guarantee that it can be readily used by the target population of persons in LIS or CLIS. A conclusion that is supported by a few previous findings in BCI and should now trigger research to assess the reasons of such a gap in order to propose new and efficient solutions.

CLINICAL TRIAL REGISTRATIONS

No. NCT02567201 (2015) and NCT03233282 (2013).

Target of selective auditory attention can be robustly followed with MEG

Selective auditory attention enables filtering of relevant acoustic information from irrelevant. Specific auditory responses, measurable by magneto- and electroencephalography (MEG/EEG), are known to be modulated by attention to the evoking stimuli. However, such attention effects have typically been studied in unnatural conditions (e.g. during dichotic listening of pure tones) and have been demonstrated mostly in averaged auditory evoked responses. To test how reliably we can detect the attention target from unaveraged brain responses, we recorded MEG data from 15 healthy subjects that were presented with two human speakers uttering continuously the words "Yes" and "No" in an interleaved manner. The subjects were asked to attend to one speaker. To investigate which temporal and spatial aspects of the responses carry the most information about the target of auditory attention, we performed spatially and temporally resolved classification of the unaveraged MEG responses using a support vector machine. Sensor-level decoding of the responses to attended vs. unattended words resulted in a mean accuracy of [Formula: see text] (N = 14) for both stimulus words. The discriminating information was mostly available 200-400 ms after the stimulus onset. Spatially-resolved source-level decoding indicated that the most informative sources were in the auditory cortices, in both the left and right hemisphere. Our result corroborates attention modulation of auditory evoked responses and shows that such modulations are detectable in unaveraged MEG responses at high accuracy, which could be exploited e.g. in an intuitive brain-computer interface.

Simultaneous multiple-stimulus auditory brain-computer interface with semi-supervised learning and prior probability distribution tuning

Objective.Auditory brain-computer interfaces (BCIs) enable users to select commands based on the brain activity elicited by auditory stimuli. However, existing auditory BCI paradigms cannot increase the number of available commands without decreasing the selection speed, because each stimulus needs to be presented independently and sequentially under the standard oddball paradigm. To solve this problem, we propose a double-stimulus paradigm that simultaneously presents multiple auditory stimuli.Approach.For addition to an existing auditory BCI paradigm, the best discriminable sound was chosen following a subjective assessment. The new sound was located on the right-hand side and presented simultaneously with an existing sound from the left-hand side. A total of six sounds were used for implementing the auditory BCI with a 6 × 6 letter matrix. We employ semi-supervised learning (SSL) and prior probability distribution tuning to improve the accuracy of the paradigm. The SSL method involved updating of the classifier weights, and their prior probability distributions were adjusted using the following three types of distributions: uniform, empirical, and extended empirical (e-empirical). The performance was evaluated based on the BCI accuracy and information transfer rate (ITR).Main results.The double-stimulus paradigm resulted in a BCI accuracy of 67.89 ± 11.46% and an ITR of 2.67 ± 1.09 bits min^-1, in the absence of SSL and with uniform distribution. The proposed combination of SSL with e-empirical distribution improved the BCI accuracy and ITR to 74.59 ± 12.12% and 3.37 ± 1.27 bits min^-1, respectively. The event-related potential analysis revealed that contralateral and right-hemispheric dominances contributed to the BCI performance improvement.Significance.Our study demonstrated that a BCI based on multiple simultaneous auditory stimuli, incorporating SSL and e-empirical prior distribution, can increase the number of commands without sacrificing typing speed beyond the acceptable level of accuracy.

A systematic review of research on augmentative and alternative communication brain-computer interface systems for individuals with disabilities

Augmentative and alternative communication brain-computer interface (AAC-BCI) systems are intended to offer communication access to people with severe speech and physical impairment (SSPI) without requiring volitional movement. As the field moves toward clinical implementation of AAC-BCI systems, research involving participants with SSPI is essential. Research has demonstrated variability in AAC-BCI system performance across users, and mixed results for comparisons of performance for users with and without disabilities. The aims of this systematic review were to (1) describe study, system, and participant characteristics reported in BCI research, (2) summarize the communication task performance of participants with disabilities using AAC-BCI systems, and (3) explore any differences in performance for participants with and without disabilities. Electronic databases were searched in May, 2018, and March, 2021, identifying 6065 records, of which 73 met inclusion criteria. Non-experimental study designs were common and sample sizes were typically small, with approximately half of studies involving five or fewer participants with disabilities. There was considerable variability in participant characteristics, and in how those characteristics were reported. Over 60% of studies reported an average selection accuracy ≤70% for participants with disabilities in at least one tested condition. However, some studies excluded participants who did not reach a specific system performance criterion, and others did not state whether any participants were excluded based on performance. Twenty-nine studies included participants both with and without disabilities, but few reported statistical analyses comparing performance between the two groups. Results suggest that AAC-BCI systems show promise for supporting communication for people with SSPI, but they remain ineffective for some individuals. The lack of standards in reporting outcome measures makes it difficult to synthesize data across studies. Further research is needed to demonstrate efficacy of AAC-BCI systems for people who experience SSPI of varying etiologies and severity levels, and these individuals should be included in system design and testing. Consensus in terminology and consistent participant, protocol, and performance description will facilitate the exploration of user and system characteristics that positively or negatively affect AAC-BCI use, and support innovations that will make this technology more useful to a broader group of people.

Clinical trial registration

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42018095345, PROSPERO: CRD42018095345.

Visuo-auditory stimuli with semantic, temporal and spatial congruence for a P300-based BCI: An exploratory test with an ALS patient in a completely locked-in state

BACKGROUND

Brain-computer interfaces (BCIs) are a promising tool for communication with completely locked-in state (CLIS) patients. Despite the great efforts already made by the BCI research community, the cases of success are still very few, very exploratory, limited in time, and based on simple 'yes/no' paradigms.

NEW METHOD

A P300-based BCI is proposed comparing two conditions, one corresponding to purely spatial auditory stimuli (AU-S) and the other corresponding to hybrid visual and spatial auditory stimuli (HVA-S). In the HVA-S condition, there is a semantic, temporal, and spatial congruence between visual and auditory stimuli. The stimuli comprise a lexicon of 7 written and spoken words. Spatial sounds are generated through the head-related transfer function. Given the good results obtained with 10 able-bodied participants, we investigated whether a patient entering CLIS could use the proposed BCI.

RESULTS

The able-bodied group achieved 71.3 % and 90.5 % online classification accuracy for the auditory and hybrid BCIs respectively, while the patient achieved 30 % and chance level accuracies, for the same conditions. Notwithstanding, the patient's event-related potentials (ERPs) showed statistical discrimination between target and non-target events in different time windows.

COMPARISON WITH EXISTING METHODS

The results of the control group compare favorably with the state-of-the-art, considering a 7-class BCI controlled visual-covertly and with auditory stimuli. The integration of visual and auditory stimuli has not been tested before with CLIS patients.

CONCLUSIONS

The semantic, temporal, and spatial congruence of the stimuli increased the performance of the control group, but not of the CLIS patient, which can be due to impaired attention and cognitive function. The patient's unique ERP patterns make interpretation difficult, requiring further tests/paradigms to decouple patients' responses at different levels (reflexive, perceptual, cognitive). The ERPs discrimination found indicates that a simplification of the proposed approaches may be feasible.

Neural decoding of semantic concepts: A systematic literature review

Objective Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Results Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

Poststroke Cognitive Impairment Research Progress on Application of Brain-Computer Interface

Brain-computer interfaces (BCIs), a new type of rehabilitation technology, pick up nerve cell signals, identify and classify their activities, and convert them into computer-recognized instructions. This technique has been widely used in the rehabilitation of stroke patients in recent years and appears to promote motor function recovery after stroke. At present, the application of BCI in poststroke cognitive impairment is increasing, which is a common complication that also affects the rehabilitation process. This paper reviews the promise and potential drawbacks of using BCI to treat poststroke cognitive impairment, providing a solid theoretical basis for the application of BCI in this area.

Spatial localization in target detection based on decoding N2pc component

BACKGROUND

The Gaze-independent BCI system is used to restore communication in patients with eye movement disorders. One available control mechanism is the utilization of spatial attention. However, spatial information is mostly used to simply answer the "True/False" target recognition question and is seldom used to improve the efficiency of target detection. Therefore, it is necessary to utilize the potential advantages of spatial attention to improving the target detection efficiency.

NEW METHOD

We found that N2pc could be used to assess spatial attention shift and determine target position. It was a negative wave in the posterior brain on the contralateral target stimulus. From this, we designed a novel spatial coding paradigm to achieve two main purposes at each stimulus presentation: target recognition and spatial localization.

COMPARISON WITH EXISTING METHODS

We used a two-step classification framework to decode the P300 and N2pc components.

RESULTS

The average decoding accuracy of fourteen subjects was 84.43% (σ = 1.14%), and the classification accuracy of six subjects was more than 85%. The information transfer rate of the spatial coding paradigm could reach 60.52 bits/min. Compared with the single stimulus paradigm, the target detection efficiency was successfully improved by approximately 10%.

CONCLUSIONS

The spatial coding paradigm proposed in this paper answered both "True/False" and "Left/Right" questions by decoding spatial attention information. This method could significantly improve image detection efficiencies, such as visual search tasks, Internet image screening, or military target determination.

Decoding of semantic categories of imagined concepts of animals and tools in fNIRS

Objective.Semantic decoding refers to the identification of semantic concepts from recordings of an individual's brain activity. It has been previously reported in functional magnetic resonance imaging and electroencephalography. We investigate whether semantic decoding is possible with functional near-infrared spectroscopy (fNIRS). Specifically, we attempt to differentiate between the semantic categories of animals and tools. We also identify suitable mental tasks for potential brain-computer interface (BCI) applications.Approach.We explore the feasibility of a silent naming task, for the first time in fNIRS, and propose three novel intuitive mental tasks based on imagining concepts using three sensory modalities: visual, auditory, and tactile. Participants are asked to visualize an object in their minds, imagine the sounds made by the object, and imagine the feeling of touching the object. A general linear model is used to extract hemodynamic responses that are then classified via logistic regression in a univariate and multivariate manner.Main results.We successfully classify all tasks with mean accuracies of 76.2% for the silent naming task, 80.9% for the visual imagery task, 72.8% for the auditory imagery task, and 70.4% for the tactile imagery task. Furthermore, we show that consistent neural representations of semantic categories exist by applying classifiers across tasks.Significance.These findings show that semantic decoding is possible in fNIRS. The study is the first step toward the use of semantic decoding for intuitive BCI applications for communication.

Speech stream segregation to control an ERP-based auditory BCI

OBJECTIVE

The use of natural sounds in auditory Brain-Computer Interfaces (BCI) has been shown to improve classification results and usability. Some auditory BCIs are based on stream segregation, in which the subjects must attend one audio stream and ignore the other(s); these streams include some kind of stimuli to be detected. In this work we focus on Event-Related Potentials (ERP) and study whether providing intelligible content to each audio stream could help the users to better concentrate on the desired stream and so to better attend the target stimuli and to ignore the non-target ones.

APPROACH

In addition to a control condition, two experimental conditions, based on the selective attention and the cocktail party effect, were tested using two simultaneous and spatialized audio streams: i) the condition A2 consisted of an overlap of auditory stimuli (single syllables) on a background consisting of natural speech for each stream, ii) in condition A3, brief alterations of the natural flow of each speech were used as stimuli.

MAIN RESULTS

The two experimental proposals improved the results of the control condition (single words as stimuli without a speech background) both in a cross validation analysis of the calibration part and in the online test. The analysis of the ERP responses also presented better discriminability for the two proposals in comparison to the control condition. The results of subjective questionnaires support the better usability of the first experimental condition.

SIGNIFICANCE

The use of natural speech as background improves the stream segregation in an ERP-based auditory BCI (with significant results in the performance metrics, the ERP waveforms, and in the preference parameter in subjective questionnaires). Future work in the field of ERP-based stream segregation should study the use of natural speech in combination with easily perceived but not distracting stimuli.

Impact of Stimulus Features on the Performance of a Gaze-Independent Brain-Computer Interface Based on Covert Spatial Attention Shifts

Regaining communication abilities in patients who are unable to speak or move is one of the main goals in decoding brain waves for brain-computer interface (BCI) control. Many BCI approaches designed for communication rely on attention to visual stimuli, commonly applying an oddball paradigm, and require both eye movements and adequate visual acuity. These abilities may, however, be absent in patients who depend on BCI communication. We have therefore developed a response-based communication BCI, which is independent of gaze shifts but utilizes covert shifts of attention to the left or right visual field. We recorded the electroencephalogram (EEG) from 29 channels and coregistered the vertical and horizontal electrooculogram. Data-driven decoding of small attention-based differences between the hemispheres, also known as N2pc, was performed using 14 posterior channels, which are expected to reflect correlates of visual spatial attention. Eighteen healthy participants responded to 120 statements by covertly directing attention to one of two colored symbols (green and red crosses for "yes" and "no," respectively), presented in the user's left and right visual field, respectively, while maintaining central gaze fixation. On average across participants, 88.5% (std: 7.8%) of responses were correctly decoded online. In order to investigate the potential influence of stimulus features on accuracy, we presented the symbols with different visual angles, by altering symbol size and eccentricity. The offline analysis revealed that stimulus features have a minimal impact on the controllability of the BCI. Hence, we show with our novel approach that spatial attention to a colored symbol is a robust method with which to control a BCI, which has the potential to support severely paralyzed people with impaired eye movements and low visual acuity in communicating with their environment.

Decoding the covert shift of spatial attention from electroencephalographic signals permits reliable control of a brain-computer interface

OBJECTIVE

One of the main goals of brain-computer interfaces (BCI) is to restore communication abilities in patients. BCIs often use event-related potentials (ERPs) like the P300 which signals the presence of a target in a stream of stimuli. The P300 and related approaches, however, are inherently limited, as they require many stimulus presentations to obtain a usable control signal. Many approaches depend on gaze direction to focus the target, which is also not a viable approach in many cases, because eye movements might be impaired in potential users. Here we report on a BCI that avoids both shortcomings by decoding spatial target information, independent of gaze shifts.

APPROACH

We present a new method to decode from the electroencephalogram (EEG) covert shifts of attention to one out of four targets simultaneously presented in the left and right visual field. The task is designed to evoke the N2pc component-a hemisphere lateralized response, elicited over the occipital scalp contralateral to the attended target. The decoding approach involves decoding of the N2pc based on data-driven estimation of spatial filters and a correlation measure.

MAIN RESULTS

Despite variability of decoding performance across subjects, 22 out of 24 subjects performed well above chance level. Six subjects even exceeded 80% (cross-validated: 89%) correct predictions in a four-class discrimination task. Hence, the single-trial N2pc proves to be a component that allows for reliable BCI control. An offline analysis of the EEG data with respect to their dependence on stimulation time and number of classes demonstrates that the present method is also a workable approach for two-class tasks.

SIGNIFICANCE

Our method extends the range of strategies for gaze-independent BCI control. The proposed decoding approach has the potential to be efficient in similar applications intended to decode ERPs.

Stimulus modality influences session-to-session transfer of training effects in auditory and tactile streaming-based P300 brain-computer interfaces

Despite recent successes, patients suffering from locked-in syndrome (LIS) still struggle to communicate using vision-independent brain–computer interfaces (BCIs). In this study, we compared auditory and tactile BCIs, regarding training effects and cross-stimulus-modality transfer effects, when switching between stimulus modalities. We utilized a streaming-based P300 BCI, which was developed as a low workload approach to prevent potential BCI-inefficiency. We randomly assigned 20 healthy participants to two groups. The participants received three sessions of training either using an auditory BCI or using a tactile BCI. In an additional fourth session, BCI versions were switched to explore possible cross-stimulus-modality transfer effects. Both BCI versions could be operated successfully in the first session by the majority of the participants, with the tactile BCI being experienced as more intuitive. Significant training effects were found mostly in the auditory BCI group and strong evidence for a cross-stimulus-modality transfer occurred for the auditory training group that switched to the tactile version but not vice versa. All participants were able to control at least one BCI version, suggesting that the investigated paradigms are generally feasible and merit further research into their applicability with LIS end-users. Individual preferences regarding stimulus modality should be considered.

Auditory Electrooculogram-based Communication System for ALS Patients in Transition from Locked-in to Complete Locked-in State

Patients in the transition from locked-in (i.e., a state of almost complete paralysis with voluntary eye movement control, eye blinks or twitches of face muscles, and preserved consciousness) to complete locked-in state (i.e., total paralysis including paralysis of eye-muscles and loss of gaze-fixation, combined with preserved consciousness) are left without any means of communication. An auditory communication system based on electrooculogram (EOG) was developed to enable such patients to communicate. Four amyotrophic lateral sclerosis patients in transition from locked-in state to completely locked-in state, with ALSFRS-R score of 0, unable to use eye trackers for communication, learned to use an auditory EOG-based communication system. The patients, with eye-movement amplitude between the range of ±200μV and ±40μV, were able to form complete sentences and communicate independently and freely, selecting letters from an auditory speller system. A follow-up of one year with one patient shows the feasibility of the proposed system in long-term use and the correlation between speller performance and eye-movement decay. The results of the auditory speller system have the potential to provide a means of communication to patient populations without gaze fixation ability and with low eye-movement amplitude range.

Covert Intention to Answer "Yes" or "No" Can Be Decoded from Single-Trial Electroencephalograms (EEGs)

Interpersonal communication is based on questions and answers, and the most useful and simplest case is the binary “yes or no” question and answer. The purpose of this study is to show that it is possible to decode intentions on “yes” or “no” answers from multichannel single-trial electroencephalograms, which were recorded while covertly answering to self-referential questions with either “yes” or “no.” The intention decoding algorithm consists of a common spatial pattern and support vector machine, which are employed for the feature extraction and pattern classification, respectively, after dividing the overall time-frequency range into subwindows of 200 ms × 2 Hz. The decoding accuracy using the information within each subwindow was investigated to find useful temporal and spectral ranges and found to be the highest for 800–1200 ms in the alpha band or 200–400 ms in the theta band. When the features from multiple subwindows were utilized together, the accuracy was significantly increased up to ∼86%. The most useful features for the “yes/no” discrimination was found to be focused in the right frontal region in the theta band and right centroparietal region in the alpha band, which may reflect the violation of autobiographic facts and higher cognitive load for “no” compared to “yes.” Our task requires the subjects to answer self-referential questions just as in interpersonal conversation without any self-regulation of the brain signals or high cognitive efforts, and the “yes” and “no” answers are decoded directly from the brain activities. This implies that the “mind reading” in a true sense is feasible. Beyond its contribution in fundamental understanding of the neural mechanism of human intention, the decoding of “yes” or “no” from brain activities may eventually lead to a natural brain-computer interface.

Behind the Scenes of Noninvasive Brain-Computer Interfaces: A Review of Electroencephalography Signals, How They Are Recorded, and Why They Matter

Purpose

Brain-computer interface (BCI) techniques may provide computer access for individuals with severe physical impairments. However, the relatively hidden nature of BCI control obscures how BCI systems work behind the scenes, making it difficult to understand how electroencephalography (EEG) records the BCI related brain signals, what brain signals are recorded by EEG, and why these signals are targeted for BCI control. Furthermore, in the field of speech-language-hearing, signals targeted for BCI application have been of primary interest to clinicians and researchers in the area of augmentative and alternative communication (AAC). However, signals utilized for BCI control reflect sensory, cognitive and motor processes, which are of interest to a range of related disciplines including speech science.

Method

This tutorial was developed by a multidisciplinary team emphasizing primary and secondary BCI-AAC related signals of interest to speech-language-hearing.

Results

An overview of BCI-AAC related signals are provided discussing 1) how BCI signals are recorded via EEG, 2) what signals are targeted for non-invasive BCI control, including the P300, sensorimotor rhythms, steady state evoked potentials, contingent negative variation, and the N400, and 3) why these signals are targeted. During tutorial creation, attention was given to help support EEG and BCI understanding for those without an engineering background.

Conclusion

Tutorials highlighting how BCI-AAC signals are elicited and recorded can help increase interest and familiarity with EEG and BCI techniques and provide a framework for understanding key principles behind BCI-AAC design and implementation.

Electrophysiological responses of relatedness to consecutive word stimuli in relation to an actively recollected target word

In this paper, we investigate the robustness of electrophysiological responses of relatedness to multiple consecutive word stimuli (probes), in relation to an actively recollected target word. Such relatedness information could be used by a Brain Computer Interface to infer the active semantic concept on a user’s mind, by integrating the knowledge of the relationship between the multiple probe words and the ‘unknown’ target. Such a BCI can take advantage of the N400: an event related potential that is sensitive to semantic content of a stimulus in relation to an established semantic context. However, it is unknown whether the N400 is suited for the multiple probing paradigm we propose, as other intervening words might distract from the established context (i.e., the target word). We perform an experiment in which we present up to ten words after an initial target word, and find no attenuation of the strength of the N400 in grand average ERPs and no decrease in classification accuracy for probes occurring later in the sequences. These results are groundwork for developing a BCI that infers the concept on a user’s mind through repeated probing, however, low single trial decoding accuracy, and high subject variability may limit practical applicability.

Analysis of Prefrontal Single-Channel EEG Data for Portable Auditory ERP-Based Brain-Computer Interfaces

An electroencephalogram (EEG)-based brain-computer interface (BCI) is a tool to non-invasively control computers by translating the electrical activity of the brain. This technology has the potential to provide patients who have severe generalized myopathy, such as those suffering from amyotrophic lateral sclerosis (ALS), with the ability to communicate. Recently, auditory oddball paradigms have been developed to implement more practical event-related potential (ERP)-based BCIs because they can operate without ocular activities. These paradigms generally make use of clinical (over 16-channel) EEG devices and natural sound stimuli to maintain the user's motivation during the BCI operation; however, most ALS patients who have taken part in auditory ERP-based BCIs tend to complain about the following factors: (i) total device cost and (ii) setup time. The development of a portable auditory ERP-based BCI could overcome considerable obstacles that prevent the use of this technology in communication in everyday life. To address this issue, we analyzed prefrontal single-channel EEG data acquired from a consumer-grade single-channel EEG device using a natural sound-based auditory oddball paradigm. In our experiments, EEG data was gathered from nine healthy subjects and one ALS patient. The performance of auditory ERP-based BCI was quantified under an offline condition and two online conditions. The offline analysis indicated that our paradigm maintained a high level of detection accuracy (%) and ITR (bits/min) across all subjects through a cross-validation procedure (for five commands: 70.0 ± 16.1 and 1.29 ± 0.93, for four commands: 73.8 ± 14.2 and 1.16 ± 0.78, for three commands: 78.7 ± 11.8 and 0.95 ± 0.61, and for two commands: 85.7 ± 8.6 and 0.63 ± 0.38). Furthermore, the first online analysis demonstrated that our paradigm also achieved high performance for new data in an online data acquisition stream (for three commands: 80.0 ± 19.4 and 1.16 ± 0.83). The second online analysis measured online performances on the different day of offline and first online analyses on a different day (for three commands: 62.5 ± 14.3 and 0.43 ± 0.36). These results indicate that prefrontal single-channel EEGs have the potential to contribute to the development of a user-friendly portable auditory ERP-based BCI.

Guidelines for Feature Matching Assessment of Brain-Computer Interfaces for Augmentative and Alternative Communication

PURPOSE

Brain-computer interfaces (BCIs) can provide access to augmentative and alternative communication (AAC) devices using neurological activity alone without voluntary movements. As with traditional AAC access methods, BCI performance may be influenced by the cognitive-sensory-motor and motor imagery profiles of those who use these devices. Therefore, we propose a person-centered, feature matching framework consistent with clinical AAC best practices to ensure selection of the most appropriate BCI technology to meet individuals' communication needs.

METHOD

The proposed feature matching procedure is based on the current state of the art in BCI technology and published reports on cognitive, sensory, motor, and motor imagery factors important for successful operation of BCI devices.

RESULTS

Considerations for successful selection of BCI for accessing AAC are summarized based on interpretation from a multidisciplinary team with experience in AAC, BCI, neuromotor disorders, and cognitive assessment. The set of features that support each BCI option are discussed in a hypothetical case format to model possible transition of BCI research from the laboratory into clinical AAC applications.

CONCLUSIONS

This procedure is an initial step toward consideration of feature matching assessment for the full range of BCI devices. Future investigations are needed to fully examine how person-centered factors influence BCI performance across devices.

Communication Matters-Pitfalls and Promise of Hightech Communication Devices in Palliative Care of Severely Physically Disabled Patients With Amyotrophic Lateral Sclerosis

Amyotrophic lateral sclerosis (ALS) is the most common motor neuron disease, leading to progressive paralysis, dysarthria, dysphagia, and respiratory disabilities. Therapy is mostly focused on palliative interventions. During the course of the disease, verbal as well as nonverbal communicative abilities become more and more impaired. In this light, communication has been argued to be “the essence of human life” and crucial for patients' quality of life. High-tech augmentative and alternative communication (HT-AAC) technologies such as eyetracking based computer devices and brain-computer-interfaces provide the possibility to maintain caregiver-independent communication and environmental control even in the advanced disease state of ALS. Thus, they enable patients to preserve social participation and to independently communicate end-of-life-decisions. In accordance with these functions of HT-AAC, their use is reported to strengthen self-determination, increase patients' quality of life and reduce caregiver burden. Therefore, HT-AAC should be considered as standard of (palliative) care for people with ALS. On the other hand, the supply with individually tailored HT-AAC technologies is limited by external and patient-inherent variables. This review aims to provide an overview of the possibilities and limitations of HT-AAC technologies and discuss their role in the palliative care for patients with ALS.

Independent home use of a brain-computer interface by people with amyotrophic lateral sclerosis

OBJECTIVE

To assess the reliability and usefulness of an EEG-based brain-computer interface (BCI) for patients with advanced amyotrophic lateral sclerosis (ALS) who used it independently at home for up to 18 months.

METHODS

Of 42 patients consented, 39 (93%) met the study criteria, and 37 (88%) were assessed for use of the Wadsworth BCI. Nine (21%) could not use the BCI. Of the other 28, 27 (men, age 28-79 years) (64%) had the BCI placed in their homes, and they and their caregivers were trained to use it. Use data were collected by Internet. Periodic visits evaluated BCI benefit and burden and quality of life.

RESULTS

Over subsequent months, 12 (29% of the original 42) left the study because of death or rapid disease progression and 6 (14%) left because of decreased interest. Fourteen (33%) completed training and used the BCI independently, mainly for communication. Technical problems were rare. Patient and caregiver ratings indicated that BCI benefit exceeded burden. Quality of life remained stable. Of those not lost to the disease, half completed the study; all but 1 patient kept the BCI for further use.

CONCLUSION

The Wadsworth BCI home system can function reliably and usefully when operated by patients in their homes. BCIs that support communication are at present most suitable for people who are severely disabled but are otherwise in stable health. Improvements in BCI convenience and performance, including some now underway, should increase the number of people who find them useful and the extent to which they are used.

Brain-Computer Interfaces for Augmentative and Alternative Communication: A Tutorial

PURPOSE

Brain-computer interfaces (BCIs) have the potential to improve communication for people who require but are unable to use traditional augmentative and alternative communication (AAC) devices. As BCIs move toward clinical practice, speech-language pathologists (SLPs) will need to consider their appropriateness for AAC intervention.

METHOD

This tutorial provides a background on BCI approaches to provide AAC specialists foundational knowledge necessary for clinical application of BCI. Tutorial descriptions were generated based on a literature review of BCIs for restoring communication.

RESULTS

The tutorial responses directly address 4 major areas of interest for SLPs who specialize in AAC: (a) the current state of BCI with emphasis on SLP scope of practice (including the subareas: the way in which individuals access AAC with BCI, the efficacy of BCI for AAC, and the effects of fatigue), (b) populations for whom BCI is best suited, (c) the future of BCI as an addition to AAC access strategies, and (d) limitations of BCI.

CONCLUSION

Current BCIs have been designed as access methods for AAC rather than a replacement; therefore, SLPs can use existing knowledge in AAC as a starting point for clinical application. Additional training is recommended to stay updated with rapid advances in BCI.

Examining sensory ability, feature matching and assessment-based adaptation for a brain-computer interface using the steady-state visually evoked potential

PURPOSE

We investigated how overt visual attention and oculomotor control influence successful use of a visual feedback brain-computer interface (BCI) for accessing augmentative and alternative communication (AAC) devices in a heterogeneous population of individuals with profound neuromotor impairments. BCIs are often tested within a single patient population limiting generalization of results. This study focuses on examining individual sensory abilities with an eye toward possible interface adaptations to improve device performance.

METHODS

Five individuals with a range of neuromotor disorders participated in four-choice BCI control task involving the steady state visually evoked potential. The BCI graphical interface was designed to simulate a commercial AAC device to examine whether an integrated device could be used successfully by individuals with neuromotor impairment.

RESULTS

All participants were able to interact with the BCI and highest performance was found for participants able to employ an overt visual attention strategy. For participants with visual deficits to due to impaired oculomotor control, effective performance increased after accounting for mismatches between the graphical layout and participant visual capabilities.

CONCLUSION

As BCIs are translated from research environments to clinical applications, the assessment of BCI-related skills will help facilitate proper device selection and provide individuals who use BCI the greatest likelihood of immediate and long term communicative success. Overall, our results indicate that adaptations can be an effective strategy to reduce barriers and increase access to BCI technology. These efforts should be directed by comprehensive assessments for matching individuals to the most appropriate device to support their complex communication needs. Implications for Rehabilitation Brain computer interfaces using the steady state visually evoked potential can be integrated with an augmentative and alternative communication device to provide access to language and literacy for individuals with neuromotor impairment. Comprehensive assessments are needed to fully understand the sensory, motor, and cognitive abilities of individuals who may use brain-computer interfaces for proper feature matching as selection of the most appropriate device including optimization device layouts and control paradigms. Oculomotor impairments negatively impact brain-computer interfaces that use the steady state visually evoked potential, but modifications to place interface stimuli and communication items in the intact visual field can improve successful outcomes.

A Multifunctional Brain-Computer Interface Intended for Home Use: An Evaluation with Healthy Participants and Potential End Users with Dry and Gel-Based Electrodes

Current brain-computer interface (BCIs) software is often tailored to the needs of scientists and technicians and therefore complex to allow for versatile use. To facilitate home use of BCIs a multifunctional P300 BCI with a graphical user interface intended for non-expert set-up and control was designed and implemented. The system includes applications for spelling, web access, entertainment, artistic expression and environmental control. In addition to new software, it also includes new hardware for the recording of electroencephalogram (EEG) signals. The EEG system consists of a small and wireless amplifier attached to a cap that can be equipped with gel-based or dry contact electrodes. The system was systematically evaluated with a healthy sample, and targeted end users of BCI technology, i.e., people with a varying degree of motor impairment tested the BCI in a series of individual case studies. Usability was assessed in terms of effectiveness, efficiency and satisfaction. Feedback of users was gathered with structured questionnaires. Two groups of healthy participants completed an experimental protocol with the gel-based and the dry contact electrodes (N = 10 each). The results demonstrated that all healthy participants gained control over the system and achieved satisfactory to high accuracies with both gel-based and dry electrodes (average error rates of 6 and 13%). Average satisfaction ratings were high, but certain aspects of the system such as the wearing comfort of the dry electrodes and design of the cap, and speed (in both groups) were criticized by some participants. Six potential end users tested the system during supervised sessions. The achieved accuracies varied greatly from no control to high control with accuracies comparable to that of healthy volunteers. Satisfaction ratings of the two end-users that gained control of the system were lower as compared to healthy participants. The advantages and disadvantages of the BCI and its applications are discussed and suggestions are presented for improvements to pave the way for user friendly BCIs intended to be used as assistive technology by persons with severe paralysis.

Evolutionary Algorithm Based Feature Optimization for Multi-Channel EEG Classification

The most BCI systems that rely on EEG signals employ Fourier based methods for time-frequency decomposition for feature extraction. The band-limited multiple Fourier linear combiner is well-suited for such band-limited signals due to its real-time applicability. Despite the improved performance of these techniques in two channel settings, its application in multiple-channel EEG is not straightforward and challenging. As more channels are available, a spatial filter will be required to eliminate the noise and preserve the required useful information. Moreover, multiple-channel EEG also adds the high dimensionality to the frequency feature space. Feature selection will be required to stabilize the performance of the classifier. In this paper, we develop a new method based on Evolutionary Algorithm (EA) to solve these two problems simultaneously. The real-valued EA encodes both the spatial filter estimates and the feature selection into its solution and optimizes it with respect to the classification error. Three Fourier based designs are tested in this paper. Our results show that the combination of Fourier based method with covariance matrix adaptation evolution strategy (CMA-ES) has the best overall performance.

Workshops of the Sixth International Brain-Computer Interface Meeting: brain-computer interfaces past, present, and future

The Sixth International Brain-Computer Interface (BCI) Meeting was held 30 May-3 June 2016 at the Asilomar Conference Grounds, Pacific Grove, California, USA. The conference included 28 workshops covering topics in BCI and brain-machine interface research. Topics included BCI for specific populations or applications, advancing BCI research through use of specific signals or technological advances, and translational and commercial issues to bring both implanted and non-invasive BCIs to market. BCI research is growing and expanding in the breadth of its applications, the depth of knowledge it can produce, and the practical benefit it can provide both for those with physical impairments and the general public. Here we provide summaries of each workshop, illustrating the breadth and depth of BCI research and highlighting important issues and calls for action to support future research and development.

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.

Near-infrared spectroscopy (NIRS)-based eyes-closed brain-computer interface (BCI) using prefrontal cortex activation due to mental arithmetic

We propose a near-infrared spectroscopy (NIRS)-based brain-computer interface (BCI) that can be operated in eyes-closed (EC) state. To evaluate the feasibility of NIRS-based EC BCIs, we compared the performance of an eye-open (EO) BCI paradigm and an EC BCI paradigm with respect to hemodynamic response and classification accuracy. To this end, subjects performed either mental arithmetic or imagined vocalization of the English alphabet as a baseline task with very low cognitive loading. The performances of two linear classifiers were compared; resulting in an advantage of shrinkage linear discriminant analysis (LDA). The classification accuracy of EC paradigm (75.6 ± 7.3%) was observed to be lower than that of EO paradigm (77.0 ± 9.2%), which was statistically insignificant (p = 0.5698). Subjects reported they felt it more comfortable (p = 0.057) and easier (p < 0.05) to perform the EC BCI tasks. The different task difficulty may become a cause of the slightly lower classification accuracy of EC data. From the analysis results, we could confirm the feasibility of NIRS-based EC BCIs, which can be a BCI option that may ultimately be of use for patients who cannot keep their eyes open consistently.

An Evaluation of Training with an Auditory P300 Brain-Computer Interface for the Japanese Hiragana Syllabary

Gaze-independent brain-computer interfaces (BCIs) are a possible communication channel for persons with paralysis. We investigated if it is possible to use auditory stimuli to create a BCI for the Japanese Hiragana syllabary, which has 46 Hiragana characters. Additionally, we investigated if training has an effect on accuracy despite the high amount of different stimuli involved. Able-bodied participants (N = 6) were asked to select 25 syllables (out of fifty possible choices) using a two step procedure: First the consonant (ten choices) and then the vowel (five choices). This was repeated on 3 separate days. Additionally, a person with spinal cord injury (SCI) participated in the experiment. Four out of six healthy participants reached Hiragana syllable accuracies above 70% and the information transfer rate increased from 1.7 bits/min in the first session to 3.2 bits/min in the third session. The accuracy of the participant with SCI increased from 12% (0.2 bits/min) to 56% (2 bits/min) in session three. Reliable selections from a 10 × 5 matrix using auditory stimuli were possible and performance is increased by training. We were able to show that auditory P300 BCIs can be used for communication with up to fifty symbols. This enables the use of the technology of auditory P300 BCIs with a variety of applications.

BCI in practice

Brain-computer interfaces are systems that use signals recorded from the brain to enable communication and control applications for individuals who have impaired function. This technology has developed to the point that it is now being used by individuals who can actually benefit from it. However, there are several outstanding issues that prevent widespread use. These include the ease of obtaining high-quality recordings by home users, the speed, and accuracy of current devices and adapting applications to the needs of the user. In this chapter, we discuss some of these unsolved issues.

P300-based brain-computer interface (BCI) event-related potentials (ERPs): People with amyotrophic lateral sclerosis (ALS) vs. age-matched controls

OBJECTIVE

Brain-computer interfaces (BCIs) aimed at restoring communication to people with severe neuromuscular disabilities often use event-related potentials (ERPs) in scalp-recorded EEG activity. Up to the present, most research and development in this area has been done in the laboratory with young healthy control subjects. In order to facilitate the development of BCI most useful to people with disabilities, the present study set out to: (1) determine whether people with amyotrophic lateral sclerosis (ALS) and healthy, age-matched volunteers (HVs) differ in the speed and accuracy of their ERP-based BCI use; (2) compare the ERP characteristics of these two groups; and (3) identify ERP-related factors that might enable improvement in BCI performance for people with disabilities.

METHODS

Sixteen EEG channels were recorded while people with ALS or healthy age-matched volunteers (HVs) used a P300-based BCI. The subjects with ALS had little or no remaining useful motor control (mean ALS Functional Rating Scale-Revised 9.4 (±9.5SD) (range 0-25)). Each subject attended to a target item as the items in a 6×6 visual matrix flashed. The BCI used a stepwise linear discriminant function (SWLDA) to determine the item the user wished to select (i.e., the target item). Offline analyses assessed the latencies, amplitudes, and locations of ERPs to the target and non-target items for people with ALS and age-matched control subjects.

RESULTS

BCI accuracy and communication rate did not differ significantly between ALS users and HVs. Although ERP morphology was similar for the two groups, their target ERPs differed significantly in the location and amplitude of the late positivity (P300), the amplitude of the early negativity (N200), and the latency of the late negativity (LN).

CONCLUSIONS

The differences in target ERP components between people with ALS and age-matched HVs are consistent with the growing recognition that ALS may affect cortical function. The development of BCIs for use by this population may begin with studies in HVs but also needs to include studies in people with ALS. Their differences in ERP components may affect the selection of electrode montages, and might also affect the selection of presentation parameters (e.g., matrix design, stimulation rate).

SIGNIFICANCE

P300-based BCI performance in people severely disabled by ALS is similar to that of age-matched control subjects. At the same time, their ERP components differ to some degree from those of controls. Attention to these differences could contribute to the development of BCIs useful to those with ALS and possibly to others with severe neuromuscular disabilities.

A Gaze Independent Brain-Computer Interface Based on Visual Stimulation through Closed Eyelids

A classical brain-computer interface (BCI) based on visual event-related potentials (ERPs) is of limited application value for paralyzed patients with severe oculomotor impairments. In this study, we introduce a novel gaze independent BCI paradigm that can be potentially used for such end-users because visual stimuli are administered on closed eyelids. The paradigm involved verbally presented questions with 3 possible answers. Online BCI experiments were conducted with twelve healthy subjects, where they selected one option by attending to one of three different visual stimuli. It was confirmed that typical cognitive ERPs can be evidently modulated by the attention of a target stimulus in eyes-closed and gaze independent condition, and further classified with high accuracy during online operation (74.58% ± 17.85 s.d.; chance level 33.33%), demonstrating the effectiveness of the proposed novel visual ERP paradigm. Also, stimulus-specific eye movements observed during stimulation were verified as reflex responses to light stimuli, and they did not contribute to classification. To the best of our knowledge, this study is the first to show the possibility of using a gaze independent visual ERP paradigm in an eyes-closed condition, thereby providing another communication option for severely locked-in patients suffering from complex ocular dysfunctions.

Comparison of eye tracking, electrooculography and an auditory brain-computer interface for binary communication: a case study with a participant in the locked-in state

BACKGROUND

In this study, we evaluated electrooculography (EOG), an eye tracker and an auditory brain-computer interface (BCI) as access methods to augmentative and alternative communication (AAC). The participant of the study has been in the locked-in state (LIS) for 6 years due to amyotrophic lateral sclerosis. He was able to communicate with slow residual eye movements, but had no means of partner independent communication. We discuss the usability of all tested access methods and the prospects of using BCIs as an assistive technology.

METHODS

Within four days, we tested whether EOG, eye tracking and a BCI would allow the participant in LIS to make simple selections. We optimized the parameters in an iterative procedure for all systems.

RESULTS

The participant was able to gain control over all three systems. Nonetheless, due to the level of proficiency previously achieved with his low-tech AAC method, he did not consider using any of the tested systems as an additional communication channel. However, he would consider using the BCI once control over his eye muscles would no longer be possible. He rated the ease of use of the BCI as the highest among the tested systems, because no precise eye movements were required; but also as the most tiring, due to the high level of attention needed to operate the BCI.

CONCLUSIONS

In this case study, the partner based communication was possible due to the good care provided and the proficiency achieved by the interlocutors. To ease the transition from a low-tech AAC method to a BCI once control over all muscles is lost, it must be simple to operate. For persons, who rely on AAC and are affected by a progressive neuromuscular disease, we argue that a complementary approach, combining BCIs and standard assistive technology, can prove valuable to achieve partner independent communication and ease the transition to a purely BCI based approach. Finally, we provide further evidence for the importance of a user-centered approach in the design of new assistive devices.

Systems Neuroengineering: Understanding and Interacting with the Brain

In this paper, we review the current state-of-the-art techniques used for understanding the inner workings of the brain at a systems level. The neural activity that governs our everyday lives involves an intricate coordination of many processes that can be attributed to a variety of brain regions. On the surface, many of these functions can appear to be controlled by specific anatomical structures; however, in reality, numerous dynamic networks within the brain contribute to its function through an interconnected web of neuronal and synaptic pathways. The brain, in its healthy or pathological state, can therefore be best understood by taking a systems-level approach. While numerous neuroengineering technologies exist, we focus here on three major thrusts in the field of systems neuroengineering: neuroimaging, neural interfacing, and neuromodulation. Neuroimaging enables us to delineate the structural and functional organization of the brain, which is key in understanding how the neural system functions in both normal and disease states. Based on such knowledge, devices can be used either to communicate with the neural system, as in neural interface systems, or to modulate brain activity, as in neuromodulation systems. The consideration of these three fields is key to the development and application of neuro-devices. Feedback-based neuro-devices require the ability to sense neural activity (via a neuroimaging modality) through a neural interface (invasive or noninvasive) and ultimately to select a set of stimulation parameters in order to alter neural function via a neuromodulation modality. Systems neuroengineering refers to the use of engineering tools and technologies to image, decode, and modulate the brain in order to comprehend its functions and to repair its dysfunction. Interactions between these fields will help to shape the future of systems neuroengineering-to develop neurotechniques for enhancing the understanding of whole-brain function and dysfunction, and the management of neurological and mental disorders.

Improving BCI performance through co-adaptation: applications to the P300-speller

A well-known neurophysiological marker that can easily be captured with electroencephalography (EEG) is the so-called P300: a positive signal deflection occurring at about 300 ms after a relevant stimulus. This brain response is particularly salient when the target stimulus is rare among a series of distracting stimuli, whatever the type of sensory input. Therefore, it has been proposed and extensively studied as a possible feature for direct brain-computer communication. The most advanced non-invasive BCI application based on this principle is the P300-speller. However, it is still a matter of debate whether this application will prove relevant to any population of patients. In a series of recent theoretical and empirical studies, we have been using this P300-based paradigm to push forward the performance of non-invasive BCI. This paper summarizes the proposed improvements and obtained results. Importantly, those could be generalized to many kinds of BCI, beyond this particular application. Indeed, they relate to most of the key components of a closed-loop BCI, namely: improving the accuracy of the system by trying to detect and correct for errors automatically; optimizing the computer's speed-accuracy trade-off by endowing it with adaptive behavior; but also simplifying the hardware and time for set-up in the aim of routine use in patients. Our results emphasize the importance of the closed-loop interaction and of the ensuing co-adaptation between the user and the machine whenever possible. Most of our evaluations have been conducted in healthy subjects. We conclude with perspectives for clinical applications.

Brain-computer interface (BCI) evaluation in people with amyotrophic lateral sclerosis

Brain-computer interfaces (BCIs) might restore communication to people severely disabled by amyotrophic lateral sclerosis (ALS) or other disorders. We sought to: 1) define a protocol for determining whether a person with ALS can use a visual P300-based BCI; 2) determine what proportion of this population can use the BCI; and 3) identify factors affecting BCI performance. Twenty-five individuals with ALS completed an evaluation protocol using a standard 6 × 6 matrix and parameters selected by stepwise linear discrimination. With an 8-channel EEG montage, the subjects fell into two groups in BCI accuracy (chance accuracy 3%). Seventeen averaged 92 (± 3)% (range 71-100%), which is adequate for communication (G70 group). Eight averaged 12 (± 6)% (range 0-36%), inadequate for communication (L40 subject group). Performance did not correlate with disability: 11/17 (65%) of G70 subjects were severely disabled (i.e. ALSFRS-R < 5). All L40 subjects had visual impairments (e.g. nystagmus, diplopia, ptosis). P300 was larger and more anterior in G70 subjects. A 16-channel montage did not significantly improve accuracy. In conclusion, most people severely disabled by ALS could use a visual P300-based BCI for communication. In those who could not, visual impairment was the principal obstacle. For these individuals, auditory P300-based BCIs might be effective.