Eye blink characterization from frontal EEG electrodes using source separation and pattern recognition algorithms

Brain-computer interface for robot control with eye artifacts for assistive applications

Human-robot interaction is a rapidly developing field and robots have been taking more active roles in our daily lives. Patient care is one of the fields in which robots are becoming more present, especially for people with disabilities. People with neurodegenerative disorders might not consciously or voluntarily produce movements other than those involving the eyes or eyelids. In this context, Brain-Computer Interface (BCI) systems present an alternative way to communicate or interact with the external world. In order to improve the lives of people with disabilities, this paper presents a novel BCI to control an assistive robot with user's eye artifacts. In this study, eye artifacts that contaminate the electroencephalogram (EEG) signals are considered a valuable source of information thanks to their high signal-to-noise ratio and intentional generation. The proposed methodology detects eye artifacts from EEG signals through characteristic shapes that occur during the events. The lateral movements are distinguished by their ordered peak and valley formation and the opposite phase of the signals measured at F7 and F8 channels. This work, as far as the authors' knowledge, is the first method that used this behavior to detect lateral eye movements. For the blinks detection, a double-thresholding method is proposed by the authors to catch both weak blinks as well as regular ones, differentiating itself from the other algorithms in the literature that normally use only one threshold. Real-time detected events with their virtual time stamps are fed into a second algorithm, to further distinguish between double and quadruple blinks from single blinks occurrence frequency. After testing the algorithm offline and in realtime, the algorithm is implemented on the device. The created BCI was used to control an assistive robot through a graphical user interface. The validation experiments including 5 participants prove that the developed BCI is able to control the robot.

Ear-EEG sensitivity modeling for neural sources and ocular artifacts

The ear-EEG has emerged as a promising candidate for real-world wearable brain monitoring. While experimental studies have validated several applications of ear-EEG, the source-sensor relationship for neural sources from across the brain surface has not yet been established. In addition, modeling of the ear-EEG sensitivity to sources of artifacts is still missing. Through volume conductor modeling, the sensitivity of various configurations of ear-EEG is established for a range of neural sources, in addition to ocular artifact sources for the blink, vertical saccade, and horizontal saccade eye movements. Results conclusively support the introduction of ear-EEG into conventional EEG paradigms for monitoring neural activity that originates from within the temporal lobes, while also revealing the extent to which ear-EEG can be used for sources further away from these regions. The use of ear-EEG in scenarios prone to ocular artifacts is also supported, through the demonstration of proportional scaling of artifacts and neural signals in various configurations of ear-EEG. The results from this study can be used to support both existing and prospective experimental ear-EEG studies and applications in the context of sensitivity to both neural sources and ocular artifacts.

EEG Feature Extraction Using Evolutionary Algorithms for Brain-Computer Interface Development

Brain-computer interfaces are systems capable of mapping brain activity to specific commands, which enables to remotely automate different types of processes in hardware devices or software applications. However, the development of brain-computer interfaces has been limited by several factors that affect their performance, such as the characterization of events in brain signals and the excessive processing load generated by the high volume of data. In this paper, we propose a method based on computational intelligence techniques to handle these problems, turning them into a single optimization problem. An artificial neural network is used as a classifier for event detection, along with an evolutionary algorithm to find the optimal subset of electrodes and data points that better represents the target event. The obtained results indicate our approach is a competitive and viable alternative for feature extraction in electroencephalograms, leading to high accuracy values and allowing the reduction of a significant amount of data.

Blinking characterization for each eye from EEG analysis using wavelets

Eye blinks can be used to perform monitoring tasks such as drowsiness detection, attention measurement or other biological measurement mainly using video data. With the developement of brain computer interfaces (BCI) eye movements and blinks could be used to perform control tasks such as pointer activation or communications. This work aims to prove that it is possible to characterize eye blinks for each eye separately using only electroencephalography (EEG) signal acquired through non invasive portable device and dry electroencephalography.

EEG-Based Index for Timely Detecting User's Drowsiness Occurrence in Automotive Applications

Human errors are widely considered among the major causes of road accidents. Furthermore, it is estimated that more than 90% of vehicle crashes causing fatal and permanent injuries are directly related to mental tiredness, fatigue, and drowsiness of the drivers. In particular, driving drowsiness is recognized as a crucial aspect in the context of road safety, since drowsy drivers can suddenly lose control of the car. Moreover, the driving drowsiness episodes mostly appear suddenly without any prior behavioral evidence. The present study aimed at characterizing the onset of drowsiness in car drivers by means of a multimodal neurophysiological approach to develop a synthetic electroencephalographic (EEG)-based index, able to detect drowsy events. The study involved 19 participants in a simulated scenario structured in a sequence of driving tasks under different situations and traffic conditions. The experimental conditions were designed to induce prominent mental drowsiness in the final part. The EEG-based index, so-called “MDrow index”, was developed and validated to detect the driving drowsiness of the participants. The MDrow index was derived from the Global Field Power calculated in the Alpha EEG frequency band over the parietal brain sites. The results demonstrated the reliability of the proposed MDrow index in detecting the driving drowsiness experienced by the participants, resulting also more sensitive and timely sensible with respect to more conventional autonomic parameters, such as the EyeBlinks Rate and the Heart Rate Variability, and to subjective measurements (self-reports).

Ensemble Deep Neural Network for Automatic Classification of EEG Independent Components

OBJECTIVE

Independent component analysis (ICA) is commonly used to remove noisy artifacts from multi-channel scalp electroencephalogram (EEG) signals. ICA decomposes EEG into different independent components (ICs) and then, experts remove the noisy ones. This process is highly time-consuming and experts are not always available. To surpass this drawback, research is going on to develop models to automatically conduct IC classification. Current state-of-the-art models use power spectrum densities (PSDs) and topoplots to classify ICs. The performance of these methods may be limited by disregarding the IC time-series that would contribute to fully simulate the visual inspection performed by experts.

METHODS

We present a novel ensemble deep neural network that combines time-series, PSDs, and topoplots to classify ICs. Moreover, we study the ability to use our model in transfer learning approaches.

RESULTS

Experimental results showed that using time-series improves IC classification. Results also indicated that transfer learning obtained higher performance than simply training a new model from scratch.

CONCLUSION

Researchers should develop IC classifiers using the three sources of information. Moreover, transfer learning approaches should be considered when producing new deep learning models.

SIGNIFICANCE

This work improves IC classification, enhancing the automatic removal of EEG artifacts. Additionally, since labelled ICs are generally not publicly available, the possibility of using our model in transfer learning studies may motivate other researchers to develop their own classifiers.

Simultaneous Eye Blink Characterization and Elimination from Low-Channel Prefrontal EEG Signals Enhances Driver Drowsiness Detection

OBJECTIVE

Blink-related features derived from electroencephalography (EEG) have recently arisen as a meaningful measure of drivers cognitive state. Combined with band power features of low-channel prefrontal EEG data, blink-derived features enhance the detection of driver drowsiness. Yet, it remains unanswered whether synergy of combined blink and EEG band power features for the detection of driver drowsiness may be further boosted if a proper eye blink removal is also applied before EEG analysis. This paper proposes an algorithm for simultaneous eye blink feature extraction and elimination from low-channel prefrontal EEG data.

METHODS

Firstly, eye blink intervals (EBIs) are identified from the Fp1 EEG channel using variational mode extraction, and then blink-related features are derived. Secondly, the identified EBIs are projected to the rest of EEG channels and then filtered by a combination of principal component analysis and discrete wavelet transform. Thirdly, a support vector machine with 10-fold cross-validation is employed to classify alert and drowsy states from the derived blink and filtered EEG band power features.

MAIN RESULTS

When compared the synergy of eye blink and EEG features before and after filtering by the proposed algorithm, a significant improvement in the mean accuracy of driver drowsiness detection was achieved (71.2% vs. 78.1%, p<0.05).

SIGNIFICANCE

This paper validates a novel view of eye blinks as both a source of information and artifacts in EEG-based driver drowsiness detection.

Comparative analysis of spontaneous blinking and the corneal reflex

Ocular surface health, the cognitive status, psychological health or human neurological disorders, among others, can be assessed by studying eye blinking, which can be differentiated in spontaneous, reflex and voluntary. Its diagnostic potential has provided a great number of works that evaluate their characteristics and variations depending on the subject's condition (sex, tiredness, health, …). The objective of this study was to analyse the differences in blinking kinematics of spontaneous and reflex blinks, distinguishing between direct and consensual reflexes, using a self-developed, non-invasive and image processing-based method. A video-oculography system is proposed using an air jet driven by a syringe to induce reflex and a high-speed camera to record the blinking of both eyes. The light intensity diffused by the eye changes during blinking and peaks when the eyelid closes. Sixty-second sequences were recorded of 25 subjects blinking. Intensity curves were off-line fitted to an exponentially modified Gaussian (EMG) function, whose σ, μ and τ parameters were analysed. A two-way analysis of variance (ANOVA) of these parameters was conducted to test the influence of the subject, the eye and blink type. In the closing phase, direct and consensual corneal reflexes are faster than spontaneous blinking, but there was no significant difference between them, nor between right and left eyes. In the opening phase, the direct corneal reflex was the slowest and significant differences appeared between right and left eyes.

How Can Physiological Computing Benefit Human-Robot Interaction?

As systems grow more automatized, the human operator is all too often overlooked. Although human-robot interaction (HRI) can be quite demanding in terms of cognitive resources, the mental states (MS) of the operators are not yet taken into account by existing systems. As humans are no providential agents, this lack can lead to hazardous situations. The growing number of neurophysiology and machine learning tools now allows for efficient operators’ MS monitoring. Sending feedback on MS in a closed-loop solution is therefore at hand. Involving a consistent automated planning technique to handle such a process could be a significant asset. This perspective article was meant to provide the reader with a synthesis of the significant literature with a view to implementing systems that adapt to the operator’s MS to improve human-robot operations’ safety and performance. First of all, the need for this approach is detailed regarding remote operation, an example of HRI. Then, several MS identified as crucial for this type of HRI are defined, along with relevant electrophysiological markers. A focus is made on prime degraded MS linked to time-on-task and task demands, as well as collateral MS linked to system outputs (i.e., feedback and alarms). Lastly, the principle of symbiotic HRI is detailed and one solution is proposed to include the operator state vector into the system using a mixed-initiative decisional framework to drive such an interaction.

An oculometrics-based biofeedback system to impede fatigue development during computer work: A proof-of-concept study

A biofeedback system may objectively identify fatigue and provide an individualized timing plan for micro-breaks. We developed and implemented a biofeedback system based on oculometrics using continuous recordings of eye movements and pupil dilations to moderate fatigue development in its early stages. Twenty healthy young participants (10 males and 10 females) performed a cyclic computer task for 31–35 min over two sessions: 1) self-triggered micro-breaks (manual sessions), and 2) biofeedback-triggered micro-breaks (automatic sessions). The sessions were held with one-week inter-session interval and in a counterbalanced order across participants. Each session involved 180 cycles of the computer task and after each 20 cycles (a segment), the task paused for 5-s to acquire perceived fatigue using Karolinska Sleepiness Scale (KSS). Following the pause, a 25-s micro-break involving seated exercises was carried out whether it was triggered by the biofeedback system following the detection of fatigue (KSS≥5) in the automatic sessions or by the participants in the manual sessions. National Aeronautics and Space Administration Task Load Index (NASA-TLX) was administered after sessions. The functioning core of the biofeedback system was based on a Decision Tree Ensemble model for fatigue classification, which was developed using an oculometrics dataset previously collected during the same computer task. The biofeedback system identified fatigue with a mean accuracy of approx. 70%. Perceived workload obtained from NASA-TLX was significantly lower in the automatic sessions compared with the manual sessions, p = 0.01 Cohen’s d_z = 0.89. The results give support to the effectiveness of integrating oculometrics-based biofeedback in timing plan of micro-breaks to impede fatigue development during computer work.

Blinking characterization from high speed video records. Application to biometric authentication

The evaluation of eye blinking has been used for the diagnosis of neurological disorders and fatigue. Despite the extensive literature, no objective method has been found to analyze its kinematic and dynamic behavior. A non-contact technique based on the high-speed recording of the light reflected by the eyelid in the blinking process and the off-line processing of the sequence is presented. It allows for objectively determining the start and end of a blink, besides obtaining different physical magnitudes: position, speed, eyelid acceleration as well as the power, work and mechanical impulse developed by the muscles involved in the physiological process. The parameters derived from these magnitudes provide a unique set of features that can be used to biometric authentication. This possibility has been tested with a limited number of subjects with a correct identification rate of up to 99.7%, thus showing the potential application of the method.

Using Convolutional Neural Networks to Automatically Detect Eye-Blink Artifacts in Magnetoencephalography Without Resorting to Electrooculography

Magnetoencephelography (MEG) is a functional neuroimaging tool that records the magnetic fields induced by neuronal activity; however, signal from muscle activity often corrupts the data. Eye-blinks are one of the most common types of muscle artifact. They can be recorded by affixing eye proximal electrodes, as in electrooculography (EOG), however this complicates patient preparation and decreases comfort. Moreover, it can induce further muscular artifacts from facial twitching. We propose an EOG free, data driven approach. We begin with Independent Component Analysis (ICA), a well-known preprocessing approach that factors observed signal into statistically independent components. When applied to MEG, ICA can help separate neuronal components from non-neuronal ones, however, the components are randomly ordered. Thus, we develop a method to assign one of two labels, non-eye-blink or eye-blink, to each component. Our contributions are two-fold. First, we develop a 10-layer Convolutional Neural Network (CNN), which directly labels eye-blink artifacts. Second, we visualize the learned spatial features using attention mapping, to reveal what it has learned and bolster confidence in the method's ability to generalize to unseen data. We acquired 8-min, eyes open, resting state MEG from 44 subjects. We trained our method on the spatial maps from ICA of 14 subjects selected randomly with expertly labeled ground truth. We then tested on the remaining 30 subjects. Our approach achieves a test classification accuracy of 99.67%, sensitivity: 97.62%, specificity: 99.77%, and ROC AUC: 98.69%. We also show the learned spatial features correspond to those human experts typically use which corroborates our model's validity. This work (1) facilitates creation of fully automated processing pipelines in MEG that need to remove motion artifacts related to eye blinks, and (2) potentially obviates the use of additional EOG electrodes for the recording of eye-blinks in MEG studies.

Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.

Efficient Workload Classification based on Ignored Auditory Probes: A Proof of Concept

Mental workload is a mental state that is currently one of the main research focuses in neuroergonomics. It can notably be estimated using measurements in electroencephalography (EEG), a method that allows for direct mental state assessment. Auditory probes can be used to elicit event-related potentials (ERPs) that are modulated by workload. Although, some papers do report ERP modulations due to workload using attended or ignored probes, to our knowledge there is no literature regarding effective workload classification based on ignored auditory probes. In this paper, in order to efficiently estimate workload, we advocate for the use of such ignored auditory probes in a single-stimulus paradigm and a signal processing chain that includes a spatial filtering step. The effectiveness of this approach is demonstrated on data acquired from participants that performed the Multi-Attribute Task Battery - II. They carried out this task during two 10-min blocks. Each block corresponded to a workload condition that was pseudorandomly assigned. The easy condition consisted of two monitoring tasks performed in parallel, and the difficult one consisted of those two tasks with an additional plane driving task. Infrequent auditory probes were presented during the tasks and the participants were asked to ignore them. The EEG data were denoised and the probes' ERPs were extracted and spatially filtered using a canonical correlation analysis. Next, binary classification was performed using a Fisher LDA and a fivefold cross-validation procedure. Our method allowed for a very high estimation performance with a classification accuracy above 80% for every participant, and minimal intrusiveness thanks to the use of a single-stimulus paradigm. Therefore, this study paves the way to the efficient use of ERPs for mental state monitoring in close to real-life settings and contributes toward the development of adaptive user interfaces.

Passive BCI based on drowsiness detection: an fNIRS study

We use functional near-infrared spectroscopy (fNIRS) to discriminate the alert and drowsy states for a passive brain-computer interface (BCI). The passive brain signals for the drowsy state are acquired from the prefrontal and dorsolateral prefrontal cortex. The experiment is performed on 13 healthy subjects using a driving simulator, and their brain activity is recorded using a continuous-wave fNIRS system. Linear discriminant analysis (LDA) is employed for training and testing, using the data from the prefrontal, left- and right-dorsolateral prefrontal regions. For classification, eight features are tested: mean oxyhemoglobin, mean deoxyhemoglobin, skewness, kurtosis, signal slope, number of peaks, sum of peaks, and signal peak, in 0~5, 0~10, and 0~15 second time windows, respectively. The results show that the best performance for classification is achieved using mean oxyhemoglobin, the signal peak, and the sum of peaks as features. The average accuracies in the right dorsolateral prefrontal cortex (83.1, 83.4 and 84.9% in the 0~5, 0~10 and 0~15 second time windows, respectively) show that the proposed method has an effective utility for detection of drowsiness for a passive BCI.