Quantitative electro-oculography and electroencephalography as indices of alertness

Automatic Driver Drowsiness Detection Using Artificial Neural Network Based on Visual Facial Descriptors: Pilot Study

PURPOSE

Driving while drowsy is a major cause of traffic accidents globally. Recent technologies for detection and alarm within automobiles for this condition are limited by their reliability, practicality, cost, and lack of clinical validation. In this study, we developed an early drowsiness detection algorithm and device based on the "gold standard brain biophysiological signal" and facial expression digital data.

METHODS

The data were obtained from 10 participants. Artificial neural networks (ANN) were adopted as the model. Composite features of facial descriptors (ie, eye aspect ratio (EAR), mouth aspect ratio (MAR), face length (FL), and face width balance (FWB)) extracted from two-second video frames were investigated.

RESULTS

The ANN combined with the EAR and MAR features had the most sensitivity (70.12%) while the ANN combined with the EAR, MAR, and FL features had the most accuracy and specificity (60.76% and 58.71%, respectively). In addition, by applying the discrete Fourier transform (DFT) to the composite features, the ANN combined with the EAR and MAR features again had the highest sensitivity (72.25%), while the ANN combined with the EAR, MAR, and FL features had the highest accuracy and specificity (60.40% and 54.10%, respectively).

CONCLUSION

The ANN with DFT combined with the EAR, MAR, and FL offered the best performance. Our direct driver sleepiness detection system developed from the integration of biophysiological information and internal validation provides a valuable algorithm, specifically toward alertness level.

Algorithm for automatic analysis of electro-oculographic data

BACKGROUND

Large amounts of electro-oculographic (EOG) data, recorded during electroencephalographic (EEG) measurements, go underutilized. We present an automatic, auto-calibrating algorithm that allows efficient analysis of such data sets.

METHODS

The auto-calibration is based on automatic threshold value estimation. Amplitude threshold values for saccades and blinks are determined based on features in the recorded signal. The performance of the developed algorithm was tested by analyzing 4854 saccades and 213 blinks recorded in two different conditions: a task where the eye movements were controlled (saccade task) and a task with free viewing (multitask). The results were compared with results from a video-oculography (VOG) device and manually scored blinks.

RESULTS

The algorithm achieved 93% detection sensitivity for blinks with 4% false positive rate. The detection sensitivity for horizontal saccades was between 98% and 100%, and for oblique saccades between 95% and 100%. The classification sensitivity for horizontal and large oblique saccades (10 deg) was larger than 89%, and for vertical saccades larger than 82%. The duration and peak velocities of the detected horizontal saccades were similar to those in the literature. In the multitask measurement the detection sensitivity for saccades was 97% with a 6% false positive rate.

CONCLUSION

The developed algorithm enables reliable analysis of EOG data recorded both during EEG and as a separate metrics.

In the eye of the beholder: a survey of models for eyes and gaze

Despite active research and significant progress in the last 30 years, eye detection and tracking remains challenging due to the individuality of eyes, occlusion, variability in scale, location, and light conditions. Data on eye location and details of eye movements have numerous applications and are essential in face detection, biometric identification, and particular human-computer interaction tasks. This paper reviews current progress and state of the art in video-based eye detection and tracking in order to identify promising techniques as well as issues to be further addressed. We present a detailed review of recent eye models and techniques for eye detection and tracking. We also survey methods for gaze estimation and compare them based on their geometric properties and reported accuracies. This review shows that, despite their apparent simplicity, the development of a general eye detection technique involves addressing many challenges, requires further theoretical developments, and is consequently of interest to many other domains problems in computer vision and beyond.

Vigilance estimation by using electrooculographic features

This study aims at using electrooculographic (EOG) features, mainly slow eye movements (SEM), to estimate the human vigilance changes during a monotonous task. In particular, SEMs are first automatically detected by a method based on discrete wavelet transform, then linear dynamic system is used to find the trajectory of vigilance changes according to the SEM proportion. The performance of this system is evaluated by the correlation coefficients between the final outputs and the local error rates of the subjects. The result suggests that SEMs perform better than rapid eye movements (REM) and blinks in estimating the vigilance. Using SEM alone, the correlation can achieve 0.75 for off-line, while combined with a feature from blinks it reaches 0.79.

Wavelet analysis of electroencephalographic and electro-oculographic changes during the sleep onset period

This study investigates the relationship between changes in the electroencephalogram (EEG) and slow eye movements (SEMs) in the electro-oculogram (EOG) at the wake-sleep transition. Analysis of EEG and EOG is performed by the discrete wavelet transform and utilizes energy functions built within the multiresolution framework. In particular, SEMs are detected automatically by a computerized system, previously developed and validated; core of the system is a function of EOG energies at different scales of decomposition, which defines SEMs in rigorous energetic terms. Changes in EEG rhythms are characterized by considering the relative energy of EEG signal at each scale of decomposition. The analysis has been applied to EEG and EOG signals acquired on fifteen healthy subjects during polysomnography. In all the examined subjects, falling asleep is systematically accompanied by EEG energy redistribution among the different scales and by SEMs occurrence. In particular, SEMs anticipate EEG modifications, preceding alpha blocking and theta intrusion even by several (10-20) minutes. This result suggests that EOG activity may be used to monitor sleepiness and sleep onset and to predict decrease in behavioral performances associated with drowsiness.

Novel eye gaze tracking techniques under natural head movement

Most available remote eye gaze trackers have two characteristics that hinder them being widely used as the important computer input devices for human computer interaction. First, they have to be calibrated for each user individually; second, they have low tolerance for head movement and require the users to hold their heads unnaturally still. In this paper, by exploiting the eye anatomy, we propose two novel solutions to allow natural head movement and minimize the calibration procedure to only one time for a new individual. The first technique is proposed to estimate the 3-D eye gaze directly. In this technique, the cornea of the eyeball is modeled as a convex mirror. Via the properties of convex mirror, a simple method is proposed to estimate the 3-D optic axis of the eye. The visual axis, which is the true 3-D gaze direction of the user, can be determined subsequently after knowing the angle deviation between the visual axis and optic axis by a simple calibration procedure. Therefore, the gaze point on an object in the scene can be obtained by simply intersecting the estimated 3-D gaze direction with the object. Different from the first technique, our second technique does not need to estimate the 3-D eye gaze directly, and the gaze point on an object is estimated from a gaze mapping function implicitly. In addition, a dynamic computational head compensation model is developed to automatically update the gaze mapping function whenever the head moves. Hence, the eye gaze can be estimated under natural head movement. Furthermore, it minimizes the calibration procedure to only one time for a new individual. The advantage of the proposed techniques over the current state of the art eye gaze trackers is that it can estimate the eye gaze of the user accurately under natural head movement, without need to perform the gaze calibration every time before using it. Our proposed methods will improve the usability of the eye gaze tracking technology, and we believe that it represents an important step for the eye tracker to be accepted as a natural computer input device.

Monitoring sleepiness with on-board electrophysiological recordings for preventing sleep-deprived traffic accidents

OBJECTIVE

The objective of this study is the development and evaluation of efficient neurophysiological signal statistics, which may assess the driver's alertness level and serve as potential indicators of sleepiness in the design of an on-board countermeasure system.

METHODS

Multichannel EEG, EOG, EMG, and ECG were recorded from sleep-deprived subjects exposed to real field driving conditions. A number of severe driving errors occurred during the experiments. The analysis was performed in two main dimensions: the macroscopic analysis that estimates the on-going temporal evolution of physiological measurements during the driving task, and the microscopic event analysis that focuses on the physiological measurements' alterations just before, during, and after the driving errors. Two independent neurophysiologists visually interpreted the measurements. The EEG data were analyzed by using both linear and non-linear analysis tools.

RESULTS

We observed the occurrence of brief paroxysmal bursts of alpha activity and an increased synchrony among EEG channels before the driving errors. The alpha relative band ratio (RBR) significantly increased, and the Cross Approximate Entropy that quantifies the synchrony among channels also significantly decreased before the driving errors. Quantitative EEG analysis revealed significant variations of RBR by driving time in the frequency bands of delta, alpha, beta, and gamma. Most of the estimated EEG statistics, such as the Shannon Entropy, Kullback-Leibler Entropy, Coherence, and Cross-Approximate Entropy, were significantly affected by driving time. We also observed an alteration of eyes blinking duration by increased driving time and a significant increase of eye blinks' number and duration before driving errors.

CONCLUSIONS

EEG and EOG are promising neurophysiological indicators of driver sleepiness and have the potential of monitoring sleepiness in occupational settings incorporated in a sleepiness countermeasure device.

SIGNIFICANCE

The occurrence of brief paroxysmal bursts of alpha activity before severe driving errors is described in detail for the first time. Clear evidence is presented that eye-blinking statistics are sensitive to the driver's sleepiness and should be considered in the design of an efficient and driver-friendly sleepiness detection countermeasure device.

Use of subjective and physiological indicators of sleepiness to predict performance during a vigilance task

Sleepiness is a major risk factor for serious injury and death in accidents. Although it is important to develop countermeasures to sleepiness to reduce risks, it is equally important to determine the most effective timing for these countermeasures. To determine optimum timing for necessary countermeasures, we must be able to predict performance errors. This study examined the predictability of subjective and physiological indicators of sleepiness during a vigilance task. Thirteen healthy male volunteers (mean age, 26.9 yr; SD = 5.98 yr; range 22-43 yr) participated in the study. Participants used the Karolinska sleepiness scale (KSS) to rate their subjective sleepiness every 4 min during a 40-min Mackworth clock test. Electrophysiological and performance data were divided into 10 epochs (i.e., 1 epoch lasted for 4 min). To estimate predictability, the data from the sleepiness indicators used for the correlation analysis were preceded by one epoch to the performance data. Results showed that sleepiness indicators (KSS score and electroencephalographic [EEG] alpha activity) and standard deviation of heart rate (SDNN) were significantly correlated with succeeding performance on the vigilance test. These findings suggest that the KSS score, EEG alpha activity, and SDNN could be used to predict performance errors.

The use of two-channel electro-oculography in automatic detection of unintentional sleep onset

An automatic method was developed for detecting unintentional sleep onset. The automatic method is based on a two-channel electro-oculography (EOG) with left mastoid (M1) as reference. An automatic estimation of slow eye movements (SEM) was developed and used as the main criterion to separate sleep stage 1 (S1) from wakefulness. Additionally synchronous electroencephalographic (EEG) activity of sleep stages 1 and 2 was detected by calculating cross-correlation and amplitude difference in the 1.5-6 Hz theta band between the two EOG channels. Alpha power 8-12 Hz and beta power 18-30 Hz were used to determine wakefulness. Unintentional sleep onsets were studied using data from four separate maintenance of wakefulness test (MWT) sessions of 228 subjects. The automatic scoring of 30s sleep onset epochs using only EOG was compared to standard visual sleep stage scoring. The optimal detection thresholds were derived using data from 114 subjects and then applied to the data from different 114 subjects. Cohen's Kappa between the visual and the new automatic scoring system in separating wakefulness and sleep was substantial (0.67) with epoch by epoch agreement of 98%. The sleep epoch detection sensitivity was 70% and specificity 99%. The results are provided with a 1s delay for each 30s epoch. The developed method has to be tested in field applications. The advantage of the automatic method is that it could be applied during online recordings using only four disposable self-adhesive self-applicable electrodes.

Visual and computer-based detection of slow eye movements in overnight and 24-h EOG recordings

OBJECTIVE

The present work aimed to evaluate the performance of an automatic slow eye movement (SEM) detector in overnight and 24-h electro-oculograms (EOG) including all sleep stages (1, 2, 3, 4, REM) and wakefulness.

METHODS

Ten overnight and five 24-h EOG recordings acquired in healthy subjects were inspected by three experts to score SEMs. Computerized EOG analysis to detect SEMs was performed on 30-s epochs using an algorithm based on EOG wavelet transform, recently developed by our group and initially validated by considering only pre-sleep wakefulness, stages 1 and 2.

RESULTS

The validation procedure showed the algorithm could identify epochs containing SEM activity (concordance index k=0.62, 80.7% sensitivity, 63% selectivity). In particular, the experts and the algorithm identified SEM epochs mainly in pre-sleep wakefulness, stage 1, stage 2 and REM sleep. In addition, the algorithm yielded consistent indications as to the duration and position of SEM events within the epoch.

CONCLUSIONS

The study confirmed SEM activity at physiological sleep onset (pre-sleep wakefulness, stage 1 and stage 2), and also identified SEMs in REM sleep. The algorithm proved reliable even in the stages not used for its training.

SIGNIFICANCE

The study may enhance our understanding of SEM meaning and function. The algorithm is a reliable tool for automatic SEM detection, overcoming the inconsistency of manual scoring and reducing the time taken by experts.

A wavelet based method for automatic detection of slow eye movements: A pilot study

Electro-oculographic (EOG) activity during the wake-sleep transition is characterized by the appearance of slow eye movements (SEM). The present work describes an algorithm for the automatic localisation of SEM events from EOG recordings. The algorithm is based on a wavelet multiresolution analysis of the difference between right and left EOG tracings, and includes three main steps: (i) wavelet decomposition down to 10 detail levels (i.e., 10 scales), using Daubechies order 4 wavelet; (ii) computation of energy in 0.5s time steps at any level of decomposition; (iii) construction of a non-linear discriminant function expressing the relative energy of high-scale details to both high- and low-scale details. The main assumption is that the value of the discriminant function increases above a given threshold during SEM episodes due to energy redistribution toward higher scales. Ten EOG recordings from ten male patients with obstructive sleep apnea syndrome were used. All tracings included a period from pre-sleep wakefulness to stage 2 sleep. Two experts inspected the tracings separately to score SEMs. A reference set of SEM (gold standard) were obtained by joint examination by both experts. Parameters of the discriminant function were assigned on three tracings (design set) to minimize the disagreement between the system classification and classification by the two experts; the algorithm was then tested on the remaining seven tracings (test set). Results show that the agreement between the algorithm and the gold standard was 80.44+/-4.09%, the sensitivity of the algorithm was 67.2+/-7.37% and the selectivity 83.93+/-8.65%. However, most errors were not caused by an inability of the system to detect intervals with SEM activity against NON-SEM intervals, but were due to a different localisation of the beginning and end of some SEM episodes. The proposed method may be a valuable tool for computerized EOG analysis.

Prefrontal electrophysiologic "noise" and catechol-O-methyltransferase genotype in schizophrenia

BACKGROUND

Increased variability of stimulus-induced prefrontal electromagnetic activity ("noise") has been associated with genetic risk for schizophrenia. On the basis of animal experiments and computational models, we have predicted that this prefrontal "noise" phenotype would be related to variation in prefrontal dopamine (DA) signaling, which itself might be abnormal in schizophrenia. In the present study, the effect of a functional single nucleotide polymorphism (val(108/158)met) within the catechol-O-methyltransferase (COMT) gene on prefrontal "noise" was examined, because the COMT enzyme is involved in cortical synaptic dopamine metabolism and weakly predictive of risk for schizophrenia.

METHODS

A Caucasian sample comprising 112 unrelated normal subjects, 83 schizophrenic probands, and 87 of their unaffected siblings was investigated, all of whom had measures of prefrontal "noise" estimated from event-related electroencephalogram during an auditory oddball task.

RESULTS

The val(108/158)met genotype was significantly associated with prefrontal "noise"; homozygous Val-carriers had greatest prefrontal "noise" values; odds ratio (OR) = 2.37 (95% confidence interval [CI] 1.37-4.10), p = 003. The genotype-phenotype association was stronger when only considering male subjects with an OR = 3.37 (95% CI: 1.63-6.98), p = 002.

CONCLUSIONS

The results suggest that COMT genotype impacts the level of prefrontal physiologic "noise."

Oculomotor changes are associated to daytime sleepiness in the multiple sleep latency test

Sleep onsets in the diurnal multiple sleep latency test (MSLT), following different sleep lengths of the preceding night sleep (8, 5, 4, 3, 2, 1 h) and following the corresponding recovery nights, were considered for a study on changes of oculomotor activity during sleep onset. The study aimed to assess the individual time course in spontaneous blinks (SBs) and slow eye movements (SEMs) during the sleep onset period and also the relationship with sleep latencies in the MSLT. Group analyses compared oculomotor changes between conditions characterized by a different level of daytime sleepiness. The results show a clear inverse relation between the two oculomotor measures, with a linear SB decrease and quadratic SEM increase across the wake-sleep transition. A 150 s sample of SB and SEM activity at the start of MSLT trials correlates with individual subsequent sleep latency. Finally, mean changes in daytime sleepiness as measured by the MSLT are paralleled by coherent oculomotor changes, with a significant linear decrease of SB as sleepiness increases as a consequence of previous sleep reduction. Both individual and group results show that endogenous blinking is associated with moderate changes in daytime sleepiness.

Eyelid movements measured by Nightcap predict slow eye movements during quiet wakefulness in humans

A precipitous decline in eyelid movements (ELMs) has been shown to be a highly reliable indicator of sleep onset. While ELMs correlate well with eye movements during waking and rapid eye movement (REM) sleep, the eye sensor remains silent during the period of slow eye movements (SEMs) typical of sleep onset. If the ELM density (e.g. ELMs per minute) dropped simultaneously with the appearance of SEMs prior to sleep onset, it could be a promising tool for identifying decreases in alertness prior to overt sleep onset. The present study was designed to determine whether the presence of SEMs in the transitional period preceding stage 1 sleep is reflected in decreases in ELM density. ELM densities were computed for 2.5-s epochs with and without SEMs, as well as for 15-s epochs. Decreases in ELM density not only were an excellent correlate of the appearance of SEMs during wakefulness with closed eyes, but also a good predictor of their occurrence (c. 82% accuracy) at a time resolution of 2.5 s. Based on these results, we conclude that ELM density reliably predicts moderate changes in the level of alertness during quiet wakefulness.

Vigilance on the civil flight deck: incidence of sleepiness and sleep during long-haul flights and associated changes in physiological parameters

The study investigated sleepiness and sleep in aircrew during long-haul flights. The objectives were to identify loss of alertness and to recommend a practical approach to the design of an alerting system to be used by aircrew to prevent involuntary sleep. The flights were between London and Miami, covering both day- and night-time sectors, each with a duration of approximately 9 h. The subjects were 12 British Airways pilots. Various physiological variables were measured that could potentially be used to indicate the presence of drowsiness and involuntary sleep: brain electrical activity (electroencephalogram, EEG), eye movements via the electro-oculogram (EOG), wrist activity, head movements and galvanic skin resistance. The EEG and EOG identified sleepiness and sleep, as well as being potential measures on which to base an alarm system. Ten pilots either slept or showed evidence of sleepiness as assessed by the EEG and EOG. Many of the episodes of sleepiness lasted < 20 s, which could mean that the subjects were unaware of their occurrence and of the potential consequences on performance and vigilance. All physiological parameters showed changes during sleep, although only the EEG and EOG were modified by sleepiness. During sleep, skin resistance was increased, and wrist activity and head movements were absent for long periods. The study indicated that the measurement of eye movements (either alone or in combination with the EEG), wrist activity or head movement may be used as the basis of an alarm system to prevent involuntary sleep. Skin resistance is considered to be unsuitable, however, being related in a more general way to fatigue rather than to sleep episodes. The optimal way to monitor the onset of sleep would be to measure eye movements; however, this is not feasible in the flight deck environment at the present time due to the intrusive nature of the recording methodology. Wrist activity is therefore recommended as the basis of an alertness alarm. Such a device would alert the pilot after approximately 4-5 min of wrist inactivity, since this duration has been shown by the present study to be associated with sleep. The possibility that sleep inertia (reduced alertness immediately after awakening from sleep) could follow periods of sleep lasting 5 min needs to be considered. The findings reported here might be applicable to other occupational environments where fatigue and sleepiness are known to occur.