Driver sleepiness detection with deep neural networks using electrophysiological data

OBJECTIVE

The objective of this paper is to present a driver sleepiness detection model based on electrophysiological data and a neural network consisting of Convolutional Neural Networks and a Long Short Term Memory architecture.

APPROACH

The model was developed and evaluated on data from 12 different experiments with 269 drivers and 1187 driving sessions during daytime (low sleepiness condition) and night-time (high sleepiness condition), collected during naturalistic driving conditions on real roads in Sweden or in an advanced moving-base driving simulator. Electrooculographic and electroencephalographic time series data, split up in 16634 2.5-minute data segments was used as input to the deep neural network. This probably constitutes the largest labelled driver sleepiness dataset in the world. The model outputs a binary decision as alert (defined as ≤6 on the Karolinska Sleepiness Scale, KSS) or sleepy (KSS≥8) or a regression output corresponding to KSS ϵ [1-5,6,7,8,9].

MAIN RESULTS

The subject-independent mean absolute error (MAE) was 0.78. Binary classification accuracy for the regression model was 82.6% as compared to 82.0% for a model that was trained specifically for the binary classification task. Data from the eyes were more informative than data from the brain. A combined input improved performance for some models, but the gain was very limited.

SIGNIFICANCE

Improved classification results were achieved with the regression model compared to the classification model. This suggests that the implicit order of the KSS ratings, i.e. the progression from alert to sleepy, provides important information for robust modelling of driver sleepiness, and that class labels should not simply be aggregated into an alert and a sleepy class. Furthermore, the model consistently showed better results than a model trained on manually extracted features based on expert knowledge, indicating that the model can detect sleepiness that is not covered by traditional algorithms.

Heart Rate Variability for Driver Sleepiness Classification in Real Road Driving Conditions

Approximately 20-30% of all road fatalities are related to driver sleepiness. A long-lasting goal in driver state research has therefore been to develop a robust sleepiness detection system. Since the alertness level is reflected in autonomous nervous system activity, it has been suggested that various heart rate variability (HRV) metrics can be used as features for driver sleepiness classification. Since the heart rate is modulated by many different factors, and not just by sleepiness, it is relevant to question the high driver sleepiness classification accuracies that have occasionally been presented in the literature. The main objective of this paper is thus to test how well a sleepiness classification system based on HRV features really is. A unique data set with 86 drivers, obtained while driving on real roads in real traffic, both in alert and sleep deprived conditions, was used to train and test a support vector machine (SVM) classifier. Subjective ratings based on the Karolinska sleepiness scale (KSS) was used as ground truth to divide the data into three classes (alert, somewhat sleepy and severely sleepy). Even though nearly all the 24 investigated HRV metrics showed significant differences between sleepiness levels, the SVM results only reached a mean accuracy of 61 %, with the worst results originating from the severely sleepy cases. In summary, the high classification performance that may arise in studies with high experimental control could not be replicated under realistic driving conditions. Future works should focus on how various confounding factors should be accounted for when using HRV based metrics as input to a driver sleepiness detection system.

Evaluation of methods for the assessment of attention while driving

The ability to assess the current attentional state of the driver is important for many aspects of driving, not least in the field of partial automation for transfer of control between vehicle and driver. Knowledge about the driver's attentional state is also necessary for the assessment of the effects of additional tasks on attention. The objective of this paper is to evaluate different methods that can be used to assess attention, first theoretically, and then empirically in a controlled field study and in the laboratory. Six driving instructors participated in all experimental conditions of the study, delivering within-subjects data for all tested methods. Additional participants were recruited for some of the conditions. The test route consisted of 14km of motorway with low to moderate traffic, which was driven three times per participant per condition. The on-road conditions were: baseline, driving with eye tracking and self-paced visual occlusion, and driving while thinking aloud. The laboratory conditions were: Describing how attention should be distributed on a motorway, and thinking aloud while watching a video from the baseline drive. The results show that visual occlusion, especially in combination with eye tracking, was appropriate for assessing spare capacity. The think aloud protocol was appropriate to gain insight about the driver's actual mental representation of the situation at hand. Expert judgement in the laboratory was not reliable for the assessment of drivers' attentional distribution in traffic. Across all assessment techniques, it is evident that meaningful assessment of attention in a dynamic traffic situation can only be achieved when the infrastructure layout, surrounding road users, and intended manoeuvres are taken into account. This requires advanced instrumentation of the vehicle, and subsequent data reduction, analysis and interpretation are demanding. In conclusion, driver attention assessment in real traffic is a complex task, but a combination of visual occlusion, eye tracking and thinking aloud is a promising combination of methods to come further on the way.

Creating a framework for the prioritization of biosecurity risks to the New Zealand dairy industry

The New Zealand dairy sector relies on robust biosecurity measures to control and mitigate a wide range of threats to the industry. To optimize the prioritization of organisms and manage the risk they pose to the sector in a transparent and credible way, the Dairy Biosecurity Risk Evaluation Framework (D-BRiEF) was developed. This comprehensive framework was specifically designed for decision support, using a standardized approach to address the full spectrum of biosecurity threats to the sector, including exotic and endemic animal disease organisms, pest plants and insects. D-BRiEF is underpinned by three main processes, namely (i) hazard identification; (ii) multicriteria risk assessment; and (iii) communication for risk management. Expert knowledge and empirical data, including associated uncertainty, are harnessed in a standardized format. Results feed into a probability-impact model that was developed in close collaboration with dairy sector economists to provide overall comparative 10-year quantitative economic impact estimates for each assessed risk organism. A description of the overarching framework, which applies to diverse organism groups, is presented with detailed methodology on both endemic and exotic animal disease risk organisms. Examples of visual outputs are included, although actual ranking results are not reported due to industry confidentiality. D-BRiEF can provide a decision advantage to DairyNZ biosecurity risk managers and sector stakeholders by creating a transparent process that can be interrogated and updated at multiple levels to fully understand the layers of risk posed by different organisms.

Automated EEG Artifact Handling With Application in Driver Monitoring

Automated analyses of electroencephalographic (EEG) signals acquired in naturalistic environments are becoming increasingly important in areas such as brain-computer interfaces and behavior science. However, the recorded EEG in such environments is often heavily contaminated by motion artifacts and eye movements. This poses new requirements on artifact handling. The objective of this paper is to present an automated EEG artifacts handling algorithm, which will be used as a preprocessing step in a driver monitoring application. The algorithm, named Automated aRTifacts handling in EEG (ARTE), is based on wavelets, independent component analysis, and hierarchical clustering. The algorithm is tested on a dataset obtained from a driver sleepiness study including 30 drivers and 540 30-min 30-channel EEG recordings. The algorithm is evaluated by a clinical neurophysiologist, by quantitative criteria (signal quality index, mean square error, relative error, and mean absolute error), and by demonstrating its usefulness as a preprocessing step in driver monitoring, here exemplified with driver sleepiness classification. All results are compared with a state-of-the-art algorithm called FORCe. The quantitative and expert evaluation results show that the two algorithms are comparable, and that both algorithms significantly reduce the impact of artifacts in recorded EEG signals. When artifact handling is used as a preprocessing step in driver sleepiness classification, the classification accuracy increased by 5% when using ARTE and by 2% when using FORCe. The advantage with ARTE is that it is data driven and does not rely on additional reference signals or manually defined thresholds, making it well suited for use in dynamic settings where unforeseen and rare artifacts are commonly encountered.

Minimum Required Attention: A Human-Centered Approach to Driver Inattention

OBJECTIVE

To propose a driver attention theory based on the notion of driving as a satisficing and partially self-paced task and, within this framework, present a definition for driver inattention.

BACKGROUND

Many definitions of driver inattention and distraction have been proposed, but they are difficult to operationalize, and they are either unreasonably strict and inflexible or suffer from hindsight bias.

METHOD

Existing definitions of driver distraction are reviewed and their shortcomings identified. We then present the minimum required attention (MiRA) theory to overcome these shortcomings. Suggestions on how to operationalize MiRA are also presented.

RESULTS

MiRA describes which role the attention of the driver plays in the shared "situation awareness of the traffic system." A driver is considered attentive when sampling sufficient information to meet the demands of the system, namely, that he or she fulfills the preconditions to be able to form and maintain a good enough mental representation of the situation. A driver should only be considered inattentive when information sampling is not sufficient, regardless of whether the driver is concurrently executing an additional task or not.

CONCLUSIONS

The MiRA theory builds on well-established driver attention theories. It goes beyond available driver distraction definitions by first defining what a driver needs to be attentive to, being free from hindsight bias, and allowing the driver to adapt to the current demands of the traffic situation through satisficing and self-pacing. MiRA has the potential to provide the stepping stone for unbiased and operationalizable inattention detection and classification.

Local changes in the wake electroencephalogram precedes lane departures

The objective of this exploratory study is to investigate if lane departures are associated with local sleep, measured via source-localized electroencephalography (EEG) theta power in the 5-9 Hz frequency range. Thirty participants drove in an advanced driving simulator, resulting in 135 lane departures at high levels of self-reported sleepiness. These lane departures were compared to matching non-departures at the same sleepiness level within the same individual. There was no correspondence between lane departures and global theta activity. However, at the local level an increased risk for lane departures was associated with increased theta content in brain regions related to motor function.

Inferring source attribution from a multiyear multisource data set of Salmonella in Minnesota

Salmonella enterica is a global health concern because of its widespread association with foodborne illness. Bayesian models have been developed to attribute the burden of human salmonellosis to specific sources with the ultimate objective of prioritizing intervention strategies. Important considerations of source attribution models include the evaluation of the quality of input data, assessment of whether attribution results logically reflect the data trends and identification of patterns within the data that might explain the detailed contribution of different sources to the disease burden. Here, more than 12,000 non-typhoidal Salmonella isolates from human, bovine, porcine, chicken and turkey sources that originated in Minnesota were analysed. A modified Bayesian source attribution model (available in a dedicated R package), accounting for non-sampled sources of infection, attributed 4,672 human cases to sources assessed here. Most (60%) cases were attributed to chicken, although there was a spike in cases attributed to a non-sampled source in the second half of the study period. Molecular epidemiological analysis methods were used to supplement risk modelling, and a visual attribution application was developed to facilitate data exploration and comprehension of the large multiyear data set assessed here. A large amount of within-source diversity and low similarity between sources was observed, and visual exploration of data provided clues into variations driving the attribution modelling results. Results from this pillared approach provided first attribution estimates for Salmonella in Minnesota and offer an understanding of current data gaps as well as key pathogen population features, such as serotype frequency, similarity and diversity across the sources. Results here will be used to inform policy and management strategies ultimately intended to prevent and control Salmonella infection in the state.

Changes in glance behaviour when using a visual eco-driving system - A field study

While in-vehicle eco-driving support systems have the potential to reduce greenhouse gas emissions and save fuel, they may also distract drivers, especially if the system makes use of a visual interface. The objective of this study is to investigate the visual behaviour of drivers interacting with such a system, implemented on a five-inch screen mounted above the middle console. Ten drivers participated in a real-world, on-road driving study where they drove a route nine times (2 pre-baseline drives, 5 treatment drives, 2 post-baseline drives). The route was 96 km long and consisted of rural roads, urban roads and a dual-lane motorway. The results show that drivers look at the system for 5-8% of the time, depending on road type, with a glance duration of about 0.6 s, and with 0.05% long glances (>2s) per kilometre. These figures are comparable to what was found for glances to the speedometer in this study. Glance behaviour away from the windscreen is slightly increased in treatment as compared to pre- and post-baseline, mirror glances decreased in treatment and post-baseline compared to pre-baseline, and speedometer glances increased compared to pre-baseline. The eco-driving support system provided continuous information interspersed with additional advice pop-ups (announced by a beep) and feedback pop-ups (no auditory cue). About 20% of sound initiated advice pop-ups were disregarded, and the remaining cases were usually looked at within the first two seconds. About 40% of the feedback pop-ups were disregarded. The amount of glances to the system immediately before the onset of a pop-up was clearly higher for feedback than for advice. All in all, the eco-driving support system under investigation is not likely to have a strong negative impact on glance behaviour. However, there is room for improvements. We recommend that eco-driving information is integrated with the speedometer, that optional activation of sound alerts for intermittent information is made available, and that the pop-up duration should be extended to facilitate self-regulation of information intake.

Effects of electronic billboards on driver distraction

OBJECTIVE

There is an increase in electronic advertising billboards along major roads, which may cause driver distraction due to the highly conspicuous design of the electronic billboards. Yet limited research on the impact of electronic billboards on driving performance and driver behavior is available. The Swedish Transport Administration recently approved the installation of 12 electronic billboards for a trial period along a 3-lane motorway with heavy traffic running through central Stockholm, Sweden. The aim of this study was to evaluate the effect of these electronic billboards on visual behavior and driving performance.

METHOD

A total of 41 drivers were recruited to drive an instrumented vehicle passing 4 of the electronic billboards during day and night conditions. A driver was considered visually distracted when looking at a billboard continuously for more than 2 s or if the driver looked away from the road for a high percentage of time. Dependent variables were eye-tracking measures and driving performance measures.

RESULTS

The visual behavior data showed that drivers had a significantly longer dwell time, a greater number of fixations, and longer maximum fixation duration when driving past an electronic billboard compared to other signs on the same road stretches. No differences were found for the factors day/night, and no effect was found for the driving behavior data.

CONCLUSION

Electronic billboards have an effect on gaze behavior by attracting more and longer glances than regular traffic signs. Whether the electronic billboards attract too much attention and constitute a traffic safety hazard cannot be answered conclusively based on the present data.

The impact of tunnel design and lighting on the performance of attentive and visually distracted drivers

The crash risk in tunnels is lower than on the open road network, but the consequences of a crash are often severe. Proper tunnel design is one measure to reduce the likelihood of crashes, and the objective of this work is to investigate how driving performance is influenced by design factors, and whether there is an interaction with secondary task load. Twenty-eight drivers participated in the simulator study. A full factorial within subject design was used to investigate the tunnel wall colour (dark or light-coloured walls), illumination (three different levels) and task load (with or without a visual secondary task). The results show that tunnel design and illumination have some influence on the drivers' behaviour, but visual attention given to the driving task is the most crucial factor, giving rise to significant changes in both driving behaviour and visual behaviour. The results also indicate that light-coloured tunnel walls are more important than strong illumination to keep the drivers' visual attention focused forward.

Sources of Klebsiella and Raoultella species on dairy farms: be careful where you walk

Klebsiella spp. are a common cause of mastitis, milk loss, and culling on dairy farms. Control of Klebsiella mastitis is largely based on prevention of exposure of the udder to the pathogen. To identify critical control points for mastitis prevention, potential Klebsiella sources and transmission cycles in the farm environment were investigated, including oro-fecal transmission, transmission via the indoor environment, and transmission via the outdoor environment. A total of 305 samples was collected from 3 dairy farms in upstate New York in the summer of 2007, and included soil, feed crops, feed, water, rumen content, feces, bedding, and manure from alleyways and holding pens. Klebsiella spp. were detected in 100% of rumen samples, 89% of water samples, and approximately 64% of soil, feces, bedding, alleyway, and holding pen samples. Detection of Klebsiella spp. in feed crops and feed was less common. Genotypic identification of species using rpoB sequence data showed that Klebsiella pneumoniae was the most common species in rumen content, feces, and alleyways, whereas Klebsiella oxytoca, Klebsiella variicola, and Raoultella planticola were the most frequent species among isolates from soil and feed crops. Random amplified polymorphic DNA-based strain typing showed heterogeneity of Klebsiella spp. in rumen content and feces, with a median of 4 strains per 5 isolates. Observational and bacteriological data support the existence of an oro-fecal transmission cycle, which is primarily maintained through direct contact with fecal contamination or through ingestion of contaminated drinking water. Fecal shedding of Klebsiella spp. contributes to pathogen loads in the environment, including bedding, alleyways, and holding pens. Hygiene of alleyways and holding pens is an important component of Klebsiella control on dairy farms.

Predicting visual distraction using driving performance data

Behavioral variables are often used as performance indicators (PIs) of visual or internal distraction induced by secondary tasks. The objective of this study is to investigate whether visual distraction can be predicted by driving performance PIs in a naturalistic setting. Visual distraction is here defined by a gaze based real-time distraction detection algorithm called AttenD. Seven drivers used an instrumented vehicle for one month each in a small scale field operational test. For each of the visual distraction events detected by AttenD, seven PIs such as steering wheel reversal rate and throttle hold were calculated. Corresponding data were also calculated for time periods during which the drivers were classified as attentive. For each PI, means between distracted and attentive states were calculated using t-tests for different time-window sizes (2 - 40 s), and the window width with the smallest resulting p-value was selected as optimal. Based on the optimized PIs, logistic regression was used to predict whether the drivers were attentive or distracted. The logistic regression resulted in predictions which were 76 % correct (sensitivity = 77 % and specificity = 76 %). The conclusion is that there is a relationship between behavioral variables and visual distraction, but the relationship is not strong enough to accurately predict visual driver distraction. Instead, behavioral PIs are probably best suited as complementary to eye tracking based algorithms in order to make them more accurate and robust.

Heart sounds are altered by open cardiac surgery

BACKGROUND AND OBJECTIVE

Patients have reported that they perceive their own heart sounds differently after open cardiac surgery than before the surgery. The present study was designed to investigate whether changes in heart sounds can be quantitatively measured.

METHOD

Heart sounds were recorded from 57 patients undergoing coronary artery bypass graft (CABG) surgery and from a control group of 10 subjects. The so-called Hjorth descriptors and the main frequency peak were compared before and after surgery to determine whether the characteristics of the heart sounds had changed.

RESULTS

At a group level, the first heart sound was found to be significantly different after CABG surgery. Generally, the heart sounds shifted toward a lower frequency after surgery in the CABG group. No significant changes were found in the control group.

CONCLUSIONS

Heart sounds are altered after CABG surgery. The changes are objectively quantifiable and may also be subjectively perceived by the patients.

Assessment of suspected aortic stenosis by auto mutual information analysis of murmurs

Mild sclerotic thickening of the aortic valve affects 25% of the population, and the condition causes aortic valve stenosis (AS) in 2% of adults above 65 years. Echocardiography is today the clinical standard for assessing AS. However, a cost effective and uncomplicated technique that can be used for decision support in the primary health care would be of great value. In this study, recorded phonocardiographic signals were analyzed using the first local minimum of the auto mutual information (AMI) function. The AMI method measures the complexity in the sound signal, which is related to the amount of turbulence in the blood flow and thus to the severity of the stenosis. Two previously developed phonocardiographic methods for assessing AS severity were used for comparison, the murmur energy ratio and the sound spectral averaging technique. Twenty-nine patients with suspected AS were examined with Doppler echocardiography. The aortic jet velocity was used as a reference of AS severity, and it was found to correlate with the AMI method, the murmur energy ratio and the sound spectral averaging technique with the correlation coefficient R = 0.82, R = 0.73 and R = 0.76, respectively.

A respiration monitor based on electrocardiographic and photoplethysmographic sensor fusion

Respiratory variations are present in the pulse wave transit time (PTT) and, as a consequence of respiratory sinus arrhythmia, in the electrocardiogram (ECG) and the photoplethysmogram (PPG). The aim of this study was to investigate these variations in healthy subjects during rest and invoked blood pressure changes. The primary goal was to develop a non-invasive respiration monitor. The error rates for breath detection during rest were 14%, 11% and 10% for PTT, ECG and PPG respectively. Significantly higher error rates were found in hypotension and hypertension. To improve accuracy and robustness, the signals were merged in a neural network resulting in an error rate of 9%.

Feature extraction for systolic heart murmur classification

Heart murmurs are often the first signs of pathological changes of the heart valves, and they are usually found during auscultation in the primary health care. Distinguishing a pathological murmur from a physiological murmur is however difficult, why an "intelligent stethoscope" with decision support abilities would be of great value. Phonocardiographic signals were acquired from 36 patients with aortic valve stenosis, mitral insufficiency or physiological murmurs, and the data were analyzed with the aim to find a suitable feature subset for automatic classification of heart murmurs. Techniques such as Shannon energy, wavelets, fractal dimensions and recurrence quantification analysis were used to extract 207 features. 157 of these features have not previously been used in heart murmur classification. A multi-domain subset consisting of 14, both old and new, features was derived using Pudil's sequential floating forward selection (SFFS) method. This subset was compared with several single domain feature sets. Using neural network classification, the selected multi-domain subset gave the best results; 86% correct classifications compared to 68% for the first runner-up. In conclusion, the derived feature set was superior to the comparative sets, and seems rather robust to noisy data.

Pulse wave transit time for monitoring respiration rate

In this study, we investigate the beat-to-beat respiratory fluctuations in pulse wave transit time (PTT) and its subcomponents, the cardiac pre-ejection period (PEP) and the vessel transit time (VTT) in ten healthy subjects. The three transit times were found to fluctuate in pace with respiration. When applying a simple breath detecting algorithm, 88% of the breaths seen in a respiration air-flow reference could be detected correctly in PTT. Corresponding numbers for PEP and VTT were 76 and 81%, respectively. The performance during hypo- and hypertension was investigated by invoking blood pressure changes. In these situations, the error rates in breath detection were significantly higher. PTT can be derived from signals already present in most standard monitoring set-ups. The transit time technology thus has prospects to become an interesting alternative for respiration rate monitoring.

Noninvasive investigation of blood pressure changes using the pulse wave transit time: a novel approach in the monitoring of hemodialysis patients

Severe blood pressure changes are well known in hemodialysis. Detection and prediction of these are important for the well-being of the patient and for optimizing treatment. New noninvasive methods for this purpose are required. The pulse wave transit time technique is an indirect estimation of blood pressure, and our intention is to investigate whether this technique is applicable for hemodialysis treatment. A measurement setup utilizing lower body negative pressure and isometric contraction was used to simulate dialysis-related blood pressure changes in normal test subjects. Systolic blood pressure levels were compared to different pulse wave transit times, including and excluding the cardiac preejection period. Based on the results of these investigations, a pulse wave transit time technique adapted for dialysis treatment was developed and tried out on patients. To determine systolic blood pressure in the normal group, the total pulse wave transit time was found most suitable (including the cardiac preejection period). Correlation coefficients were r = 0.80 +/- 0.06 (mean +/- SD) overall and r = 0.81 +/- 0.16 and r = 0.09 +/- 0.62 for the hypotension and hypertension phases, respectively. When applying the adapted technique in dialysis patients, large blood pressure variations could easily be detected when present. Pulse wave transit time is correlated to systolic blood pressure within the acceptable range for a trend-indicating system. The method's applicability for dialysis treatment requires further studies. The results indicate that large sudden pressure drops, like those seen in sudden hypovolemia, can be detected.