The effects of flight complexity on gaze entropy: An experimental study with fighter pilots

Development and validation of an assessment index for quantifying cognitive task load in pilots under simulated flight conditions using heart rate variability and principal component analysis

To investigate whether high cognitive task load (CTL) for aircraft pilots can be identified by analysing heart-rate variability, electrocardiograms were recorded while cadet pilots (n = 68) performed the plane tracking, anti-gravity pedalling, and reaction tasks during simulated flight missions. Data for standard electrocardiogram parameters were extracted from the R-R-interval series. In the research phase, low frequency power (LF), high frequency power (HF), normalised HF, and LF/HF differed significantly between high and low CTL conditions (p < .05 for all). A principal component analysis identified three components contributing 90.62% of cumulative heart-rate variance. These principal components were incorporated into a composite index. Validation in a separate group of cadet pilots (n = 139) under similar conditions showed that the index value significantly increased with increasing CTL (p < .05). The heart-rate variability index can be used to objectively identify high CTL flight conditions.Practitioner summary: We used principal component analysis of electrocardiogram data to construct a composite index for identifying high cognitive task load in pilots during simulated flight. We validated the index in a separate group of pilots under similar conditions. The index can be used to improve cadet training and flight safety.Abbreviations: ANOVA: a one-way analysis of variance; AP: anti-gravity pedaling task; CTL: cognitive task load; ECG: electrocardiograms; HR: heart rate; HRV: heart-rate variability; HRVI: heart-rate variability index; PT: plane-tracking task; RMSSD: root-mean square of differences between consecutive R-R intervals; RT: reaction task; SDNN: standard deviation of R-R intervals; HF: high frequency power; HFnu: normalized HF; LF: low frequency power; LFnu: normalized LF; PCA: principal component analysis.

Quantifying Workload and Stress in Intensive Care Unit Nurses: Preliminary Evaluation Using Continuous Eye-Tracking

OBJECTIVE

(1) To assess mental workloads of intensive care unit (ICU) nurses in 12-hour working shifts (days and nights) using eye movement data; (2) to explore the impact of stress on the ocular metrics of nurses performing patient care in the ICU.

BACKGROUND

Prior studies have employed workload scoring systems or accelerometer data to assess ICU nurses' workload. This is the first naturalistic attempt to explore nurses' mental workload using eye movement data.

METHODS

Tobii Pro Glasses 2 eye-tracking and Empatica E4 devices were used to collect eye movement and physiological data from 15 nurses during 12-hour shifts (252 observation hours). We used mixed-effect models and an ordinal regression model with a random effect to analyze the changes in eye movement metrics during high stress episodes.

RESULTS

While the cadence and characteristics of nurse workload can vary between day shift and night shift, no significant difference in eye movement values was detected. However, eye movement metrics showed that the initial handoff period of nursing shifts has a higher mental workload compared with other times. Analysis of ocular metrics showed that stress is positively associated with an increase in number of eye fixations and gaze entropy, but negatively correlated with the duration of saccades and pupil diameter.

CONCLUSION

Eye-tracking technology can be used to assess the temporal variation of stress and associated changes with mental workload in the ICU environment. A real-time system could be developed for monitoring stress and workload for intervention development.

How flight experience impacts pilots' decision-making and visual scanning pattern in low-visibility approaches: preliminary evidence from eye tracking

The visual approach is the most accident-prone phase of a flight, especially in low-visibility conditions. This preliminary study aimed to examine the effects of flight experience on pilots' decision-making and visual scanning pattern in low-visibility approaches. Twenty pilots were separated into two groups based on their flight experience and completed the high- and low-visibility approaches in balanced order using a high-fidelity flight simulator. Pilots' mental workload and visual scanning patterns were recorded via an eye tracker. The results showed that, compared to less flight-experienced pilots (20%, 3/15), experienced pilots (80%, 4/5) were more likely to make go-around decisions in the low-visibility approaches. Furthermore, they exhibited a more flexible and adaptable visual scanning pattern by quickly shifting their attention, as evidenced by decreased fixations and increased saccades. These findings suggest that the integration of visual scanning strategy and training solution with a marginally meteorological approach may enhance decision-making safety for novice pilots.

Assessing subjective workload for live captioners

Live captioning is a challenging task that requires intense concentration to convert audio to text in real-time. Despite the importance of live captioning for accessibility, little is known about the subjective workload of captioners in this context. This study aimed to measure the subjective workload of live captioners using the NASA-TLX and to explore the factors that contribute to their mental workload. Thirty live captioners participated in the study, completing questionnaires and interviews. Results showed that the subjective mental workload of live captioners is high, similar to that of neurosurgeons. The mental workload was found to be associated with caption paraphrasing and employment status. The challenges of the job, such as the speed of speaking in live content, cognitive tasks involved in paraphrasing, the concern about poor performance, the impact on audiences, and the lack of control over job scheduling contribute to this high workload. These findings suggest the need for modulating the scheduling of the workers, having longer breaks, and working in teams rather than independently. Introducing paradigm changes for live captioning workflow, such as reducing the human effort of typing by adopting auto-generated captions, so that captioners become decision-makers or managers of generational AI systems should also be considered. By addressing these issues, we can help improve the well-being of live captioners and the quality of captions, ultimately enhancing accessibility for all.

A Neurophysiological Evaluation of Cognitive Load during Augmented Reality Interactions in Various Industrial Maintenance and Assembly Tasks

Augmented reality (AR) has been shown to improve productivity in industry, but its adverse effects (e.g., headaches, eye strain, nausea, and mental workload) on users warrant further investigation. The objective of this study is to investigate the effects of different instruction methods (i.e., HoloLens AR-based and paper-based instructions) and task complexity (low and high-demanding tasks) on cognitive workloads and performance. Twenty-eight healthy males with a mean age of 32.12 (SD 2.45) years were recruited in this study and were randomly divided into two groups. The first group performed the experiment using AR-based instruction, and the second group used paper-based instruction. Performance was measured using total task time (TTT). The cognitive workload was measured using the power of electroencephalograph (EEG) features and the NASA task load index (NASA TLX). The results showed that using AR instructions resulted in a reduction in maintenance times and an increase in mental workload compared to paper instructions, particularly for the more demanding tasks. With AR instruction, 0.45% and 14.94% less time was spent on low- and high-demand tasks, respectively, as compared to paper instructions. According to the EEG features, employing AR to guide employees during highly demanding maintenance tasks increased information processing, which could be linked with an increased germane cognitive load. Increased germane cognitive load means participants can better facilitate long-term knowledge and skill acquisition. These results suggested that AR is superior and recommended for highly demanding maintenance tasks since it speeds up maintenance times and increases the possibility that information is stored in long-term memory and encrypted for recalls.

Using machine learning methods and EEG to discriminate aircraft pilot cognitive workload during flight

Pilots of aircraft face varying degrees of cognitive workload even during normal flight operations. Periods of low cognitive workload may be followed by periods of high cognitive workload and vice versa. During such changing demands, there exists potential for increased error on behalf of the pilots due to periods of boredom or excessive cognitive task demand. To further understand cognitive workload in aviation, the present study involved collection of electroencephalogram (EEG) data from ten (10) collegiate aviation students in a live-flight environment in a single-engine aircraft. Each pilot possessed a Federal Aviation Administration (FAA) commercial pilot certificate and either FAA class I or class II medical certificate. Each pilot flew a standardized flight profile representing an average instrument flight training sequence. For data analysis, we used four main sub-bands of the recorded EEG signals: delta, theta, alpha, and beta. Power spectral density (PSD) and log energy entropy of each sub-band across 20 electrodes were computed and subjected to two feature selection algorithms (recursive feature elimination (RFE) and lasso cross-validation (LassoCV), and a stacking ensemble machine learning algorithm composed of support vector machine, random forest, and logistic regression. Also, hyperparameter optimization and tenfold cross-validation were used to improve the model performance, reliability, and generalization. The feature selection step resulted in 15 features that can be considered an indicator of pilots' cognitive workload states. Then these features were applied to the stacking ensemble algorithm, and the highest results were achieved using the selected features by the RFE algorithm with an accuracy of 91.67% (± 0.11), a precision of 93.89% (± 0.09), recall of 91.67% (± 0.11), F-score of 91.22% (± 0.12), and the mean ROC-AUC of 0.93 (± 0.06). The achieved results indicated that the combination of PSD and log energy entropy, along with well-designed machine learning algorithms, suggest the potential for the use of EEG to discriminate periods of the low, medium, and high workload to augment aircraft system design, including flight automation features to improve aviation safety.

Influencing factors of novice pilot SA based on DEMATEL-AISM method: From pilots' view

Pilot situation awareness (SA) regulates flight safety, and inexperience may impair novice pilot reliability in SA. This study aims to determine the key influencing factors of novice pilot SA and to analyze the interrelationship and interaction mechanism of the factors. We investigated 55 novice pilots trained at aviation schools and identified the influencing factor index system by the Delphi survey. The method of Decision Making Trial and Evaluation (DEMATEL) combined with Adversarial Interpretive Structure Modeling (AISM) was adopted. The results show that: (1) The influencing factor index system includes 18 factors, divided into four categories: individual factors, team factors, task and human-machine system factors, and cockpit environment factors. (2) Team communication, team cooperation, basic cognitive ability, interface design, occupational age and experience, and authority gradient are the six crucial influencing factors. The former three have the greatest association with other factors, while the latter three are most likely to affect other factors. (3) Team communication, basic cognitive ability, and interface design are root-cause factors, of which team communication is the most fundamental. (4) The results of DEMATEL and AISM are consistent, both disclosing team communication as the fundamental factor with the highest priority, and cockpit environmental factors as the direct influencing factors but most susceptible to other factors. The present study can be viewed as a conducive attempt to extract vital influencing factors of novice pilot SA, and to provide ergonomic insights for determining the priorities to improve novice pilot SA in training and aircraft design for flight safety.

Intraocular pressure responses to a virtual reality shooting simulation in active-duty members of the Spanish Army: The influence of task complexity

Ocular physiology is sensitive to cognitively demanding tasks. However, it is unknown whether the intraocular pressure is also affected by the cognitive demands of military operations. The main objective was to determine the impact of a virtual reality shooting simulation with two levels of complexity on intraocular pressure levels in military personnel. Eighteen active-duty members of the Spanish Army and eighteen civilians performed two 4 min simulated shooting tasks with two levels of complexity using a virtual reality. In the "easy" task participants performed a simulated shoot when the stimulus (military with a rifle) appeared, while in the "difficult" task the stimulus randomly was a military with a rifle or with his hands on the air and participants were instructed to respond only when the military with a rifle appeared. Intraocular pressure was measured with a rebound tonometer before and immediately after each task. Complementarily, perceived levels of mental load and shooting performance (reaction time) were assessed. Intraocular pressure was greater after completing the more complex task in both military personnel (p-value < 0.01, Cohen´s d = 1.19) and civilians (p-value < 0.01, Cohen´s d = 1.16). Also, perceived levels of task load and reaction time were higher in the difficult compared to the easy shooting tasks (both p < 0.001). The rise in intraocular pressure is positively associated with the cognitive demands of simulated military operations. The potential application of this finding is the development of objective tools based on intraocular pressure for the evaluation of the mental state in real-world contexts, permitting to improve soldiers´ safety and performance.

Aviation and neurophysiology: A systematic review

This paper systematically reviews 20 years of publications (N = 54) on aviation and neurophysiology. The main goal is to provide an account of neurophysiological changes associated with flight training with the aim of identifying neurometrics indicative of pilot's flight training level and task relevant mental states, as well as to capture the current state-of-art of (neuro)ergonomic design and practice in flight training. We identified multiple candidate neurometrics of training progress and workload, such as frontal theta power, the EEG Engagement Index and the Cognitive Stability Index. Furthermore, we discovered that several types of classifiers could be used to accurately detect mental states, such as the detection of drowsiness and mental fatigue. The paper advances practical guidelines on terminology usage, simulator fidelity, and multimodality, as well as future research ideas including the potential of Virtual Reality flight simulations for training, and a brain-computer interface for flight training.

The Measurement of Cognitive Workload in Surgery Using Pupil Metrics: A Systematic Review and Narrative Analysis

INTRODUCTION

Increased cognitive workload (CWL) is a well-established entity that can impair surgical performance and increase the likelihood of surgical error. The use of pupil and gaze tracking data is increasingly being used to measure CWL objectively in surgery. The aim of this review is to summarize and synthesize the existing evidence that surrounds this.

METHODS

A systematic review was undertaken in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. A search of OVID MEDLINE, IEEE Xplore, Web of Science, Google Scholar, APA PsychINFO, and EMBASE was conducted for articles published in English between 1990 and January 2021. In total, 6791 articles were screened and 32 full-text articles were selected based on the inclusion criteria. A narrative analysis was undertaken in view of the heterogeneity of studies.

RESULTS

Seventy-eight percent of selected studies were deemed high quality. The most frequent surgical environment and task studied was surgical simulation (75%) and performance of laparoscopic skills (56%) respectively. The results demonstrated that the current literature can be broadly categorized into pupil, blink, and gaze metrics used in the assessment of CWL. These can be further categorized according to their use in the context of CWL: (1) direct measurement of CWL (n = 16), (2) determination of expertise level (n = 14), and (3) predictors of performance (n = 2).

CONCLUSIONS

Eye-tracking data provide a wealth of information; however, there is marked study heterogeneity. Pupil diameter and gaze entropy demonstrate promise in CWL assessment. Future work will entail the use of artificial intelligence in the form of deep learning and the use of a multisensor platform to accurately measure CWL.

Experimental Study on Panic during Simulated Fire Evacuation Using Psycho- and Physiological Metrics

Under circumstances of fire, panic usually brings uncertainty and unpredictability to evacuation. Therefore, a deep understanding of panic is desired. This study aims to dig into the underlying mechanism of fire evacuation panic by measuring and analysing psycho- and physiological indicators. In the experiment, participants watched a simulated train station within which three sets of stimuli were triggered separately. Eye movement and brain haemodynamic responses were collected during the watch, while questionnaires and interviews of emotions were conducted after. The analysed physiological indicators include the amplitude of pupil dilation, the time ratios of fixation and saccade, the binned entropy of gaze location, and the brain activation coefficients. The results of this research indicate that fire evacuation panic can be broken down into two elements. (1) Unawareness of situation: less knowledge of the situation leads to a higher level of panic; (2) Intensity of visual stimulation: the panic level is escalated with increased severity of fire that is perceived.

Effects of interface layout design on mobile learning efficiency: a comparison of interface layouts for mobile learning platform

Purpose

The purpose of this study is to explore the impact of mobile learning platforms on users' study efficiency and develop cognitive indicators to evaluate users' study efficiency on mobile learning platforms.

Design/methodology/approach

Layout style was the only independent factor that was investigated. A between-group experimental design was employed. Eye movement data were recorded during the experiment, following which participants were asked to complete an after-scenario questionnaire. This study evaluated the usability of the proposed new design using both subjective and objective data. The computer system usability questionnaire V3 (CSUQ) was used to measure subjective data. For the eye-tracking measure, gaze entropy, the proportion of fixation count and duration of each AOI were calculated. Gaze entropy reflects the complexity of information organization. Fixation counts and AOI duration represent the difficulty of information processing and attention distribution, respectively during the task.

Findings

The results indicated that interface layout presents significant effects on user's learning efficiency, usability and cognitive load. Sequential layout improved efficiency and satisfaction among participants and reduced information complexity. The results provided useful insights for designers whose goal is to improve user's learning efficiency under mobile learning scheme.

Originality/value

This study investigated the effects of interface layout on usability, user performance and cognitive load using subjective ratings and eye-tracking technology. Gaze entropy was used to measure the complexity of information organized by the interface design. Fixation count and duration proportion were used to identify the difficulty of information processing and distinguish users' distribution of cognitive resources. The results indicated that a vertical layout panel design was more efficient than a horizontal layout panel design. The design implications of the eye tracking indicators and research results were then summarized. This study is expected to encourage designers to optimize their design proposals using eye tracking testing.

The effect of mental schema evolution on mental workload measurement: an EEG study with simulated quadrotor UAV operation

OBJECTIVE

Mental workload is the result of the interactions between the demands of an operation task and the skills, behavior and perception of the performer. Working under a high mental workload can significantly affect an operator's ability to choose optimal decisions. However, the effect of mental schema, which reflects the level of expertise of an operator, on mental workload remains unclear. Here, we propose a theoretical framework for describing how the evolution of mental schema affects mental workload from the perspective of cognitive processing.

APPROACH

we recruited 51 students to participate in a 10-day simulated UAV flight training. The EEG PSD metrics were used to investigate the changes in neural responses caused by variations in the mental workload at different stages of mental schema evolution.

MAIN RESULTS

It was found that mental schema evolution influenced the direction and change trends of the frontal theta PSD, parietal alpha PSD, and central beta PSD. Initially, before the mental schema was formed, only the frontal theta PSD increased with increasing task difficulty; when the mental schema was initially being developed, the frontal theta PSD and the parietal alpha PSD decreased with increasing task difficulty, while the central beta PSD increased with increasing task difficulty. Finally, as the mental schema gradually matured, the trend of the three indicators did not change with increasing task difficulty. However, differences in the frontal PSD became more pronounced across task difficulty levels, while differences in the parietal PSD narrowed.

SIGNIFICANCE

Our results describe the relationship between the EEG power spectrum and the mental workload of UAV operators as the mental schema evolved. This suggests that EEG indicators can not only provide more accurate measurements of mental workload but also provide insights into the development of an operator's skill level.

Improving the tactical scanning of student pilots: A gaze-based training intervention for transition from visual flight into instrument meteorological conditions

Eye tracking has been applied to train novice drivers and clinicians; however, such applications in aviation are limited. This study develops a gaze-based intervention using video-based, expert commentary, and 3M (Mistake, Mitigation, Mastery) training to instruct visual flight rule student pilots on an instrument cross-check to mitigate the risk of losing aircraft control when they inadvertently enter instrument meteorological conditions (IMC). Twenty general aviation student pilots were randomized into control and experimental groups. Dwell time, return time, entropy, Kullback-Leibler divergence, and deviations from flight paths were compared before and after training to straight-and-level-flight (LF) and standard left level turn (LT) scenarios. After the training, the experimental pilots significantly increased dwell time on primary instruments (PIs), reduced randomness in visual search, and fixated on the PIs in shorter times (in the scenario of LT). In terms of piloting, all experimental pilots successfully kept the aircraft control while five control pilots lost control in IMC; significant differences in altitude and rate of climb deviations were observed between groups (in the scenario of LF).

Assessment of the mental fatigue state of an operator influenced by electromagnetic noise radiation

Objectives. The problem of determining the mental fatigue of the operator, which is processing the confidential information under the influence of electromagnetic noise radiation, is solved. The relevance of the problem is associated with the need to process the confidential information by the operator located in the limited space of a mobile technical system.

Methods. The methods of spectral analysis theory of the results, received in the process of experimental studies the electroencephalograms, are used.

Results. The results of experimental studies were obtained based on the processing and analysis of electroencephalograms recorded in standard leads Fp1, Fp2, F3, F4, C3, C4, P3, P4, O1, O2, F7, F8, T3, T4, T5, T6. The following quantitative parameters have been analyzed: power spectral density of delta-, theta-, alpha-, beta-, gamma-rhythms under the influence of electromagnetic noise radiation.

Conclusion. The presence of mental fatigue of the operator is based on a comparative analysis of the obtained results and the data described in the scientific literature related to the occurrence of chronic mental fatigue; the fatigue caused by the military performing simulation tasks; mental fatigue caused by performing an RVP test.

The Impact of Automation on Air Traffic Controller’s Behaviors

Air traffic controllers have to make quick decisions to keep air traffic safe. Their behaviors have a significant impact on the operation of the air traffic management (ATM) system. Automation tools have enhanced the ATM system’s capability by reducing the controller’s task-load. Much attention has been devoted to developing advanced automation in the last decade. However, less is known about the impact of automation on the behaviors of air traffic controllers. Here, we empirically tested the effects of three levels of automation—including manual, attention-guided, and automated—as well as varying traffic levels on eye movements, situation awareness and mental workload. The results showed that there are significant differences in the gaze and saccade behaviors between the attention-guided group and automated group. Traffic affected eye movements under the manual mode or under the attention-guided mode, but had no effect on eye movements under the automated mode. The results also supported the use of automation for enhancing situation awareness while reducing mental workload. Our work has potential implications for the design of automation and operation procedures.

Using eye-tracking to investigate the effects of pre-takeover visual engagement on situation awareness during automated driving

Automated driving systems are becoming increasingly prevalent throughout society. In conditionally automated vehicles, drivers may engage in non-driving-related tasks (NDRTs), which can negatively affect their situation awareness (SA) and preparedness to resume control of the vehicle, when necessary. Previous work has investigated engagement in NDRTs, but questions remain unanswered regarding its effect on drivers' SA during a takeover event. The objective of the current study is to use eye-tracking to aid in understanding how visual engagement in NDRTs affects changes in SA of the driving environment after a takeover request (TOR) has been issued. Thirty participants rode in a simulated SAE Level 3 automated driving environment and engaged in three separate pre-TOR tasks (Surrogate Reference Task, Monitoring Task, and Peripheral Detection Task) until presented with a TOR. Situation Awareness Global Assessment Technique (SAGAT) scores and gaze behavior were recorded during the post-TOR segment. Overall, longer times spent viewing the driving scene, and more dispersed visual attention allocation, were observed to be associated with better overall SA. Also, location-based eye tracking metrics show most promise in differentiating between task conditions with significantly different SAGAT scores. Findings from this work can inform the development of real-time SA assessment techniques using eye movements and ultimately contribute to improved operator roadway awareness for next-generation automated transportation.

Application of Eye Tracking Technology in Aviation, Maritime, and Construction Industries: A Systematic Review

Most accidents in the aviation, maritime, and construction industries are caused by human error, which can be traced back to impaired mental performance and attention failure. In 1596, Du Laurens, a French anatomist and medical scientist, said that the eyes are the windows of the mind. Eye tracking research dates back almost 150 years and it has been widely used in different fields for several purposes. Overall, eye tracking technologies provide the means to capture in real time a variety of eye movements that reflect different human cognitive, emotional, and physiological states, which can be used to gain a wider understanding of the human mind in different scenarios. This systematic literature review explored the different applications of eye tracking research in three high-risk industries, namely aviation, maritime, and construction. The results of this research uncovered the demographic distribution and applications of eye tracking research, as well as the different technologies that have been integrated to study the visual, cognitive, and attentional aspects of human mental performance. Moreover, different research gaps and potential future research directions were highlighted in relation to the usage of additional technologies to support, validate, and enhance eye tracking research to better understand human mental performance.

Measuring inter- and intra-individual differences in visual scan patterns in a driving simulator experiment using active information storage

Scan pattern analysis has been discussed as a promising tool in the context of real-time gaze-based applications. In particular, information-theoretic measures of scan path predictability, such as the gaze transition entropy (GTE), have been proposed for detecting relevant changes in user state or task demand. These measures model scan patterns as first-order Markov chains, assuming that only the location of the previous fixation is predictive of the next fixation in time. However, this assumption may not be sufficient in general, as recent research has shown that scan patterns may also exhibit more long-range temporal correlations. Thus, we here evaluate the active information storage (AIS) as a novel information-theoretic approach to quantifying scan path predictability in a dynamic task. In contrast to the GTE, the AIS provides means to statistically test and account for temporal correlations in scan path data beyond the previous last fixation. We compare AIS to GTE in a driving simulator experiment, in which participants drove in a highway scenario, where trials were defined based on an experimental manipulation that encouraged the driver to start an overtaking maneuver. Two levels of difficulty were realized by varying the time left to complete the task. We found that individual observers indeed showed temporal correlations beyond a single past fixation and that the length of the correlation varied between observers. No effect of task difficulty was observed on scan path predictability for either AIS or GTE, but we found a significant increase in predictability during overtaking. Importantly, for participants for which the first-order Markov chain assumption did not hold, this was only shown using AIS but not GTE. We conclude that accounting for longer time horizons in scan paths in a personalized fashion is beneficial for interpreting gaze pattern in dynamic tasks.

Visual scanning strategies in the cockpit are modulated by pilots' expertise: A flight simulator study

During a flight, pilots must rigorously monitor their flight instruments since it is one of the critical activities that contribute to update their situation awareness. The monitoring is cognitively demanding, but is necessary for timely intervention in the event of a parameter deviation. Many studies have shown that a large part of commercial aviation accidents involved poor cockpit monitoring from the crew. Research in eye-tracking has developed numerous metrics to examine visual strategies in fields such as art viewing, sports, chess, reading, aviation, and space. In this article, we propose to use both basic and advanced eye metrics to study visual information acquisition, gaze dispersion, and gaze patterning among novices and pilots. The experiment involved a group of sixteen certified professional pilots and a group of sixteen novice during a manual landing task scenario performed in a flight simulator. The two groups landed three times with different levels of difficulty (manipulated via a double task paradigm). Compared to novices, professional pilots had a higher perceptual efficiency (more numerous and shorter dwells), a better distribution of attention, an ambient mode of visual attention, and more complex and elaborate visual scanning patterns. We classified pilot's profiles (novices-experts) by machine learning based on Cosine KNN (K-Nearest Neighbors) using transition matrices. Several eye metrics were also sensitive to the landing difficulty. Our results can benefit the aviation domain by helping to assess the monitoring performance of the crews, improve initial and recurrent training and ultimately reduce incidents, and accidents due to human error.

Quantifying the Predictability of Visual Scanpaths Using Active Information Storage

Entropy-based measures are an important tool for studying human gaze behavior under various conditions. In particular, gaze transition entropy (GTE) is a popular method to quantify the predictability of a visual scanpath as the entropy of transitions between fixations and has been shown to correlate with changes in task demand or changes in observer state. Measuring scanpath predictability is thus a promising approach to identifying viewers' cognitive states in behavioral experiments or gaze-based applications. However, GTE does not account for temporal dependencies beyond two consecutive fixations and may thus underestimate the actual predictability of the current fixation given past gaze behavior. Instead, we propose to quantify scanpath predictability by estimating the active information storage (AIS), which can account for dependencies spanning multiple fixations. AIS is calculated as the mutual information between a processes' multivariate past state and its next value. It is thus able to measure how much information a sequence of past fixations provides about the next fixation, hence covering a longer temporal horizon. Applying the proposed approach, we were able to distinguish between induced observer states based on estimated AIS, providing first evidence that AIS may be used in the inference of user states to improve human-machine interaction.

Eye-head coordination and dynamic visual scanning as indicators of visuo-cognitive demands in driving simulator

Driving is an everyday task involving a complex interaction between visual and cognitive processes. As such, an increase in the cognitive and/or visual demands can lead to a mental overload which can be detrimental for driving safety. Compiling evidence suggest that eye and head movements are relevant indicators of visuo-cognitive demands and attention allocation. This study aims to investigate the effects of visual degradation on eye-head coordination as well as visual scanning behavior during a highly demanding task in a driving simulator. A total of 21 emmetropic participants (21 to 34 years old) performed dual-task driving in which they were asked to maintain a constant speed on a highway while completing a visual search and detection task on a navigation device. Participants did the experiment with optimal vision and with contact lenses that introduced a visual perturbation (myopic defocus). The results indicate modifications of eye-head coordination and the dynamics of visual scanning in response to the visual perturbation induced. More specifically, the head was more involved in horizontal gaze shifts when the visual needs were not met. Furthermore, the evaluation of visual scanning dynamics, based on time-based entropy which measures the complexity and randomness of scanpaths, revealed that eye and gaze movements became less explorative and more stereotyped when vision was not optimal. These results provide evidence for a reorganization of both eye and head movements in response to increasing visual-cognitive demands during a driving task. Altogether, these findings suggest that eye and head movements can provide relevant information about visuo-cognitive demands associated with complex tasks. Ultimately, eye-head coordination and visual scanning dynamics may be good candidates to estimate drivers' workload and better characterize risky driving behavior.

Operators' Load Monitoring and Management

Due to the introduction of highly automated vehicles and systems, the tasks of operators (drivers, pilots, air traffic controllers, production process managers) are in transition from "active control" to "passive monitoring" and "supervising". As a result of this transition, the roles of task load and workload are decreasing while the role of the mental load is increasing, thereby the new type of loads might be defined as information load and communication load. This paper deals with operators' load monitoring and management in highly automated systems. This research (i) introduces the changes in the role of operators and requirements in load management, (ii) defines the operators' models, (iii) describes the possible application of sensors and their integration into the working environment of operators, and (iv) develops the load observation and management concept. There are some examples of analyses of measurements and the concept of validation is discussed. This paper mainly deals with operators, particularly pilots and air traffic controllers (ATCOs).

EEG Theta Power Activity Reflects Workload among Army Combat Drivers: An Experimental Study

We aimed to evaluate the effects of mental workload variations, as a function of the road environment, on the brain activity of army drivers performing combat and non-combat scenarios in a light multirole vehicle dynamic simulator. Forty-one non-commissioned officers completed three standardized driving exercises with different terrain complexities (low, medium, and high) while we recorded their electroencephalographic (EEG) activity. We focused on variations in the theta EEG power spectrum, a well-known index of mental workload. We also assessed performance and subjective ratings of task load. The theta EEG power spectrum in the frontal, temporal, and occipital areas were higher during the most complex scenarios. Performance (number of engine stops) and subjective data supported these findings. Our findings strengthen previous results found in civilians on the relationship between driver mental workload and the theta EEG power spectrum. This suggests that EEG activity can give relevant insight into mental workload variations in an objective, unbiased fashion, even during real training and/or operations. The continuous monitoring of the warfighter not only allows instantaneous detection of over/underload but also might provide online feedback to the system (either automated equipment or the crew) to take countermeasures and prevent fatal errors.

Gaze entropy measures detect alcohol-induced driver impairment

Driving under the influence of alcohol is an ongoing cause of road traffic accidents. The biphasic nature of alcohol effects on subjective experience appears to contribute to the prevalence of drink-driving, as people perceive the declining phase of the BAC curve as recovery from intoxication and are more willing to drive despite significant impairments in objectively measured functions. The present study investigates whether alcohol-induced changes in gaze behaviour can be detected during engagement in a simulated driving task. In a repeated-measures and placebo-controlled design, this study examines the biphasic influence of moderate alcohol intake (0.6 g/kg) on measures of gaze behaviour and simulated driving performance. Twenty-two healthy young adults completed three driving sessions (baseline, ascending and descending) under two conditions (placebo, alcohol) while their eye movements were simultaneously recorded. The results revealed that gaze behaviour as measured by gaze transition entropy (GTE) and stationary gaze entropy (SGE) and driving performance measured by the standard deviation of lateral position (SDLP) of the vehicle, were significantly affected by alcohol across the ascending and descending sessions. The alcohol-induced reduction in GTE with an increase in SGE is discussed as alcohol's impact on top-down modulation of gaze resulting in more dispersed and erratic pattern of visual scanning. The observed changes in gaze behaviour also mediated the influence of alcohol upon driving performance. These results have significant implications for the development of driver monitoring systems that can detect alcohol-induced impairment.

Nasal skin temperature reveals changes in arousal levels due to time on task: An experimental thermal infrared imaging study

Infrared thermography, thanks to technological developments and lowering prices, is now getting considerable attention as a potential arousal monitor in the safety industry. Nasal skin temperature might be a valid index to track physiological variations due to reduced arousal levels, and its use could prevent a drowsiness-related deterioration of performance. However, the few studies that have investigated nasal skin temperature in applied settings have had inconsistent results. Here, we assessed the validity of nasal skin temperature to monitor changes in arousal levels (from alertness to drowsiness). The participants performed a 2-h simulated driving task while we simultaneously recorded their nasal skin temperature, brain activity (we used frontal delta electroencephalographic [EEG] activity as the reference index of alertness), and driving performance (speeding time). For those variables, we calculated growth curve models. We also collected subjective ratings of alertness and fatigue before and after the driving session. We found that the nasal skin temperature showed a cubic trajectory (it increased for the first 75 min, and then it began to decrease, but such deceleration gradually diminished over time). As expected, frontal delta EEG activity showed an inverted U-shaped quadratic trend (EEG power increased for the first hour and half, and gradually decreased during the last thirty minutes). The speeding time exhibited a similar pattern of change. Subjective sleepiness and fatigue increased after the task. Overall, our results suggest that nasal skin temperature seems to be a valid measure of arousal variations while performing a complex and dynamic everyday task.

Towards a Multimodal Model of Cognitive Workload Through Synchronous Optical Brain Imaging and Eye Tracking Measures

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field.

Validation of Electroencephalographic Recordings Obtained with a Consumer-Grade, Single Dry Electrode, Low-Cost Device: A Comparative Study

The functional validity of the signal obtained with low-cost electroencephalography (EEG) devices is still under debate. Here, we have conducted an in-depth comparison of the EEG-recordings obtained with a medical-grade golden-cup electrodes ambulatory device, the SOMNOwatch + EEG-6, vs those obtained with a consumer-grade, single dry electrode low-cost device, the NeuroSky MindWave, one of the most affordable devices currently available. We recorded EEG signals at Fp1 using the two different devices simultaneously on 21 participants who underwent two experimental phases: a 12-minute resting state task (alternating two cycles of closed/open eyes periods), followed by 60-minute virtual-driving task. We evaluated the EEG recording quality by comparing the similarity between the temporal data series, their spectra, their signal-to-noise ratio, the reliability of EEG measurements (comparing the closed eyes periods), as well as their blink detection rate. We found substantial agreement between signals: whereas, qualitatively, the NeuroSky MindWave presented higher levels of noise and a biphasic shape of blinks, the similarity metric indicated that signals from both recording devices were significantly correlated. While the NeuroSky MindWave was less reliable, both devices had a similar blink detection rate. Overall, the NeuroSky MindWave is noise-limited, but provides stable recordings even through long periods of time. Furthermore, its data would be of adequate quality compared to that of conventional wet electrode EEG devices, except for a potential calibration error and spectral differences at low frequencies.