Improving pilot mental workload evaluation with combined measures

Evaluation of physiological responses to mental workload in n-back and arithmetic tasks

Working memory tasks, such as n-back and arithmetic tasks, are frequently used in studying mental workload. The present study investigated and compared the sensitivity of several physiological measures at three levels of difficulty of n-back and arithmetic tasks. The results showed significant differences in fixation duration and pupil diameter among three task difficulty levels for both n-back and arithmetic tasks. Pupil diameters increase with increasing mental workload, whereas fixation duration decreases. Blink duration and heart rate (HR) were significantly increased as task difficulty increased in the n-back task, while root mean square of successive differences (RMSSD) and standard deviation of R-R intervals (SDNN) were significantly decreased in the arithmetic task. On the other hand, blink rate and Galvanic Skin Response (GSR) were not sensitive enough to assess the differences in task difficulty for both tasks. All significant physiological measures yielded significant differences between low and high task difficulty except for SDNN.Practitioner summary: This study aimed to assess the sensitivity levels of several physiological measures of mental workload in n-back and arithmetic tasks. It showed that pupil diameter was the most sensitive in both tasks. This study also found that most physiological indices are sensitive to an extreme change in task difficulty levels.

Fatigue, Work Overload, and Sleepiness in a Sample of Spanish Commercial Airline Pilots

Commercial aviation pilots are an occupational group that work in particular conditions, with frequent schedule changes, shift work, unfavorable environmental conditions, etc. These circumstances can lead to fatigue, work overload (WO), and daytime sleepiness, factors that can affect their health and safety. This study aimed to assess the prevalence and the association between these parameters in a sample of Spanish commercial airline pilots. The Raw TLX, Fatigue Severity Scale, and the Epworth Sleepiness Scale questionnaires were administered in a sample of 283 participants. The relationships of the total scores between all the questionnaires were studied by the chi-square test and the risk scores (odds ratio) were calculated. Different models using multiple linear regression were carried out to evaluate the effects of WO, fatigue, and daytime sleepiness, among the total scores, age, and flight hours. Additionally, the internal consistency of each questionnaire was estimated. A total of 28.2% presented WO above the 75th percentile, with mental and temporal demand the dimensions with the greatest weight. A total of 18% of pilots presented fatigue, 15.8% moderate sleepiness, and 3.9% severe sleepiness. We observed an association among WO, fatigue, and daytime sleepiness, important factors related to pilot health and aviation safety.

Stress and Workload Assessment in Aviation-A Narrative Review

In aviation, any detail can have massive consequences. Among the potential sources of failure, human error is still the most troublesome to handle. Therefore, research concerning the management of mental workload, attention, and stress is of special interest in aviation. Recognizing conditions in which a pilot is over-challenged or cannot act lucidly could avoid serious outcomes. Furthermore, knowing in depth a pilot's neurophysiological and cognitive-behavioral responses could allow for the optimization of equipment and procedures to minimize risk and increase safety. In addition, it could translate into a general enhancement of both the physical and mental well-being of pilots, producing a healthier and more ergonomic work environment. This review brings together literature on the study of stress and workload in the specific case of pilots of both civil and military aircraft. The most common approaches for studying these phenomena in the avionic context are explored in this review, with a focus on objective methodologies (e.g., the collection and analysis of neurophysiological signals). This review aims to identify the pros, cons, and applicability of the various approaches, to enable the design of an optimal protocol for a comprehensive study of these issues.

Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics

With the development of the autopilot system, the main task of a pilot has changed from controlling the aircraft to supervising the autopilot system and making critical decisions. Therefore, the human–machine interaction system needs to be improved accordingly. A key step to improving the human–machine interaction system is to improve its understanding of the pilots’ status, including fatigue, stress, workload, etc. Monitoring pilots’ status can effectively prevent human error and achieve optimal human–machine collaboration. As such, there is a need to recognize pilots’ status and predict the behaviors responsible for changes of state. For this purpose, in this study, 14 Air Force cadets fly in an F-35 Lightning II Joint Strike Fighter simulator through a series of maneuvers involving takeoff, level flight, turn and hover, roll, somersault, and stall. Electro cardio (ECG), myoelectricity (EMG), galvanic skin response (GSR), respiration (RESP), and skin temperature (SKT) measurements are derived through wearable physiological data collection devices. Physiological indicators influenced by the pilot’s behavioral status are objectively analyzed. Multi-modality fusion technology (MTF) is adopted to fuse these data in the feature layer. Additionally, four classifiers are integrated to identify pilots’ behaviors in the strategy layer. The results indicate that MTF can help to recognize pilot behavior in a more comprehensive and precise way.

Evaluating mental workload during multitasking in simulated flight

BACKGROUND

Pilots must process multiple streams of information simultaneously. Mental workload is one of the main issues in man-machine interactive mode when dealing with multiple tasks. This study aimed to combine functional near-infrared spectroscopy (fNIRS) and electrocardiogram (ECG) to detect changes in mental workload during multitasking in a simulated flight.

METHODS

Twenty-six participants performed three multitasking tasks at different mental workload levels. These mental workload levels were set by varying the number of subtasks. fNIRS and ECG signals were recorded during tasks. Participants filled in the national aeronautics and space administration task load index (NASA-TLX) scale after each task. The effects of mental workload on scores of NASA-TLX, performance of tasks, heart rate (HR), heart rate variability (HRV), and the prefrontal cortex (PFC) activation were analyzed.

RESULTS

Compared to multitasking in lower mental workload conditions, participants exhibited higher scores of NASA-TLX, HR, and PFC activation when multitasking in high mental workload conditions. Their performance was worse during the high mental workload multitasking condition, as evidenced by the higher average tracking distance, smaller number of response times, and longer response time of the meter. The standard deviation of the RR intervals (SDNN) was negatively correlated with subjective mental workload in the low task load condition and PFC activation was positively correlated with HR and subjective mental workload in the medium task load condition.

CONCLUSION

HR and PFC activation can be used to detect changes in mental workload during simulated flight multitasking tasks.

Mental workload classification based on ignored auditory probes and spatial covariance

Objective.Estimation of mental workload (MWL) levels by electroencephalography (EEG)-based mental state monitoring systems has been widely explored. Using event-related potentials (ERPs), elicited by ignored auditory probes, minimizes intrusiveness and has shown high performance for estimating MWL level when tested in laboratory situations. However, when facing real-world applications, the characteristics of ERP waveforms, like latency and amplitude, are often affected by noise, which leads to a decrease in classification performance. One approach to mitigating this is using spatial covariance, which is less sensitive to latency and amplitude distortion. In this study, we used ignored auditory probes in a single-stimulus paradigm and tested Riemannian processed covariance-based features for MWL level estimation in a realistic flight-control task.Approach.We recorded EEG data with an eight-channel system from participants while they performed a simulated drone-control task and manipulated MWL levels (high and low) by task difficulty. We compared support vector machine classification performance based on frequency band power features versus features generated via the Riemannian log map operator from spatial covariance matrices. We also compared accuracy of using data segmented as auditory ERPs versus non-ERPs, for which data windows did not overlap with the ERPs.Main results.Classification accuracy of both types of features showed no significant difference between ERPs and non-ERPs. When we ignore auditory stimuli to perform continuous decoding, covariance-based features in the gamma band had area under the receiver-operating-characteristic curve (AUC) of 0.883, which was significantly higher than band power features (AUC = 0.749).Significance.This study demonstrates that Riemannian-processed covariance features are viable for MWL classification under a realistic experimental scenario.

From 2D to VR Film: A Research on the Load of Different Cutting Rates Based on EEG Data Processing

Focusing on virtual reality (VR) and film cutting, this study compared and evaluated the effect of visual mode (2D, VR) and cutting rate (fast, medium, slow) on a load, to make an attempt for VR research to enter the cognitive field. This study uses a 2 × 3 experimental research design. Forty participants were divided into one of two groups randomly and watched films with three cutting rates. The subjective and objective data were collected during the experiment. The objective results confirm that VR films bring more powerful alpha, beta, theta wave activities, and bring a greater load. The subjective results confirm that the fast cutting rate brings a greater load. These results provide a theoretical support for further exploring the evaluation methods and standards of VR films and improving the viewing experience in the future.

Aircraft Pilots Workload Analysis: Heart Rate Variability Objective Measures and NASA-Task Load Index Subjective Evaluation

Workload and fatigue of aircraft pilots represent an argument of great interest in the framework of human factors and a pivotal point to be considered in aviation safety. 75% of aircraft accidents are related to human errors that, in most cases, are due to high level of mental workload and fatigue. There exist several subjective or objective metrics to quantify the pilots’ workload level, with both linear and nonlinear relationships reported in the literature. The main research objective of the present work is to analyze the relationships between objective and subjective workload measurements by looking for a correlation between metrics belonging to the subjective and biometric rating methods. More particularly, the Heart Rate Variability (HRV) is used for the objective analysis, whereas the NASA-TLX questionnaire is the tool chosen for the subjective evaluation of the workload. Two different flight scenarios were considered for the studies: the take-off phase with the initial climb and the final approach phase with the landing. A Maneuver Error Index (MEI) is also introduced to evaluate the pilot flight performance according to mission requirements. Both qualitative and quantitative correlation analyses were performed among the MEI, subjective and objective measurements. Monotonic relationships were found within the HRV indexes, and a nonlinear relationship is proposed among NASA-TLX and HRV indexes. These findings suggest that the relationship between workload, biometric data, and performance indexes are characterized by intricate patterns of nonlinear relationships.

Assessment of mental workload based on multi-physiological signals

BACKGROUND:

Mental workload is one of the contributing factors to human errors in road accidents or other potentially adverse incidents.

OBJECTIVE:

This research probes the effects of mental workload on the electroencephalographic (EEG) and electrocardiogram (ECG) of subjects in visual monitoring tasks, based on which a comprehensive evaluation model for mental workload is established effectively.

METHODS:

Three degrees of mental workload were obtained by monitoring tasks with different levels of difficulty. 20 healthy subjects were selected to take part in the research.

RESULTS:

The subjective scores showed a significant increase with the increase of task difficulty, meanwhile the reaction time (RT) increased and the accuracy decreased significantly, which proved the validity of three degrees of mental workload induced. For the EEG parameters, a significant decrease of θ energy was found in Frontal, Parietal and Occipital with the increase of level of mental workload, as well as a significant decrease of α energy in Frontal, Central and Occipital, meanwhile a significant increase of β energy occurred in Frontal and Occipital. There was a significant decrease of α/θ in Occipital, and significant increases of θ/β and (α+β)/θ in Frontal, Central and Occipital, meanwhile (α+θ)/β and WPE decreased significantly in Frontal and Occipital. Among the ECG parameters, it was shown that Mean RR, RMSSD, HF_norm and SampEn decreased significantly with the increase of task difficulty, while LF_norm and LF/HF showed significant increases. These EEG indictors in Occipital and ECG indictors were chosen and constituted a multidimensional original sample. Principal Component Analysis (PCA) was used to extract the principal elements and decreased the dimension of sample space in order to simplify the calculation, based on which an effective classification model with accuracy of 80% was achieved by support vector machine (SVM).

CONCLUSION:

This study demonstrates that the proposed algorithm can be applied to mental workload monitoring.

A Systematic Review of Physiological Measures of Mental Workload

Mental workload (MWL) can affect human performance and is considered critical in the design and evaluation of complex human-machine systems. While numerous physiological measures are used to assess MWL, there appears no consensus on their validity as effective agents of MWL. This study was conducted to provide a comprehensive understanding of the use of physiological measures of MWL and to synthesize empirical evidence on the validity of the measures to discriminate changes in MWL. A systematical literature search was conducted with four electronic databases for empirical studies measuring MWL with physiological measures. Ninety-one studies were included for analysis. We identified 78 physiological measures, which were distributed in cardiovascular, eye movement, electroencephalogram (EEG), respiration, electromyogram (EMG) and skin categories. Cardiovascular, eye movement and EEG measures were the most widely used across varied research domains, with 76%, 66%, and 71% of times reported a significant association with MWL, respectively. While most physiological measures were found to be able to discriminate changes in MWL, they were not universally valid in all task scenarios. The use of physiological measures and their validity for MWL assessment also varied across different research domains. Our study offers insights into the understanding and selection of appropriate physiological measures for MWL assessment in varied human-machine systems.

Cardiac Measures of Cognitive Workload: A Meta-Analysis

OBJECTIVE

We aimed to provide an assessment of the impact of workload manipulations on various cardiac measurements. We further sought to determine the most effective measurement approaches of cognitive workload as well as quantify the conditions under which these measures are most effective for interpretation.

BACKGROUND

Cognitive workload affects human performance, particularly when load is relatively high (overload) or low (underload). Despite ongoing interest in assessing cognitive workload through cardiac measures, it is currently unclear which cardiac-based assessments best indicate cognitive workload. Although several quantitative studies and qualitative reviews have sought to provide guidance, no meta-analytic integration of cardiac assessment(s) of cognitive workload exists to date.

METHOD

We used Morris and DeShon's meta-analytic procedures to quantify the changes in cardiac measures due to task load conditions.

RESULTS

Sample-weighted Cohen's d values suggest that several metrics of cardiac activity demonstrate sensitivity in response to cognitive workload manipulations. Heart rate variability measures show sensitivity to task load, conditions of event rate, and task duration. Authors of future work should seek to quantify the utility of leveraging multiple metrics to understand workload.

CONCLUSION

Results suggest that assessment of cognitive workload can be done using various cardiac activity indicators. Further, given the number of valid and reliable measures available, researchers and practitioners should base their selection of a psychophysiological measure on the experimental and practical concerns inherent to their task/protocol.

APPLICATIONS

Findings bear implications for future assessment of cognitive workload within basic and applied settings. Future research should seek to validate conditions under which measurements are best interpreted, including but not limited to individual differences.

Measuring mental workload using physiological measures: A systematic review

Technological advances have led to physiological measurement being increasingly used to measure and predict operator states. Mental workload (MWL) in particular has been characterised using a variety of physiological sensor data. This systematic review contributes a synthesis of the literature summarising key findings to assist practitioners to select measures for use in evaluation of MWL. We also describe limitations of the methods to assist selection when being deployed in applied or laboratory settings. We detail fifty-eight peer reviewed journal articles which present original data using physiological measures to include electrocardiographic, respiratory, dermal, blood pressure and ocular. Electroencephalographic measures have been included if they are presented with another measure to constrain scope. The literature reviewed covers a range of applied and experimental studies across various domains, safety-critical applications being highly represented in the sample of applied literature reviewed. We present a summary of the six measures and provide an evidence base which includes how to deploy each measure, and characteristics that can affect or preclude the use of a measure in research. Measures can be used to discriminate differences in MWL caused by task type, task load, and in some cases task difficulty. Varying ranges of sensitivity to sudden or gradual changes in taskload are also evident across the six measures. We conclude that there is no single measure that clearly discriminates mental workload but there is a growing empirical basis with which to inform both science and practice.

A comprehensive prediction and evaluation method of pilot workload

BACKGROUND:

The prediction and evaluation of pilot workload is a key problem in human factor airworthiness of cockpit.

OBJECTIVE:

A pilot traffic pattern task was designed in a flight simulation environment in order to carry out the pilot workload prediction and improve the evaluation method.

METHODS:

The prediction of typical flight subtasks and dynamic workloads (cruise, approach, and landing) were built up based on multiple resource theory, and a favorable validity was achieved by the correlation analysis verification between sensitive physiological data and the predicted value.

RESULTS:

Statistical analysis indicated that eye movement indices (fixation frequency, mean fixation time, saccade frequency, mean saccade time, and mean pupil diameter), Electrocardiogram indices (mean normal-to-normal interval and the ratio between low frequency and sum of low frequency and high frequency), and Electrodermal Activity indices (mean tonic and mean phasic) were all sensitive to typical workloads of subjects.

CONCLUSION:

A multinominal logistic regression model based on combination of physiological indices (fixation frequency, mean normal-to-normal interval, the ratio between low frequency and sum of low frequency and high frequency, and mean tonic) was constructed, and the discriminate accuracy was comparatively ideal with a rate of 84.85%.

Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload

BACKGROUND

Surgeons in the operating theatre deal constantly with high-demand tasks that require simultaneous processing of a large amount of information. In certain situations, high cognitive load occurs, which may impact negatively on a surgeon's performance. This systematic review aims to provide a comprehensive understanding of the different methods used to assess surgeons' cognitive load, and a critique of the reliability and validity of current assessment metrics.

METHODS

A search strategy encompassing MEDLINE, Embase, Web of Science, PsycINFO, ACM Digital Library, IEEE Xplore, PROSPERO and the Cochrane database was developed to identify peer-reviewed articles published from inception to November 2016. Quality was assessed by using the Medical Education Research Study Quality Instrument (MERSQI). A summary table was created to describe study design, setting, specialty, participants, cognitive load measures and MERSQI score.

RESULTS

Of 391 articles retrieved, 84 met the inclusion criteria, totalling 2053 unique participants. Most studies were carried out in a simulated setting (59 studies, 70 per cent). Sixty studies (71 per cent) used self-reporting methods, of which the NASA Task Load Index (NASA-TLX) was the most commonly applied tool (44 studies, 52 per cent). Heart rate variability analysis was the most used real-time method (11 studies, 13 per cent).

CONCLUSION

Self-report instruments are valuable when the aim is to assess the overall cognitive load in different surgical procedures and assess learning curves within competence-based surgical education. When the aim is to assess cognitive load related to specific operative stages, real-time tools should be used, as they allow capture of cognitive load fluctuation. A combination of both subjective and objective methods might provide optimal measurement of surgeons' cognition.