Mental Workload Estimation Based on Physiological Features for Pilot-UAV Teaming Applications

A passive brain-computer interface for operator mental fatigue estimation in monotonous surveillance operations: time-on-task and performance labeling issues

Objective: A central component of search and rescue missions is the visual search of survivors. In large parts, this depends on human operators and is, therefore, subject to the constraints of human cognition, such as mental fatigue (MF). This makes detecting MF a critical step to be implemented in future systems. However, to the best of our knowledge, it has seldom been evaluated using a realistic visual search task. In addition, an accuracy discrepancy exists between studies that use time-on-task (TOT)-the popular method-and performance metrics for labels. Yet, to our knowledge, they have never been directly compared.Approach: This study was designed to address both issues: the use of a realistic task to elicit MF during a monotonous visual search task and the labeling type used for intra-participant fatigue estimation. Over four blocks of 15 min, participants had to identify targets on a computer while their cardiac, cerebral (EEG), and eye-movement activities were recorded. The recorded data were then fed into several physiological computing pipelines.Main results: The results show that the capability of a machine learning algorithm to detect MF depends less on the input data but rather on how MF is defined. Using TOT, very high classification accuracies are obtained (e.g. 99.3%). On the other hand, if MF is estimated based on behavioral performance, a metric with a much greater operational value, classification accuracies return to chance level (i.e. 52.2%).Significance: TOT-based MF estimation is popular, and strong classification accuracies can be achieved with a multitude of sensors. These factors contribute to the popularity of this method, but both usability and the relation to the concept of MF are neglected.

Investigation of eye movement measures of mental workload in healthcare: Can pupil dilations reflect fatigue or overload when it comes to health information system use?

The use of health information systems (HIS) can result in high workloads and, consequently, poor performance characterized by e.g. increased occurrence of errors among clinicians. Pupillometry offers a good possibility to measure mental workload in a dynamic work setting. Currently, there is a lack of empirical research in the context of healthcare settings. Therefore, the aim of the present study was to examine whether specific eye movement measures are suitable for measuring mental workload in the healthcare setting, especially when working with HIS. 49 persons participated in our simulation-lab study. They had to complete a system-related task as well as an increasing n-back task. Both tasks were modified regarding task difficulty. Results show significant differences for objective and subjective workload measures between increasing task levels. There are also hints for an overload/fatigue indicator in pupil data. Our results are limited in terms of external validity, causality and effects. Future work should focus on high-fidelity simulations and less time-consuming analysis approaches.

Adaptive Human-Robotic Interaction for Robotic-assisted Surgical Settings

INTRODUCTION

Increased complexity in robotic-assisted surgical system interfaces introduces problems with human-robot collaboration that result in excessive mental workload (MWL), adversely impacting a surgeon's task performance and increasing error probability. Real-time monitoring of the operator's MWL will aid in identifying when and how interventions can be best provided to moderate MWL. In this study, an MWL-based adaptive automation system is constructed and evaluated for its effectiveness during robotic-assisted surgery.

MATERIALS AND METHODS

This study recruited 10 participants first to perform surgical tasks under different cognitive workload levels. Physiological signals were obtained and employed to build a real-time system for cognitive workload monitoring. To evaluate the effectiveness of the proposed system, 15 participants were recruited to perform the surgical task with and without the proposed system. The participants' task performance and perceived workload were collected and compared.

RESULTS

The proposed neural network model achieved an accuracy of 77.9% in cognitive workload classification. In addition, better task performance and lower perceived workload were observed when participants completed the experimental task under the task condition supplemented with adaptive aiding using the proposed system.

CONCLUSIONS

The proposed MWL monitoring system successfully diminished the perceived workload of participants and increased their task performance under high-stress conditions via interventions by a semi-autonomous suction tool. The preliminary results from the comparative study show the potential impact of automated adaptive aiding systems in enhancing surgical task performance via cognitive workload-triggered interventions in robotic-assisted surgery.

The effect of mental schema evolution on mental workload measurement: an EEG study with simulated quadrotor UAV operation

OBJECTIVE

Mental workload is the result of the interactions between the demands of an operation task and the skills, behavior and perception of the performer. Working under a high mental workload can significantly affect an operator's ability to choose optimal decisions. However, the effect of mental schema, which reflects the level of expertise of an operator, on mental workload remains unclear. Here, we propose a theoretical framework for describing how the evolution of mental schema affects mental workload from the perspective of cognitive processing.

APPROACH

we recruited 51 students to participate in a 10-day simulated UAV flight training. The EEG PSD metrics were used to investigate the changes in neural responses caused by variations in the mental workload at different stages of mental schema evolution.

MAIN RESULTS

It was found that mental schema evolution influenced the direction and change trends of the frontal theta PSD, parietal alpha PSD, and central beta PSD. Initially, before the mental schema was formed, only the frontal theta PSD increased with increasing task difficulty; when the mental schema was initially being developed, the frontal theta PSD and the parietal alpha PSD decreased with increasing task difficulty, while the central beta PSD increased with increasing task difficulty. Finally, as the mental schema gradually matured, the trend of the three indicators did not change with increasing task difficulty. However, differences in the frontal PSD became more pronounced across task difficulty levels, while differences in the parietal PSD narrowed.

SIGNIFICANCE

Our results describe the relationship between the EEG power spectrum and the mental workload of UAV operators as the mental schema evolved. This suggests that EEG indicators can not only provide more accurate measurements of mental workload but also provide insights into the development of an operator's skill level.

Retrospective on the First Passive Brain-Computer Interface Competition on Cross-Session Workload Estimation

As is the case in several research domains, data sharing is still scarce in the field of Brain-Computer Interfaces (BCI), and particularly in that of passive BCIs-i.e., systems that enable implicit interaction or task adaptation based on a user's mental state(s) estimated from brain measures. Moreover, research in this field is currently hindered by a major challenge, which is tackling brain signal variability such as cross-session variability. Hence, with a view to develop good research practices in this field and to enable the whole community to join forces in working on cross-session estimation, we created the first passive brain-computer interface competition on cross-session workload estimation. This competition was part of the 3rd International Neuroergonomics conference. The data were electroencephalographic recordings acquired from 15 volunteers (6 females; average 25 y.o.) who performed 3 sessions-separated by 7 days-of the Multi-Attribute Task Battery-II (MATB-II) with 3 levels of difficulty per session (pseudo-randomized order). The data -training and testing sets-were made publicly available on Zenodo along with Matlab and Python toy code (https://doi.org/10.5281/zenodo.5055046). To this day, the database was downloaded more than 900 times (unique downloads of all version on the 10th of December 2021: 911). Eleven teams from 3 continents (31 participants) submitted their work. The best achieving processing pipelines included a Riemannian geometry-based method. Although better than the adjusted chance level (38% with an α at 0.05 for a 3-class classification problem), the results still remained under 60% of accuracy. These results clearly underline the real challenge that is cross-session estimation. Moreover, they confirmed once more the robustness and effectiveness of Riemannian methods for BCI. On the contrary, chance level results were obtained by one third of the methods-4 teams- based on Deep Learning. These methods have not demonstrated superior results in this contest compared to traditional methods, which may be due to severe overfitting. Yet this competition is the first step toward a joint effort to tackle BCI variability and to promote good research practices including reproducibility.