Monitoring Pilot's Mental Workload Using ERPs and Spectral Power with a Six-Dry-Electrode EEG System in Real Flight Conditions

The moving wave: Applications of the mobile EEG approach to study human attention

Although historically confined to traditional research laboratories, electroencephalography (EEG) paradigms are now being applied to study a wide array of behaviors, from daily activities to specialized tasks in diverse fields such as sports science, neurorehabilitation, and education. This transition from traditional to real-world mobile research can provide new tools for understanding attentional processes as they occur naturally. Early mobile EEG research has made progress, despite the large size and wired connections. Recent developments in hardware and software have expanded the possibilities of mobile EEG, enabling a broader range of applications. Despite these advancements, limitations influencing mobile EEG remain that must be overcome to achieve adequate reliability and validity. In this review, we first assess the feasibility of mobile paradigms, including electrode selection, artifact correction techniques, and methodological considerations. This review underscores the importance of ecological, construct, and predictive validity in ensuring the trustworthiness and applicability of mobile EEG findings. Second, we explore studies on attention in naturalistic settings, focusing on replicating classic P3 component studies in mobile paradigms like stationary biking in our lab, and activities such as walking, cycling, and dual-tasking outside of the lab. We emphasize how the mobile approach complements traditional laboratory paradigms and the types of insights gained in naturalistic research settings. Third, we discuss promising applications of portable EEG in workplace safety and other areas including road safety, rehabilitation medicine, and brain-computer interfaces. In summary, this review explores the expanding possibilities of mobile EEG while recognizing the existing challenges in fully realizing its potential.

Classification of mental workload using brain connectivity and machine learning on electroencephalogram data

Mental workload refers to the cognitive effort required to perform tasks, and it is an important factor in various fields, including system design, clinical medicine, and industrial applications. In this paper, we propose innovative methods to assess mental workload from EEG data that use effective brain connectivity for the purpose of extracting features, a hierarchical feature selection algorithm to select the most significant features, and finally machine learning models. We have used the Simultaneous Task EEG Workload (STEW) dataset, an open-access collection of raw EEG data from 48 subjects. We extracted brain-effective connectivities by the direct directed transfer function and then selected the top 30 connectivities for each standard frequency band. Then we applied three feature selection algorithms (forward feature selection, Relief-F, and minimum-redundancy-maximum-relevance) on the top 150 features from all frequencies. Finally, we applied sevenfold cross-validation on four machine learning models (support vector machine (SVM), linear discriminant analysis, random forest, and decision tree). The results revealed that SVM as the machine learning model and forward feature selection as the feature selection method work better than others and could classify the mental workload levels with accuracy equal to 89.53% (± 1.36).

Evaluation of a headphones-fitted EEG system for the recording of auditory evoked potentials and mental workload assessment

Advancements in portable neuroimaging technologies open up new opportunities to gain insight into the neural dynamics and cognitive processes underlying day-to-day behaviors. In this study, we evaluated the relevance of a headphone- mounted electroencephalogram (EEG) system for monitoring mental workload. The participants (N = 12) were instructed to pay attention to auditory alarms presented sporadically while performing the Multi-Attribute Task Battery (MATB) whose difficulty was staged across three conditions to manipulate mental workload. The P300 Event-Related Potentials (ERP) elicited by the presentation of auditory alarms were used as probes of attentional resources available. The amplitude and latency of P300 ERPs were compared across experimental conditions. Our findings indicate that the P300 ERP component can be captured using a headphone-mounted EEG system. Moreover, neural responses to alarm could be used to classify mental workload with high accuracy (over 80%) at a single-trial level. Our analyses indicated that the signal-to-noise ratio acquired by the sponge-based sensors remained stable throughout the recordings. These results highlight the potential of portable neuroimaging technology for the development of neuroassistive applications while underscoring the current limitations and challenges associated with the integration of EEG sensors in everyday-life wearable technologies. Overall, our study contributes to the growing body of research exploring the feasibility and validity of wearable neuroimaging technologies for the study of human cognition and behavior in real-world settings.

EEG Dataset Collection for Mental Workload Predictions in Flight-Deck Environment

High mental workload reduces human performance and the ability to correctly carry out complex tasks. In particular, aircraft pilots enduring high mental workloads are at high risk of failure, even with catastrophic outcomes. Despite progress, there is still a lack of knowledge about the interrelationship between mental workload and brain functionality, and there is still limited data on flight-deck scenarios. Although recent emerging deep-learning (DL) methods using physiological data have presented new ways to find new physiological markers to detect and assess cognitive states, they demand large amounts of properly annotated datasets to achieve good performance. We present a new dataset of electroencephalogram (EEG) recordings specifically collected for the recognition of different levels of mental workload. The data were recorded from three experiments, where participants were induced to different levels of workload through tasks of increasing cognition demand. The first involved playing the N-back test, which combines memory recall with arithmetical skills. The second was playing Heat-the-Chair, a serious game specifically designed to emphasize and monitor subjects under controlled concurrent tasks. The third was flying in an Airbus320 simulator and solving several critical situations. The design of the dataset has been validated on three different levels: (1) correlation of the theoretical difficulty of each scenario to the self-perceived difficulty and performance of subjects; (2) significant difference in EEG temporal patterns across the theoretical difficulties and (3) usefulness for the training and evaluation of AI models.

Blink-Related Oscillations Provide Naturalistic Assessments of Brain Function and Cognitive Workload within Complex Real-World Multitasking Environments

BACKGROUND

There is a significant need to monitor human cognitive performance in complex environments, with one example being pilot performance. However, existing assessments largely focus on subjective experiences (e.g., questionnaires) and the evaluation of behavior (e.g., aircraft handling) as surrogates for cognition or utilize brainwave measures which require artificial setups (e.g., simultaneous auditory stimuli) that intrude on the primary tasks. Blink-related oscillations (BROs) are a recently discovered neural phenomenon associated with spontaneous blinking that can be captured without artificial setups and are also modulated by cognitive loading and the external sensory environment-making them ideal for brain function assessment within complex operational settings.

METHODS

Electroencephalography (EEG) data were recorded from eight adult participants (five F, M = 21.1 years) while they completed the Multi-Attribute Task Battery under three different cognitive loading conditions. BRO responses in time and frequency domains were derived from the EEG data, and comparisons of BRO responses across cognitive loading conditions were undertaken. Simultaneously, assessments of blink behavior were also undertaken.

RESULTS

Blink behavior assessments revealed decreasing blink rate with increasing cognitive load (p < 0.001). Prototypical BRO responses were successfully captured in all participants (p < 0.001). BRO responses reflected differences in task-induced cognitive loading in both time and frequency domains (p < 0.05). Additionally, reduced pre-blink theta band desynchronization with increasing cognitive load was also observed (p < 0.05).

CONCLUSION

This study confirms the ability of BRO responses to capture cognitive loading effects as well as preparatory pre-blink cognitive processes in anticipation of the upcoming blink during a complex multitasking situation. These successful results suggest that blink-related neural processing could be a potential avenue for cognitive state evaluation in operational settings-both specialized environments such as cockpits, space exploration, military units, etc. and everyday situations such as driving, athletics, human-machine interactions, etc.-where human cognition needs to be seamlessly monitored and optimized.

Modeling Complex EEG Data Distribution on the Riemannian Manifold Toward Outlier Detection and Multimodal Classification

OBJECTIVE

The usage of Riemannian geometry for Brain-computer interfaces (BCIs) has gained momentum in recent years. Most of the machine learning techniques proposed for Riemannian BCIs consider the data distribution on a manifold to be unimodal. However, the distribution is likely to be multimodal rather than unimodal since high-data variability is a crucial limitation of electroencephalography (EEG). In this paper, we propose a novel data modeling method for considering complex data distributions on a Riemannian manifold of EEG covariance matrices, aiming to improve BCI reliability.

METHODS

Our method, Riemannian spectral clustering (RiSC), represents EEG covariance matrix distribution on a manifold using a graph with proposed similarity measurement based on geodesic distances, then clusters the graph nodes through spectral clustering. This allows flexibility to model both a unimodal and a multimodal distribution on a manifold. RiSC can be used as a basis to design an outlier detector named outlier detection Riemannian spectral clustering (odenRiSC) and a multimodal classifier named multimodal classifier Riemannian spectral clustering (mcRiSC). All required parameters of odenRiSC/mcRiSC are selected in data-driven manner. Moreover, there is no need to pre-set a threshold for outlier detection and the number of modes for multimodal classification.

RESULTS

The experimental evaluation revealed odenRiSC can detect EEG outliers more accurately than existing methods and mcRiSC outperformed the standard unimodal classifier, especially on high-variability datasets.

CONCLUSION

odenRiSC/mcRiSC are anticipated to contribute to making real-life BCIs outside labs and neuroergonomics applications more robust.

SIGNIFICANCE

RiSC can work as a robust EEG outlier detector and multimodal classifier.

Towards ubiquitous and nonintrusive measurements of brain function in the real world: assessing blink-related oscillations during simulated flight using portable low-cost EEG

Blink-related oscillations (BRO) are newly discovered neurophysiological phenomena associated with spontaneous blinking and represent cascading neural mechanisms including visual sensory, episodic memory, and information processing responses. These phenomena have been shown to be present at rest and during tasks and are modulated by cognitive load, creating the possibility for brain function assessments that can be integrated seamlessly into real-world settings. Prior works have largely examined the BRO phenomenon within controlled laboratory environments using magnetoencephalography and high-density electroencephalography (EEG) that are ill-suited for real-world deployment. Investigating BROs using low-density EEG within complex environments reflective of the real-world would further our understanding of how BRO responses can be utilized in real-world settings. We evaluated whether the BRO response could be captured in a high-fidelity flight simulation environment using a portable, low-density wireless EEG system. The effects of age and task demands on BRO responses were also examined. EEG data from 30 licensed pilots (age 43.37 +/- 17.86, 2 females) were collected during simulated flights at two cognitive workload levels. Comparisons of signal amplitudes were undertaken to confirm the presence of BRO responses and mixed model ANOVAs quantified the effects of workload and age group on BRO amplitudes. Significant increases in neural activity were observed post-blink compared to the baseline period (p < 0.05), confirming the presence of BRO responses. In line with prior studies, results showed BRO time-domain responses from the delta band (0.5-4 Hz) consisting of an early negative peak followed by a positive peak post-blink in temporal and parietal electrodes. Additionally, task workload and age-related effects were also found, with observations of the enhancement of BRO amplitudes with older age and attenuation of BRO responses in high workloads (p < 0.05). These findings demonstrate that it is possible to capture BRO responses within simulated flight environments using portable, low-cost, easy-to-use EEG systems. Furthermore, biological and task salience were reflected in these BRO responses. The successful detection and demonstration of both task-and age-related modulation of BRO responses in this study open the possibility of assessing human brain function across the lifespan with BRO responses in complex and realistic environments.

MST-net: A multi-scale swin transformer network for EEG-based cognitive load assessment

Cognitive load assessment plays a crucial role in monitoring safe production, resource allocation, and subjective initiative in human-computer interaction. Due to its high time resolution and convenient acquisition, Electroencephalography (EEG) is widely applied in brain monitoring and cognitive state assessment. In this study, a multi-scale Swin Transformer network (MST-Net) was proposed for cognitive load assessment, which extracts local features with different sensory fields using a multi-scale parallel convolution model and introduces the attention mechanism of the Swin Transformer to obtain the feature correlations among multi-scale local features. The performance of the proposed network was validated using the EEG signals collected during cognitive tasks and N-back tasks with three different load levels. Results show that the MST-Net network achieved the best classification accuracy on both local and public datasets, and was higher than the mainstream Swin Transformer and CNN. Furthermore, results of ablation experiments and feature visualization revealed that the proposed MST-Net could well characterize different cognitive loads, which not only provided novel and powerful tools for cognitive load assessment but also showed potential for broad application in brain-computer interface (BCI) systems.

Advanced Electrode Technologies for Noninvasive Brain-Computer Interfaces

Brain-computer interfaces (BCIs) have garnered significant attention in recent years due to their potential applications in medical, assistive, and communication technologies. Building on this, noninvasive BCIs stand out as they provide a safe and user-friendly method for interacting with the human brain. In this work, we provide a comprehensive overview of the latest developments and advancements in material, design, and application of noninvasive BCIs electrode technology. We also explore the challenges and limitations currently faced by noninvasive BCI electrode technology and sketch out the technological roadmap from three dimensions: Materials and Design; Performances; Mode and Function. We aim to unite research efforts within the field of noninvasive BCI electrode technology, focusing on the consolidation of shared goals and fostering integrated development strategies among a diverse array of multidisciplinary researchers.

Burst c-VEP Based BCI: Optimizing stimulus design for enhanced classification with minimal calibration data and improved user experience

The utilization of aperiodic flickering visual stimuli under the form of code-modulated Visual Evoked Potentials (c-VEP) represents a pivotal advancement in the field of reactive Brain-Computer Interface (rBCI). A major advantage of the c-VEP approach is that the training of the model is independent of the number and complexity of targets, which helps reduce calibration time. Nevertheless, the existing designs of c-VEP stimuli can be further improved in terms of visual user experience but also to achieve a higher signal-to-noise ratio, while shortening the selection time and calibration process. In this study, we introduce an innovative variant of code-VEP, referred to as "Burst c-VEP". This original approach involves the presentation of short bursts of aperiodic visual flashes at a deliberately slow rate, typically ranging from two to four flashes per second. The rationale behind this design is to leverage the sensitivity of the primary visual cortex to transient changes in low-level stimuli features to reliably elicit distinctive series of visual evoked potentials. In comparison to other types of faster-paced code sequences, burst c-VEP exhibit favorable properties to achieve high bitwise decoding performance using convolutional neural networks (CNN), which yields potential to attain faster selection time with the need for less calibration data. Furthermore, our investigation focuses on reducing the perceptual saliency of c-VEP through the attenuation of visual stimuli contrast and intensity to significantly improve users' visual comfort. The proposed solutions were tested through an offline 4-classes c-VEP protocol involving 12 participants. Following a factorial design, participants were instructed to focus on c-VEP targets whose pattern (burst and maximum-length sequences) and amplitude (100% or 40% amplitude depth modulations) were manipulated across experimental conditions. Firstly, the full amplitude burst c-VEP sequences exhibited higher accuracy, ranging from 90.5% (with 17.6s of calibration data) to 95.6% (with 52.8s of calibration data), compared to its m-sequence counterpart (71.4% to 85.0%). The mean selection time for both types of codes (1.5 s) compared favorably to reports from previous studies. Secondly, our findings revealed that lowering the intensity of the stimuli only slightly decreased the accuracy of the burst code sequences to 94.2% while leading to substantial improvements in terms of user experience. Taken together, these results demonstrate the high potential of the proposed burst codes to advance reactive BCI both in terms of performance and usability. The collected dataset, along with the proposed CNN architecture implementation, are shared through open-access repositories.

Task-independent auditory probes reveal changes in mental workload during simulated quadrotor UAV training

Objective

The event-related potential (ERP) methods based on laboratory control scenes have been widely used to measure the level of mental workload during operational tasks. In this study, both task difficulty and test time were considered. Auditory probes (ignored task-irrelevant background sounds) were used to explore the changes in mental workload of unmanned aerial vehicle (UAV) operators during task execution and their ERP representations.

Approach

51 students participated in a 10-day training and test of simulated quadrotor UAV. During the experiment, background sound was played to induce ERP according to the requirements of oddball paradigm, and the relationship between mental workload and the amplitudes of N200 and P300 in ERP was explored.

Main results

Our study shows that the mental workload during operational task training is multi-dimensional, and its changes are affected by bottom-up perception and top-down cognition. The N200 component of the ERP evoked by the auditory probe corresponds to the bottom-up perceptual part; while the P300 component corresponds to the top-down cognitive part, which is positively correlated with the improvement of skill level.

Significance

This paper describes the relationship between ERP induced by auditory probes and mental workload from the perspective of multi-resource theory and human information processing. This suggests that the auditory probe can be used to reveal the mental workload during the training of operational tasks, which not only provides a possible reference for measuring the mental workload, but also provides a possibility for identifying the development of the operator's skill level and evaluating the training effect.

The Use of Quantitative Electroencephalography (QEEG) to Assess Post-COVID-19 Concentration Disorders in Professional Pilots: An Initial Concept

Announced by WHO in 2020, the global COVID-19 pandemic caused by SARS-CoV-2 has affected many people, leading to serious health consequences. These consequences are observed in the daily lives of infected patients as various dysfunctions and limitations. More and more people are suffering post-COVID-19 complications that interfere with or completely prevent them from working or even functioning independently on a daily basis. The aim of our study was to demonstrate that innovative quantitative electroencephalography (QEEG) can be used to assess cognitive function disorders reported after the COVID-19 pandemic. It is worth noting that no similar study has been conducted to date in a group of pilots. The QEEG method we used is currently one of the basic neurological examinations, enabling easy observation of post-COVID-19 changes in the nervous system. With the innovativeness of this technique, our study shows that the use of quantitative electroencephalography can be a precursor in identifying complications associated with cognitive function disorders after COVID-19. Our study was conducted on twelve 26-year-old pilots. All participants had attended the same flight academy and had contracted SARS-CoV-2 infection. The pilots began to suspect COVID-19 infection when they developed typical symptoms such as loss of smell and taste, respiratory problems, and rapid fatigue. Quantitative electroencephalography (QEEG), which is one of the most innovative forms of diagnostics, was used to diagnose the patients. Comparison of the results between the study and control groups showed significantly higher values of all measurements of alpha, theta, and beta2 waves in the study group. In the case of the sensorimotor rhythm (SMR), the measurement results were significantly higher in the control group compared to the study group. Our study, conducted on pilots who had recovered from COVID-19, showed changes in the amplitudes of brain waves associated with relaxation and concentration. The results confirmed the issues reported by pilots as evidenced by the increased amplitudes of alfa, theta, and beta2 waves. It should be emphasized that the modern diagnostic method (QEEG) presented here has significant importance in the medical diagnosis of various symptoms and observation of treatment effects in individuals who have contracted the SARS-CoV-2 virus. The present study demonstrated an innovative approach to the diagnosis of neurological complications after COVID-19.

Multimodal Approach for Pilot Mental State Detection Based on EEG

The safety of flight operations depends on the cognitive abilities of pilots. In recent years, there has been growing concern about potential accidents caused by the declining mental states of pilots. We have developed a novel multimodal approach for mental state detection in pilots using electroencephalography (EEG) signals. Our approach includes an advanced automated preprocessing pipeline to remove artefacts from the EEG data, a feature extraction method based on Riemannian geometry analysis of the cleaned EEG data, and a hybrid ensemble learning technique that combines the results of several machine learning classifiers. The proposed approach provides improved accuracy compared to existing methods, achieving an accuracy of 86% when tested on cleaned EEG data. The EEG dataset was collected from 18 pilots who participated in flight experiments and publicly released at NASA's open portal. This study presents a reliable and efficient solution for detecting mental states in pilots and highlights the potential of EEG signals and ensemble learning algorithms in developing cognitive cockpit systems. The use of an automated preprocessing pipeline, feature extraction method based on Riemannian geometry analysis, and hybrid ensemble learning technique set this work apart from previous efforts in the field and demonstrates the innovative nature of the proposed approach.

Detection of Pilot's Mental Workload Using a Wireless EEG Headset in Airfield Traffic Pattern Tasks

Elevated mental workload (MWL) experienced by pilots can result in increased reaction times or incorrect actions, potentially compromising flight safety. This study aims to develop a functional system to assist administrators in identifying and detecting pilots' real-time MWL and evaluate its effectiveness using designed airfield traffic pattern tasks within a realistic flight simulator. The perceived MWL in various situations was assessed and labeled using NASA Task Load Index (NASA-TLX) scores. Physiological features were then extracted using a fast Fourier transformation with 2-s sliding time windows. Feature selection was conducted by comparing the results of the Kruskal-Wallis (K-W) test and Sequential Forward Floating Selection (SFFS). The results proved that the optimal input was all PSD features. Moreover, the study analyzed the effects of electroencephalography (EEG) features from distinct brain regions and PSD changes across different MWL levels to further assess the proposed system's performance. A 10-fold cross-validation was performed on six classifiers, and the optimal accuracy of 87.57% was attained using a multi-class K-Nearest Neighbor (KNN) classifier for classifying different MWL levels. The findings indicate that the wireless headset-based system is reliable and feasible. Consequently, numerous wireless EEG device-based systems can be developed for application in diverse real-driving scenarios. Additionally, the current system contributes to future research on actual flight conditions.

Effects of spaceflight on the EEG alpha power and functional connectivity

Electroencephalography (EEG) can detect changes in cerebral activity during spaceflight. This study evaluates the effect of spaceflight on brain networks through analysis of the Default Mode Network (DMN)'s alpha frequency band power and functional connectivity (FC), and the persistence of these changes. Five astronauts' resting state EEGs under three conditions were analyzed (pre-flight, in-flight, and post-flight). DMN's alpha band power and FC were computed using eLORETA and phase-locking value. Eyes-opened (EO) and eyes-closed (EC) conditions were differentiated. We found a DMN alpha band power reduction during in-flight (EC: p < 0.001; EO: p < 0.05) and post-flight (EC: p < 0.001; EO: p < 0.01) when compared to pre-flight condition. FC strength decreased during in-flight (EC: p < 0.01; EO: p < 0.01) and post-flight (EC: ns; EO: p < 0.01) compared to pre-flight condition. The DMN alpha band power and FC strength reduction persisted until 20 days after landing. Spaceflight caused electrocerebral alterations that persisted after return to earth. Periodic assessment by EEG-derived DMN analysis has the potential to become a neurophysiologic marker of cerebral functional integrity during exploration missions to space.

Wireless EEG: A survey of systems and studies

The popular brain monitoring method of electroencephalography (EEG) has seen a surge in commercial attention in recent years, focusing mostly on hardware miniaturization. This has led to a varied landscape of portable EEG devices with wireless capability, allowing them to be used by relatively unconstrained users in real-life conditions outside of the laboratory. The wide availability and relative affordability of these devices provide a low entry threshold for newcomers to the field of EEG research. The large device variety and the at times opaque communication from their manufacturers, however, can make it difficult to obtain an overview of this hardware landscape. Similarly, given the breadth of existing (wireless) EEG knowledge and research, it can be challenging to get started with novel ideas. Therefore, this paper first provides a list of 48 wireless EEG devices along with a number of important-sometimes difficult-to-obtain-features and characteristics to enable their side-by-side comparison, along with a brief introduction to each of these aspects and how they may influence one's decision. Secondly, we have surveyed previous literature and focused on 110 high-impact journal publications making use of wireless EEG, which we categorized by application and analyzed for device used, number of channels, sample size, and participant mobility. Together, these provide a basis for informed decision making with respect to hardware and experimental precedents when considering new, wireless EEG devices and research. At the same time, this paper provides background material and commentary about pitfalls and caveats regarding this increasingly accessible line of research.

Stress and Workload Assessment in Aviation-A Narrative Review

In aviation, any detail can have massive consequences. Among the potential sources of failure, human error is still the most troublesome to handle. Therefore, research concerning the management of mental workload, attention, and stress is of special interest in aviation. Recognizing conditions in which a pilot is over-challenged or cannot act lucidly could avoid serious outcomes. Furthermore, knowing in depth a pilot's neurophysiological and cognitive-behavioral responses could allow for the optimization of equipment and procedures to minimize risk and increase safety. In addition, it could translate into a general enhancement of both the physical and mental well-being of pilots, producing a healthier and more ergonomic work environment. This review brings together literature on the study of stress and workload in the specific case of pilots of both civil and military aircraft. The most common approaches for studying these phenomena in the avionic context are explored in this review, with a focus on objective methodologies (e.g., the collection and analysis of neurophysiological signals). This review aims to identify the pros, cons, and applicability of the various approaches, to enable the design of an optimal protocol for a comprehensive study of these issues.

Reliability of Mental Workload Index Assessed by EEG with Different Electrode Configurations and Signal Pre-Processing Pipelines

BACKGROUND AND OBJECTIVE

Mental workload (MWL) is a relevant construct involved in all cognitively demanding activities, and its assessment is an important goal in many research fields. This paper aims at evaluating the reproducibility and sensitivity of MWL assessment from EEG signals considering the effects of different electrode configurations and pre-processing pipelines (PPPs).

METHODS

Thirteen young healthy adults were enrolled and were asked to perform 45 min of Simon's task to elicit a cognitive demand. EEG data were collected using a 32-channel system with different electrode configurations (fronto-parietal; Fz and Pz; Cz) and analyzed using different PPPs, from the simplest bandpass filtering to the combination of filtering, Artifact Subspace Reconstruction (ASR) and Independent Component Analysis (ICA). The reproducibility of MWL indexes estimation and the sensitivity of their changes were assessed using Intraclass Correlation Coefficient and statistical analysis.

RESULTS

MWL assessed with different PPPs showed reliability ranging from good to very good in most of the electrode configurations (average consistency > 0.87 and average absolute agreement > 0.92). Larger fronto-parietal electrode configurations, albeit being more affected by the choice of PPPs, provide better sensitivity in the detection of MWL changes if compared to a single-electrode configuration (18 vs. 10 statistically significant differences detected, respectively).

CONCLUSIONS

The most complex PPPs have been proven to ensure good reliability (>0.90) and sensitivity in all experimental conditions. In conclusion, we propose to use at least a two-electrode configuration (Fz and Pz) and complex PPPs including at least the ICA algorithm (even better including ASR) to mitigate artifacts and obtain reliable and sensitive MWL assessment during cognitive tasks.

Towards a versatile mental workload modeling using neurometric indices

Researchers have been working to magnify mental workload (MWL) modeling for a long time. An important aspect of its modeling is feature selection as it interprets bulky and high-dimensional EEG data and enhances the accuracy of the classification model. In this study, a feature selection technique is proposed to obtain an optimized feature set with multiple domain features that can contribute to classifying the MWL at three distinct levels. The brain signals from thirteen healthy subjects were examined while they attended an intrinsic MWL of spotting differences in a set of similar pictures. The Recursive Feature Elimination (RFE) technique selects the robust features from the feature matrix by eliminating all the least contributing features. Along with the Support Vector Machine (SVM), the overall classification accuracy with the proposed RFE reached 0.913 from 0.791 surpassing the other techniques mentioned. The results of the study also significantly display the variation in the mean values of the selected features at the three workload levels (p<0.05). This model can become the principle for defining the workload level quantification applicable to diverse fields like neuroergonomics study, intelligent assistive devices (ADs) development, blue-chip technology exploration, cognitive evaluation of students, power plant operators, traffic operators, etc.

Quantifying Cognitive Workload Using a Non-Contact Magnetocardiography (MCG) Wearable Sensor

Quantifying cognitive workload, i.e., the level of mental effort put forth by an individual in response to a cognitive task, is relevant for healthcare, training and gaming applications. However, there is currently no technology available that can readily and reliably quantify the cognitive workload of an individual in a real-world environment at a seamless way and affordable price. In this work, we overcome these limitations and demonstrate the feasibility of a magnetocardiography (MCG) sensor to reliably classify high vs. low cognitive workload while being non-contact, fully passive and low-cost, with the potential to have a wearable form factor. The operating principle relies on measuring the naturally emanated magnetic fields from the heart and subsequently analyzing the heart rate variability (HRV) matrix in three time-domain parameters: standard deviation of RR intervals (SDRR); root mean square of successive differences between heartbeats (RMSSD); and mean values of adjacent R-peaks in the cardiac signals (MeanRR). A total of 13 participants were recruited, two of whom were excluded due to low signal quality. The results show that SDRR and RMSSD achieve a 100% success rate in classifying high vs. low cognitive workload, while MeanRR achieves a 91% success rate. Tests for the same individual yield an intra-subject classification accuracy of 100% for all three HRV parameters. Future studies should leverage machine learning and advanced digital signal processing to achieve automated classification of cognitive workload and reliable operation in a natural environment.

Aviation and neurophysiology: A systematic review

This paper systematically reviews 20 years of publications (N = 54) on aviation and neurophysiology. The main goal is to provide an account of neurophysiological changes associated with flight training with the aim of identifying neurometrics indicative of pilot's flight training level and task relevant mental states, as well as to capture the current state-of-art of (neuro)ergonomic design and practice in flight training. We identified multiple candidate neurometrics of training progress and workload, such as frontal theta power, the EEG Engagement Index and the Cognitive Stability Index. Furthermore, we discovered that several types of classifiers could be used to accurately detect mental states, such as the detection of drowsiness and mental fatigue. The paper advances practical guidelines on terminology usage, simulator fidelity, and multimodality, as well as future research ideas including the potential of Virtual Reality flight simulations for training, and a brain-computer interface for flight training.

Using IoT devices for sensor-based monitoring of employees' mental workload: Investigating managers' expectations and concerns

Although the objective assessment of mental workload has been a focus of human factors research, few studies have investigated stakeholders' attitudes towards its implementation in real workplaces. The present study addresses this research gap by surveying N = 702 managers in three European countries (Germany, United Kingdom, Spain) about their expectations and concerns regarding sensor-based monitoring of employee mental workload. The data confirm the relevance of expectations regarding improvements of workplace design and employee well-being, as well as concerns about restrictions of employees' privacy and sovereignty, for the implementation of workload monitoring. Furthermore, Bayesian regression models show that the examined expectations have a substantial positive association with managers' willingness to support workload monitoring in their company. Privacy concerns are identified as a significant barrier to the acceptance of workload monitoring, both in terms of their prevalence among managers and their strong negative relationship with monitoring support.

Classification of Electrophysiological Signatures With Explainable Artificial Intelligence: The Case of Alarm Detection in Flight Simulator

Relevant sounds such as alarms are sometimes involuntarily ignored, a phenomenon called inattentional deafness. This phenomenon occurs under specific conditions including high workload (i.e., multitasking) and/or cognitive fatigue. In the context of aviation, such an error can have drastic consequences on flight safety. This study uses an oddball paradigm in which participants had to detect rare sounds in an ecological context of simulated flight. Cognitive fatigue and cognitive load were manipulated to trigger inattentional deafness, and brain activity was recorded via electroencephalography (EEG). Our results showed that alarm omission and alarm detection can be classified based on time-frequency analysis of brain activity. We reached a maximum accuracy of 76.4% when the algorithm was trained on all participants and a maximum of 90.5%, on one participant, when the algorithm was trained individually. This method can benefit from explainable artificial intelligence to develop efficient and understandable passive brain-computer interfaces, improve flight safety by detecting such attentional failures in real time, and give appropriate feedback to pilots, according to our ambitious goal, providing them with reliable and rich human/machine interactions.

Investigating mental workload-induced changes in cortical oxygenation and frontal theta activity during simulated flights

Monitoring pilots' cognitive states becomes increasingly important in aviation. Physiological measurement can detect increased mental workload (MWL) even before performance declines. Yet, changes in MWL are rarely varied systematically and few studies control for confounding effects of other cognitive states. The present study targets these shortcomings by analysing the effects of stepwise increased MWL on cortical activation, while controlling for mental fatigue (MF). 35 participants conducted a simulated flight with an incorporated adapted n-back and monitoring task. We recorded cortical activation with concurrent EEG and fNIRS measurement, performance, self-reported MWL and MF. Our results show the successful manipulation of MWL without confounding effects of MF. Higher task difficulty elicited higher subjective MWL ratings, performance decline, higher frontal theta activity and reduced frontal deoxyhaemoglobin (Hbr) concentration. Using both EEG and fNIRS, we could discriminate all induced MWL levels. fNIRS was more sensitive to tasks with low difficulty, and EEG to tasks with high difficulty. Our findings further suggest a plateau effect for high MWL that could present an upper boundary to individual cognitive capacity. Our results highlight the benefits of physiological measurement in aviation, both for assessment of cognitive states and as a data source for adaptive assistance systems.

Dual Passive Reactive Brain-Computer Interface: A Novel Approach to Human-Machine Symbiosis

The present study proposes a novel concept of neuroadaptive technology, namely a dual passive-reactive Brain-Computer Interface (BCI), that enables bi-directional interaction between humans and machines. We have implemented such a system in a realistic flight simulator using the NextMind classification algorithms and framework to decode pilots' intention (reactive BCI) and to infer their level of attention (passive BCI). Twelve pilots used the reactive BCI to perform checklists along with an anti-collision radar monitoring task that was supervised by the passive BCI. The latter simulated an automatic avoidance maneuver when it detected that pilots missed an incoming collision. The reactive BCI reached 100% classification accuracy with a mean reaction time of 1.6 s when exclusively performing the checklist task. Accuracy was up to 98.5% with a mean reaction time of 2.5 s when pilots also had to fly the aircraft and monitor the anti-collision radar. The passive BCI achieved a F1-score of 0.94. This first demonstration shows the potential of a dual BCI to improve human-machine teaming which could be applied to a variety of applications.

Zapline‐plus: A Zapline extension for automatic and adaptive removal of frequency‐specific noise artifacts in M/ EEG

Removing power line noise and other frequency‐specific artifacts from electrophysiological data without affecting neural signals remains a challenging task. Recently, an approach was introduced that combines spectral and spatial filtering to effectively remove line noise: Zapline. This algorithm, however, requires manual selection of the noise frequency and the number of spatial components to remove during spatial filtering. Moreover, it assumes that noise frequency and spatial topography are stable over time, which is often not warranted. To overcome these issues, we introduce Zapline‐plus, which allows adaptive and automatic removal of frequency‐specific noise artifacts from M/electroencephalography (EEG) and LFP data. To achieve this, our extension first segments the data into periods (chunks) in which the noise is spatially stable. Then, for each chunk, it searches for peaks in the power spectrum, and finally applies Zapline. The exact noise frequency around the found target frequency is also determined separately for every chunk to allow fluctuations of the peak noise frequency over time. The number of to‐be‐removed components by Zapline is automatically determined using an outlier detection algorithm. Finally, the frequency spectrum after cleaning is analyzed for suboptimal cleaning, and parameters are adapted accordingly if necessary before re‐running the process. The software creates a detailed plot for monitoring the cleaning. We highlight the efficacy of the different features of our algorithm by applying it to four openly available data sets, two EEG sets containing both stationary and mobile task conditions, and two magnetoencephalography sets containing strong line noise.

EEG Based Dynamic Functional Connectivity Analysis in Mental Workload Tasks with Different Types of Information

The accurate evaluation of operators' mental workload in human-machine systems plays an important role in ensuring the correct execution of tasks and the safety of operators. However, the performance of cross-task mental workload evaluation based on physiological metrics remains unsatisfactory. To explore the changes in dynamic functional connectivity properties with varying mental workload in different tasks, four mental workload tasks with different types of information were designed and a newly proposed dynamic brain network analysis method based on EEG microstate was applied in this paper. Six microstate topographies labeled as Microstate A-F were obtained to describe the task-state EEG dynamics, which was highly consistent with previous studies. Dynamic brain network analysis revealed that 15 nodes and 68 pairs of connectivity from the Frontal-Parietal region were sensitive to mental workload in all four tasks, indicating that these nodal metrics had potential to effectively evaluate mental workload in the cross-task scenario. The characteristic path length of Microstate D brain network in both Theta and Alpha bands decreased whereas the global efficiency increased significantly when the mental workload became higher, suggesting that the cognitive control network of brain tended to have higher function integration property under high mental workload state. Furthermore, by using a SVM classifier, an averaged classification accuracy of 95.8% for within-task and 80.3% for cross-task mental workload discrimination were achieved. Results implies that it is feasible to evaluate the cross-task mental workload using the dynamic functional connectivity metrics under specific microstate, which provided a new insight for understanding the neural mechanism of mental workload with different types of information.

Workshops of the Eighth International Brain-Computer Interface Meeting: BCIs: The Next Frontier

The Eighth International Brain-Computer Interface (BCI) Meeting was held June 7-9th, 2021 in a virtual format. The conference continued the BCI Meeting series' interactive nature with 21 workshops covering topics in BCI (also called brain-machine interface) research. As in the past, workshops covered the breadth of topics in BCI. Some workshops provided detailed examinations of specific methods, hardware, or processes. Others focused on specific BCI applications or user groups. Several workshops continued consensus building efforts designed to create BCI standards and increase the ease of comparisons between studies and the potential for meta-analysis and large multi-site clinical trials. Ethical and translational considerations were both the primary topic for some workshops or an important secondary consideration for others. The range of BCI applications continues to expand, with more workshops focusing on approaches that can extend beyond the needs of those with physical impairments. This paper summarizes each workshop, provides background information and references for further study, presents an overview of the discussion topics, and describes the conclusion, challenges, or initiatives that resulted from the interactions and discussion at the workshop.

Benchmarking cEEGrid and Solid Gel-Based Electrodes to Classify Inattentional Deafness in a Flight Simulator

Transfer from experiments in the laboratory to real-life tasks is challenging due notably to the inability to reproduce the complexity of multitasking dynamic everyday life situations in a standardized lab condition and to the bulkiness and invasiveness of recording systems preventing participants from moving freely and disturbing the environment. In this study, we used a motion flight simulator to induce inattentional deafness to auditory alarms, a cognitive difficulty arising in complex environments. In addition, we assessed the possibility of two low-density EEG systems a solid gel-based electrode Enobio (Neuroelectrics, Barcelona, Spain) and a gel-based cEEGrid (TMSi, Oldenzaal, Netherlands) to record and classify brain activity associated with inattentional deafness (misses vs. hits to odd sounds) with a small pool of expert participants. In addition to inducing inattentional deafness (missing auditory alarms) at much higher rates than with usual lab tasks (34.7% compared to the usual 5%), we observed typical inattentional deafness-related activity in the time domain but also in the frequency and time-frequency domains with both systems. Finally, a classifier based on Riemannian Geometry principles allowed us to obtain more than 70% of single-trial classification accuracy for both mobile EEG, and up to 71.5% for the cEEGrid (TMSi, Oldenzaal, Netherlands). These results open promising avenues toward detecting cognitive failures in real-life situations, such as real flight.

Quantifying the Effect of Noise on Cognitive Processes: A Review of Psychophysiological Correlates of Workload

Noise is present in most work environments, including emissions from machines and devices, irrelevant speech from colleagues, and traffic noise. Although it is generally accepted that noise below the permissible exposure limits does not pose a considerable risk for auditory effects like hearing impairments. Yet, noise can have a direct adverse effect on cognitive performance (non-auditory effects like workload or stress). Under certain circumstances, the observable performance for a task carried out in silence compared to noisy surroundings may not differ. One possible explanation for this phenomenon needs further investigation: individuals may invest additional cognitive resources to overcome the distraction from irrelevant auditory stimulation. Recent developments in measurements of psychophysiological correlates and analysis methods of load-related parameters can shed light on this complex interaction. These objective measurements complement subjective self-report of perceived effort by quantifying unnoticed noise-related cognitive workload. In this review, literature databases were searched for peer-reviewed journal articles that deal with an at least partially irrelevant "auditory stimulation" during an ongoing "cognitive task" that is accompanied by "psychophysiological correlates" to quantify the "momentary workload." The spectrum of assessed types of "auditory stimulations" extended from speech stimuli (varying intelligibility), oddball sounds (repeating short tone sequences), and auditory stressors (white noise, task-irrelevant real-life sounds). The type of "auditory stimulation" was related (speech stimuli) or unrelated (oddball, auditory stressor) to the type of primary "cognitive task." The types of "cognitive tasks" include speech-related tasks, fundamental psychological assessment tasks, and real-world/simulated tasks. The "psychophysiological correlates" include pupillometry and eye-tracking, recordings of brain activity (hemodynamic, potentials), cardiovascular markers, skin conductance, endocrinological markers, and behavioral markers. The prevention of negative effects on health by unexpected stressful soundscapes during mental work starts with the continuous estimation of cognitive workload triggered by auditory noise. This review gives a comprehensive overview of methods that were tested for their sensitivity as markers of workload in various auditory settings during cognitive processing.

Cross-Task Consistency of Electroencephalography-Based Mental Workload Indicators: Comparisons Between Power Spectral Density and Task-Irrelevant Auditory Event-Related Potentials

Mental workload (MWL) estimators based on ongoing electroencephalography (EEG) and event-related potentials (ERPs) have shown great potentials to build adaptive aiding systems for human-machine systems by estimating MWL in real time. However, extracting EEG features which are consistent in indicating MWL across different tasks is still one of the critical challenges. This study attempts to compare the cross-task consistency in indexing MWL variations between two commonly used EEG-based MWL indicators, power spectral density (PSD) of ongoing EEG and task-irrelevant auditory ERPs (tir-aERPs). The verbal N-back and the multi-attribute task battery (MATB), both with two difficulty levels, were employed in the experiment, along with task-irrelevant auditory probes. EEG was recorded from 17 subjects when they were performing the tasks. The tir-aERPs elicited by the auditory probes and the relative PSDs of ongoing EEG between two consecutive auditory probes were extracted and statistically analyzed to reveal the effects of MWL and task type. Discriminant analysis and support vector machine were employed to examine the generalization of tir-aERP and PSD features in indexing MWL variations across different tasks. The results showed that the amplitudes of tir-aERP components, N1, early P3a, late P3a, and the reorienting negativity, significantly decreased with the increasing MWL in both N-back and MATB. Task type had no obvious influence on the amplitudes and topological layout of the MWL-sensitive tir-aERP features. The relative PSDs in θ, α, and low β bands were also sensitive to MWL variations. However, the MWL-sensitive PSD features and their topological patterns were significantly affected by task type. The cross-task classification results based on tir-aERP features also significantly outperformed the PSD features. These results suggest that the tir-aERPs should be potentially more consistent MWL indicators across very different task types when compared to PSD. The current study may provide new insights to our understanding of the common and distinctive neuropsychological essences of MWL across different tasks.

Neural Decoding of EEG Signals with Machine Learning: A Systematic Review

Electroencephalography (EEG) is a non-invasive technique used to record the brain's evoked and induced electrical activity from the scalp. Artificial intelligence, particularly machine learning (ML) and deep learning (DL) algorithms, are increasingly being applied to EEG data for pattern analysis, group membership classification, and brain-computer interface purposes. This study aimed to systematically review recent advances in ML and DL supervised models for decoding and classifying EEG signals. Moreover, this article provides a comprehensive review of the state-of-the-art techniques used for EEG signal preprocessing and feature extraction. To this end, several academic databases were searched to explore relevant studies from the year 2000 to the present. Our results showed that the application of ML and DL in both mental workload and motor imagery tasks has received substantial attention in recent years. A total of 75% of DL studies applied convolutional neural networks with various learning algorithms, and 36% of ML studies achieved competitive accuracy by using a support vector machine algorithm. Wavelet transform was found to be the most common feature extraction method used for all types of tasks. We further examined the specific feature extraction methods and end classifier recommendations discovered in this systematic review.

Subject-Specific Cognitive Workload Classification Using EEG-Based Functional Connectivity and Deep Learning

Cognitive workload is a crucial factor in tasks involving dynamic decision-making and other real-time and high-risk situations. Neuroimaging techniques have long been used for estimating cognitive workload. Given the portability, cost-effectiveness and high time-resolution of EEG as compared to fMRI and other neuroimaging modalities, an efficient method of estimating an individual’s workload using EEG is of paramount importance. Multiple cognitive, psychiatric and behavioral phenotypes have already been known to be linked with “functional connectivity”, i.e., correlations between different brain regions. In this work, we explored the possibility of using different model-free functional connectivity metrics along with deep learning in order to efficiently classify the cognitive workload of the participants. To this end, 64-channel EEG data of 19 participants were collected while they were doing the traditional n-back task. These data (after pre-processing) were used to extract the functional connectivity features, namely Phase Transfer Entropy (PTE), Mutual Information (MI) and Phase Locking Value (PLV). These three were chosen to do a comprehensive comparison of directed and non-directed model-free functional connectivity metrics (allows faster computations). Using these features, three deep learning classifiers, namely CNN, LSTM and Conv-LSTM were used for classifying the cognitive workload as low (1-back), medium (2-back) or high (3-back). With the high inter-subject variability in EEG and cognitive workload and recent research highlighting that EEG-based functional connectivity metrics are subject-specific, subject-specific classifiers were used. Results show the state-of-the-art multi-class classification accuracy with the combination of MI with CNN at 80.87%, followed by the combination of PLV with CNN (at 75.88%) and MI with LSTM (at 71.87%). The highest subject specific performance was achieved by the combinations of PLV with Conv-LSTM, and PLV with CNN with an accuracy of 97.92%, followed by the combination of MI with CNN (at 95.83%) and MI with Conv-LSTM (at 93.75%). The results highlight the efficacy of the combination of EEG-based model-free functional connectivity metrics and deep learning in order to classify cognitive workload. The work can further be extended to explore the possibility of classifying cognitive workload in real-time, dynamic and complex real-world scenarios.

Inattentional blindness in anesthesiology: A gorilla is worth one thousand words

INTRODUCTION

People are not able to anticipate unexpected events. Inattentional blindness is demonstrated to happen not only in naïve observers engaged in an unfamiliar task but also in field experts with years of training. Anaesthesia is the perfect example of a discipline which requires a high level of attention and our aim was to evaluate if inattentional blindness can affect anesthesiologists during their daily activities.

MATERIALS AND METHODS

An online survey was distributed on Facebook between May 1, 2021 and May 31, 2021. The survey consisted of five simulated cases with questions investigating the anesthetic management of day-case surgeries. Each case had an introduction, a chest radiography, an electrocardiogram, preoperative blood testing and the last case had a gorilla embedded in the chest radiography.

RESULTS

In total 699 respondents from 17 different countries were finally included in the analysis. The main outcome was to assess the incidence of inattentional blindness. Only 34 (4.9%) respondents were able to spot the gorilla. No differences were found between anesthesiologists or residents, private or public hospitals, or between medical doctors with different experience.

DISCUSSION

Our findings assess that inattentional blindness is common in anesthesia, and ever-growing attention is deemed necessary to improve patient safety; to achieve this objective several strategies should be adopted such as an increased use of standardized protocols, promoting automation based strategies to reduce human error when performing repetitive tasks and discouraging evaluation of multiple consecutive patients in the same work shifts independently of the associated complexity.

Mental Workload Estimation Based on Physiological Features for Pilot-UAV Teaming Applications

Manned-Unmanned Teaming (MUM-T) can be defined as the teaming of aerial robots (artificial agents) along with a human pilot (natural agent), in which the human agent is not an authoritative controller but rather a cooperative team player. To our knowledge, no study has yet evaluated the impact of MUM-T scenarios on operators' mental workload (MW) using a neuroergonomic approach (i.e., using physiological measures), nor provided a MW estimation through classification applied on those measures. Moreover, the impact of the non-stationarity of the physiological signal is seldom taken into account in classification pipelines, particularly regarding the validation design. Therefore this study was designed with two goals: (i) to characterize and estimate MW in a MUM-T setting based on physiological signals; (ii) to assess the impact of the validation procedure on classification accuracy. In this context, a search and rescue (S&R) scenario was developed in which 14 participants played the role of a pilot cooperating with three UAVs (Unmanned Aerial Vehicles). Missions were designed to induce high and low MW levels, which were evaluated using self-reported, behavioral and physiological measures (i.e., cerebral, cardiac, and oculomotor features). Supervised classification pipelines based on various combinations of these physiological features were benchmarked, and two validation procedures were compared (i.e., a traditional one that does not take time into account vs. an ecological one that does). The main results are: (i) a significant impact of MW on all measures, (ii) a higher intra-subject classification accuracy (75%) reached using ECG features alone or in combination with EEG and ET ones with the Adaboost, Linear Discriminant Analysis or the Support Vector Machine classifiers. However this was only true with the traditional validation. There was a significant drop in classification accuracy using the ecological one. Interestingly, inter-subject classification with ecological validation (59.8%) surpassed both intra-subject with ecological and inter-subject with traditional validation. These results highlight the need for further developments to perform MW monitoring in such operational contexts.

Mental workload classification based on ignored auditory probes and spatial covariance

Objective.Estimation of mental workload (MWL) levels by electroencephalography (EEG)-based mental state monitoring systems has been widely explored. Using event-related potentials (ERPs), elicited by ignored auditory probes, minimizes intrusiveness and has shown high performance for estimating MWL level when tested in laboratory situations. However, when facing real-world applications, the characteristics of ERP waveforms, like latency and amplitude, are often affected by noise, which leads to a decrease in classification performance. One approach to mitigating this is using spatial covariance, which is less sensitive to latency and amplitude distortion. In this study, we used ignored auditory probes in a single-stimulus paradigm and tested Riemannian processed covariance-based features for MWL level estimation in a realistic flight-control task.Approach.We recorded EEG data with an eight-channel system from participants while they performed a simulated drone-control task and manipulated MWL levels (high and low) by task difficulty. We compared support vector machine classification performance based on frequency band power features versus features generated via the Riemannian log map operator from spatial covariance matrices. We also compared accuracy of using data segmented as auditory ERPs versus non-ERPs, for which data windows did not overlap with the ERPs.Main results.Classification accuracy of both types of features showed no significant difference between ERPs and non-ERPs. When we ignore auditory stimuli to perform continuous decoding, covariance-based features in the gamma band had area under the receiver-operating-characteristic curve (AUC) of 0.883, which was significantly higher than band power features (AUC = 0.749).Significance.This study demonstrates that Riemannian-processed covariance features are viable for MWL classification under a realistic experimental scenario.

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains a high level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The results from the online validation of the ANFIS models demonstrated that five of the ANFIS models (containing different feature combinations of eye activity and control input features) all demonstrated good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be seen with the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all the ANFIS models tested. Hence, this study demonstrates the ability for online multimodal fusion from features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.

Online recognition of handwritten characters from scalp-recorded brain activities during handwriting

Objective.Brain-computer interfaces aim to build an efficient communication with the world using neural signals, which may bring great benefits to human society, especially to people with physical impairments. To date, the ability to translate brain signals to effective communication outcome remains low. This work explores whether the handwriting process could serve as a potential interface with high performance. To this end, we first examined how much the scalp-recorded brain signals encode information related to handwriting and whether it is feasible to precisely retrieve the handwritten content solely from the scalp-recorded electrical data.Approach.Five participants were instructed to write the sentence 'HELLO, WORLD!' repeatedly on a tablet while their brain signals were simultaneously recorded by electroencephalography (EEG). The EEG signals were first decomposed by independent component analysis for extracting features to be used to train a convolutional neural network (CNN) to recognize the written symbols.Main results.The accuracy of the CNN-based classifier trained and applied on the same participant (training and test data separated) ranged from 76.8% to 97.0%. The accuracy of cross-participant application was more diverse, ranging from 14.7% to 58.7%. These results showed the possibility of recognizing the handwritten content directly from the scalp level brain signal. A demonstration of the recognition system in an online mode was presented. The major factor that grounded the recognition was the close association between the rich dynamics of electroencephalogram source activities and the kinematic information during the handwriting movements.Significance.This work revealed an explicit and precise mapping between scalp-level electrophysiological signals and linguistic information conveyed by handwriting, which provided a novel approach to developing brain computer interfaces that focus on semantic communication.

A BCI Based Alerting System for Attention Recovery of UAV Operators

As unmanned aerial vehicles have become popular, the number of accidents caused by an operator's inattention have increased. To prevent such accidents, the operator should maintain an attention status. However, limited research has been conducted on the brain-computer interface (BCI)-based system with an alerting module for the operator's attention recovery of unmanned aerial vehicles. Therefore, we introduce a detection and alerting system that prevents an unmanned aerial vehicle operator from falling into inattention status by using the operator's electroencephalogram signal. The proposed system consists of the following three components: a signal processing module, which collects and preprocesses an electroencephalogram signal of an operator, an inattention detection module, which determines whether an inattention status occurred based on the preprocessed signal, and, lastly, an alert providing module that presents stimulus to an operator when inattention is detected. As a result of evaluating the performance with a real-world dataset, it was shown that the proposed system successfully contributed to the recovery of operator attention in the evaluating dataset, although statistical significance could not be established due to the small number of subjects.

Mobile ear-EEG to study auditory attention in everyday life

Most research investigating auditory perception is conducted in controlled laboratory settings, potentially restricting its generalizability to the complex acoustic environment outside the lab. The present study, in contrast, investigated auditory attention with long-term recordings (> 6 h) beyond the lab using a fully mobile, smartphone-based ear-centered electroencephalography (EEG) setup with minimal restrictions for participants. Twelve participants completed iterations of two variants of an oddball task where they had to react to target tones and to ignore standard tones. A rapid variant of the task (tones every 2 s, 5 min total time) was performed seated and with full focus in the morning, around noon and in the afternoon under controlled conditions. A sporadic variant (tones every minute, 160 min total time) was performed once in the morning and once in the afternoon while participants followed their normal office day routine. EEG data, behavioral data, and movement data (with a gyroscope) were recorded and analyzed. The expected increased amplitude of the P3 component in response to the target tone was observed for both the rapid and the sporadic oddball. Miss rates were lower and reaction times were faster in the rapid oddball compared to the sporadic one. The movement data indicated that participants spent most of their office day at relative rest. Overall, this study demonstrated that it is feasible to study auditory perception in everyday life with long-term ear-EEG.

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010-2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges.

How Can Physiological Computing Benefit Human-Robot Interaction?

As systems grow more automatized, the human operator is all too often overlooked. Although human-robot interaction (HRI) can be quite demanding in terms of cognitive resources, the mental states (MS) of the operators are not yet taken into account by existing systems. As humans are no providential agents, this lack can lead to hazardous situations. The growing number of neurophysiology and machine learning tools now allows for efficient operators’ MS monitoring. Sending feedback on MS in a closed-loop solution is therefore at hand. Involving a consistent automated planning technique to handle such a process could be a significant asset. This perspective article was meant to provide the reader with a synthesis of the significant literature with a view to implementing systems that adapt to the operator’s MS to improve human-robot operations’ safety and performance. First of all, the need for this approach is detailed regarding remote operation, an example of HRI. Then, several MS identified as crucial for this type of HRI are defined, along with relevant electrophysiological markers. A focus is made on prime degraded MS linked to time-on-task and task demands, as well as collateral MS linked to system outputs (i.e., feedback and alarms). Lastly, the principle of symbiotic HRI is detailed and one solution is proposed to include the operator state vector into the system using a mixed-initiative decisional framework to drive such an interaction.

Aiming at ecological validity-Midfrontal theta oscillations in a toy gun shooting task

Laboratory electroencephalography (EEG) studies have already provided important insights into the neuronal mechanisms of performance monitoring. However, to our knowledge no study so far has examined neuronal correlates of performance monitoring using an ecologically valid task outside a typical laboratory setting. Therefore, we examined midfrontal theta and the feedback-related negativity (FRN) using mobile EEG in a physical shooting task within an ecologically valid environment with highly dynamical visual feedback. Participants shot a target using a toy gun while moving and looking around freely. Shots that missed the target evoked stronger midfrontal theta activity than hits and this response was rather phase-unlocked. There was no difference between misses and hits in the FRN. The results raise the question whether the absence of certain ERP components like the FRN could be due to methodological reasons or to the fact that partially different neuronal processes may be activated in the laboratory as compared to more ecologically valid tasks. Overall, our results indicate that crucial neurocognitive processes of performance monitoring can be assessed in highly dynamic and ecologically valid settings by mobile EEG.

A brain-computer interface for the continuous, real-time monitoring of working memory load in real-world environments

We developed a brain-computer interface (BCI) able to continuously monitor working memory (WM) load in real-time (considering the last 2.5 s of brain activity). The BCI is based on biomarkers derived from spectral properties of non-invasive electroencephalography (EEG), subsequently classified by a linear discriminant analysis classifier. The BCI was trained on a visual WM task, tested in a real-time visual WM task, and further validated in a real-time cross task (mental arithmetic). Throughout each trial of the cross task, subjects were given real or sham feedback about their WM load. At the end of the trial, subjects were asked whether the feedback provided was real or sham. The high rate of correct answers provided by the subjects validated not only the global behaviour of the WM-load feedback, but also its real-time dynamics. On average, subjects were able to provide a correct answer 82% of the time, with one subject having 100% accuracy. Possible cognitive and motor confounding factors were disentangled to support the claim that our EEG-based markers correspond indeed to WM.

Neuroergonomic Assessment of Hot Beverage Preparation and Consumption: An EEG and EDA Study

Neuroergonomics is an emerging field that investigates the human brain about behavioral performance in natural environments and everyday settings. This study investigated the body and brain activity correlates of a typical daily activity, hot beverage preparation, and consumption in a realistic office environment where participants performed natural daily tasks. Using wearable, battery operated and wireless Electroencephalogram (EEG) and Electrodermal activity (EDA) sensors, neural and physiological responses were measured in untethered, freely moving participants who prepared hot beverages using two different machines (a market leader and follower as determined by annual US sales). They later consumed the drinks they had prepared in three blocks. Emotional valence was estimated using frontal asymmetry in EEG alpha band power and emotional arousal was estimated from EDA tonic and phasic activity. Results from 26 participants showed that the market-leading coffee machine was more efficient to use based on self-reports, behavioral performance measures, and there were significant within-subject differences in valence between the two machine use. Moreover, the market leader user interface led to greater self-reported product preference, which was further supported by significant differences in measured arousal and valence (EDA and EEG, respectively) during coffee production and consumption. This is the first study that uses a multimodal and comprehensive assessment of coffee machine use and beverage consumption in a naturalistic work environment. Approaches described in this study can be adapted in the future to other task-specific machine usability and consumer neuroscience studies.

Towards Mixed-Initiative Human-Robot Interaction: Assessment of Discriminative Physiological and Behavioral Features for Performance Prediction

The design of human–robot interactions is a key challenge to optimize operational performance. A promising approach is to consider mixed-initiative interactions in which the tasks and authority of each human and artificial agents are dynamically defined according to their current abilities. An important issue for the implementation of mixed-initiative systems is to monitor human performance to dynamically drive task allocation between human and artificial agents (i.e., robots). We, therefore, designed an experimental scenario involving missions whereby participants had to cooperate with a robot to fight fires while facing hazards. Two levels of robot automation (manual vs. autonomous) were randomly manipulated to assess their impact on the participants’ performance across missions. Cardiac activity, eye-tracking, and participants’ actions on the user interface were collected. The participants performed differently to an extent that we could identify high and low score mission groups that also exhibited different behavioral, cardiac and ocular patterns. More specifically, our findings indicated that the higher level of automation could be beneficial to low-scoring participants but detrimental to high-scoring ones, and vice versa. In addition, inter-subject single-trial classification results showed that the studied behavioral and physiological features were relevant to predict mission performance. The highest average balanced accuracy (74%) was reached using the features extracted from all input devices. These results suggest that an adaptive HRI driving system, that would aim at maximizing performance, would be capable of analyzing such physiological and behavior markers online to further change the level of automation when it is relevant for the mission purpose.