A Machine Learning Approach to the Detection of Pilot's Reaction to Unexpected Events Based on EEG Signals

An exploratory study of pilot EEG features during the climb and descent phases of flight

OBJECTIVES

The actions and decisions of pilots are directly related to aviation safety. Therefore, understanding the neurological and cognitive processes of pilots during flight is essential. This study aims to investigate the EEG signals of pilots to understand the characteristic changes during the climb and descent stages of flight.

METHODS

By performing wavelet packet decomposition on the EEG signals, we examined EEG maps during these critical phases and analyzed changes in signal intensity. To delve deeper, we calculated the log-transformed power of electroencephalograms to investigate the EEG responses under different flight conditions. Additionally, we conducted EEG spectral coherence analysis to evaluate the degree of synchronization between different electrodes during climb and descent.

RESULTS

This analysis helps us understand the functional connectivity changes in various brain regions during these phases. Understanding these complex interactions is crucial, as it provides insights into the cognitive processes of pilots during the critical climb and descent stages of flight, contributing to enhanced aviation safety.

CONCLUSIONS

By identifying how brain activity fluctuates during these phases, we can better comprehend pilots' decision-making processes, ultimately leading to the development of more effective training programs and safety protocols. This research underscores the importance of neurological studies in safety.

Detection of Pilots' Psychological Workload during Turning Phases Using EEG Characteristics

Pilot behavior is crucial for aviation safety. This study aims to investigate the EEG characteristics of pilots, refine training assessment methodologies, and bolster flight safety measures. The collected EEG signals underwent initial preprocessing. The EEG characteristic analysis was performed during left and right turns, involving the calculation of the energy ratio of beta waves and Shannon entropy. The psychological workload of pilots during different flight phases was quantified as well. Based on the EEG characteristics, the pilots' psychological workload was classified through the use of a support vector machine (SVM). The study results showed significant changes in the energy ratio of beta waves and Shannon entropy during left and right turns compared to the cruising phase. Additionally, the pilots' psychological workload was found to have increased during these turning phases. Using support vector machines to detect the pilots' psychological workload, the classification accuracy for the training set was 98.92%, while for the test set, it was 93.67%. This research holds significant importance in understanding pilots' psychological workload.

A scoping review on the use of consumer-grade EEG devices for research

BACKGROUND

Commercial electroencephalography (EEG) devices have become increasingly available over the last decade. These devices have been used in a wide variety of fields ranging from engineering to cognitive neuroscience.

PURPOSE

The aim of this study was to chart peer-review articles that used consumer-grade EEG devices to collect neural data. We provide an overview of the research conducted with these relatively more affordable and user-friendly devices. We also inform future research by exploring the current and potential scope of consumer-grade EEG.

METHODS

We followed a five-stage methodological framework for a scoping review that included a systematic search using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines. We searched the following online databases: PsycINFO, MEDLINE, Embase, Web of Science, and IEEE Xplore. We charted study data according to application (BCI, experimental research, validation, signal processing, and clinical) and location of use as indexed by the first author's country.

RESULTS

We identified 916 studies that used data recorded with consumer-grade EEG: 531 were reported in journal articles and 385 in conference papers. Emotiv devices were used most, followed by the NeuroSky MindWave, OpenBCI, interaXon Muse, and MyndPlay Mindband. The most common usage was for brain-computer interfaces, followed by experimental research, signal processing, validation, and clinical purposes.

CONCLUSIONS

Consumer-grade EEG is a useful tool for neuroscientific research and will likely continue to be used well into the future. Our study provides a comprehensive review of their application, as well as future directions for researchers who plan to use these devices.

Illuminating the Neural Landscape of Pilot Mental States: A Convolutional Neural Network Approach with Shapley Additive Explanations Interpretability

Predicting pilots' mental states is a critical challenge in aviation safety and performance, with electroencephalogram data offering a promising avenue for detection. However, the interpretability of machine learning and deep learning models, which are often used for such tasks, remains a significant issue. This study aims to address these challenges by developing an interpretable model to detect four mental states-channelised attention, diverted attention, startle/surprise, and normal state-in pilots using EEG data. The methodology involves training a convolutional neural network on power spectral density features of EEG data from 17 pilots. The model's interpretability is enhanced via the use of SHapley Additive exPlanations values, which identify the top 10 most influential features for each mental state. The results demonstrate high performance in all metrics, with an average accuracy of 96%, a precision of 96%, a recall of 94%, and an F1 score of 95%. An examination of the effects of mental states on EEG frequency bands further elucidates the neural mechanisms underlying these states. The innovative nature of this study lies in its combination of high-performance model development, improved interpretability, and in-depth analysis of the neural correlates of mental states. This approach not only addresses the critical need for effective and interpretable mental state detection in aviation but also contributes to our understanding of the neural underpinnings of these states. This study thus represents a significant advancement in the field of EEG-based mental state detection.

Multimodal Approach for Pilot Mental State Detection Based on EEG

The safety of flight operations depends on the cognitive abilities of pilots. In recent years, there has been growing concern about potential accidents caused by the declining mental states of pilots. We have developed a novel multimodal approach for mental state detection in pilots using electroencephalography (EEG) signals. Our approach includes an advanced automated preprocessing pipeline to remove artefacts from the EEG data, a feature extraction method based on Riemannian geometry analysis of the cleaned EEG data, and a hybrid ensemble learning technique that combines the results of several machine learning classifiers. The proposed approach provides improved accuracy compared to existing methods, achieving an accuracy of 86% when tested on cleaned EEG data. The EEG dataset was collected from 18 pilots who participated in flight experiments and publicly released at NASA's open portal. This study presents a reliable and efficient solution for detecting mental states in pilots and highlights the potential of EEG signals and ensemble learning algorithms in developing cognitive cockpit systems. The use of an automated preprocessing pipeline, feature extraction method based on Riemannian geometry analysis, and hybrid ensemble learning technique set this work apart from previous efforts in the field and demonstrates the innovative nature of the proposed approach.

Analysis of Relation between Brainwave Activity and Reaction Time of Short-Haul Pilots Based on EEG Data

The purpose of this research is to examine and assess the relation between a pilot's concentration and reaction time with specific brain activity during short-haul flights. Participants took part in one-hour long flight sessions performed on the FNPT II class flight simulator. Subjects were instructed to respond to unexpected events that occurred during the flight. The brainwaves of each participant were recorded with the Emotiv EPOC+ Scientific Contextual EEG device. The majority of participants showed a statistically significant, positive correlation between Theta Power in the frontal lobe and response time. Additionally, most subjects exhibited statistically significant, positive correlations between band-power and reaction times in the Theta range for the temporal and parietal lobes. Statistically significant event-related changes (ERC) were observed for the majority of subjects in the frontal lobe for Theta frequencies, Beta waves in the frontal lobe and in all lobes for the Gamma band. Notably, significant ERC was also observed for Theta and Beta frequencies in the temporal and occipital Lobes, Alpha waves in the frontal, parietal and occipital lobes for most participants. A difference in brain activity patterns was observed, depending on the performance in time-restricted tasks.

Aviation and neurophysiology: A systematic review

This paper systematically reviews 20 years of publications (N = 54) on aviation and neurophysiology. The main goal is to provide an account of neurophysiological changes associated with flight training with the aim of identifying neurometrics indicative of pilot's flight training level and task relevant mental states, as well as to capture the current state-of-art of (neuro)ergonomic design and practice in flight training. We identified multiple candidate neurometrics of training progress and workload, such as frontal theta power, the EEG Engagement Index and the Cognitive Stability Index. Furthermore, we discovered that several types of classifiers could be used to accurately detect mental states, such as the detection of drowsiness and mental fatigue. The paper advances practical guidelines on terminology usage, simulator fidelity, and multimodality, as well as future research ideas including the potential of Virtual Reality flight simulations for training, and a brain-computer interface for flight training.

A Home-Based Approach to Auditory Brainstem Response Measurement: Proof-of-Concept and Practical Guidelines

Broad-scale neuroscientific investigations of diverse human populations are difficult to implement. This is because the primary neuroimaging methods (magnetic resonance imaging, electroencephalography [EEG]) historically have not been portable, and participants may be unable or unwilling to travel to test sites. Miniaturization of EEG technologies has now opened the door to neuroscientific fieldwork, allowing for easier access to under-represented populations. Recent efforts to conduct auditory neuroscience outside a laboratory setting are reviewed and then an in-home technique for recording auditory brainstem responses (ABRs) and frequency-following responses (FFRs) in a home setting is introduced. As a proof of concept, we have conducted two in-home electrophysiological studies: one in 27 children aged 6 to 16 years (13 with autism spectrum disorder) and another in 12 young adults aged 18 to 27 years, using portable electrophysiological equipment to record ABRs and FFRs to click and speech stimuli, spanning rural and urban and multiple homes and testers. We validate our fieldwork approach by presenting waveforms and data on latencies and signal-to-noise ratio. Our findings demonstrate the feasibility and utility of home-based ABR/FFR techniques, paving the course for larger fieldwork investigations of populations that are difficult to test or recruit. We conclude this tutorial with practical tips and guidelines for recording ABRs and FFRs in the field and discuss possible clinical and research applications of this approach.

Analysis of pilots' EEG map in take-off and landing tasks

The take-off and landing phases are considered the critical stages of aircraft flight. To ensure flight efficiency and safety in the critical stages, this research proposes a method for analyzing and monitoring pilot flight status by β-wave. The focus of the study is β potential changes on the EEG map. First, the proportion of β-wave in the electroencephalogram (EEG) of pilots during take-off and landing increases significantly. Second, the EEG map accurately and intuitively reflects the spatial distribution of potential changes in brain regions. Finally, correlation and machine learning are used for further research of β-wave. The conclusions show that the significant changes in the β-wave caused by take-off and landing can be seen in the EEG map to identify and adjust the pilot's state. Therefore, this research provides more accurate and effective reference information (like the EEG map, correlation and machine learning) for efficient and safe flight training in the critical stages.

Brain-Computer Interface: Advancement and Challenges

Brain-Computer Interface (BCI) is an advanced and multidisciplinary active research domain based on neuroscience, signal processing, biomedical sensors, hardware, etc. Since the last decades, several groundbreaking research has been conducted in this domain. Still, no comprehensive review that covers the BCI domain completely has been conducted yet. Hence, a comprehensive overview of the BCI domain is presented in this study. This study covers several applications of BCI and upholds the significance of this domain. Then, each element of BCI systems, including techniques, datasets, feature extraction methods, evaluation measurement matrices, existing BCI algorithms, and classifiers, are explained concisely. In addition, a brief overview of the technologies or hardware, mostly sensors used in BCI, is appended. Finally, the paper investigates several unsolved challenges of the BCI and explains them with possible solutions.

A Multiscale Topographical Analysis Based on Morphological Information: The HEVC Multiscale Decomposition

In this paper, we evaluate the effect of scale analysis as well as the filtering process on the performances of an original compressed-domain classifier in the field of material surface topographies classification. Each surface profile is multiscale analyzed by using a Gaussian Filter analyzing method to be decomposed into three multiscale filtered image types: Low-pass (LP), Band-pass (BP), and High-pass (HP) filtered versions, respectively. The complete set of filtered image data constitutes the collected database. First, the images are lossless compressed using the state-of-the art High-efficiency video coding (HEVC) video coding standard. Then, the Intra-Prediction Modes Histogram (IPHM) feature descriptor is computed directly in the compressed domain from each HEVC compressed image. Finally, we apply the IPHM feature descriptors as an input of a Support Vector Machine (SVM) classifier. SVM is introduced here to strengthen the performances of the proposed classification system thanks to the powerful properties of machine learning tools. We evaluate the proposed solution we called "HEVC Multiscale Decomposition" (HEVC-MD) on a huge database of nearly 42,000 multiscale topographic images. A simple preliminary version of the algorithm reaches an accuracy of 52%. We increase this accuracy to 70% by using the multiscale analysis of the high-frequency range HP filtered image data sets. Finally, we verify that considering only the highest-scale analysis of low-frequency range LP was more appropriate for classifying our six surface topographies with an accuracy of up to 81%. To compare these new topographical descriptors to those conventionally used, SVM is applied on a set of 34 roughness parameters defined on the International Standard GPS ISO 25178 (Geometrical Product Specification), and one obtains accuracies of 38%, 52%, 65%, and 57% respectively for Sa, multiscale Sa, 34 roughness parameters, and multiscale ones. Compared to conventional roughness descriptors, the HEVC-MD descriptors increase surfaces discrimination from 65% to 81%.

Prediction of Pilot's Reaction Time Based on EEG Signals

The main hypothesis of this work is that the time of delay in reaction to an unexpected event can be predicted on the basis of the brain activity recorded prior to that event. Such mental activity can be represented by electroencephalographic data. To test this hypothesis, we conducted a novel experiment involving 19 participants that took part in a 2-h long session of simulated aircraft flights. An EEG signal processing pipeline is proposed that consists of signal preprocessing, extracting bandpass features, and using regression to predict the reaction times. The prediction algorithms that are used in this study are the Least Absolute Shrinkage Operator and its Least Angle Regression modification, as well as Kernel Ridge and Radial Basis Support Vector Machine regression. The average Mean Absolute Error obtained across the 19 subjects was 114 ms. The present study demonstrates, for the first time, that it is possible to predict reaction times on the basis of EEG data. The presented solution can serve as a foundation for a system that can, in the future, increase the safety of air traffic.

EEG-Based Brain-Computer Interfaces Using Motor-Imagery: Techniques and Challenges

Electroencephalography (EEG)-based brain-computer interfaces (BCIs), particularly those using motor-imagery (MI) data, have the potential to become groundbreaking technologies in both clinical and entertainment settings. MI data is generated when a subject imagines the movement of a limb. This paper reviews state-of-the-art signal processing techniques for MI EEG-based BCIs, with a particular focus on the feature extraction, feature selection and classification techniques used. It also summarizes the main applications of EEG-based BCIs, particularly those based on MI data, and finally presents a detailed discussion of the most prevalent challenges impeding the development and commercialization of EEG-based BCIs.