A passive brain–computer interface application for the mental workload assessment on professional air traffic controllers during realistic air traffic control tasks

Classification of mental workload using brain connectivity and machine learning on electroencephalogram data

Mental workload refers to the cognitive effort required to perform tasks, and it is an important factor in various fields, including system design, clinical medicine, and industrial applications. In this paper, we propose innovative methods to assess mental workload from EEG data that use effective brain connectivity for the purpose of extracting features, a hierarchical feature selection algorithm to select the most significant features, and finally machine learning models. We have used the Simultaneous Task EEG Workload (STEW) dataset, an open-access collection of raw EEG data from 48 subjects. We extracted brain-effective connectivities by the direct directed transfer function and then selected the top 30 connectivities for each standard frequency band. Then we applied three feature selection algorithms (forward feature selection, Relief-F, and minimum-redundancy-maximum-relevance) on the top 150 features from all frequencies. Finally, we applied sevenfold cross-validation on four machine learning models (support vector machine (SVM), linear discriminant analysis, random forest, and decision tree). The results revealed that SVM as the machine learning model and forward feature selection as the feature selection method work better than others and could classify the mental workload levels with accuracy equal to 89.53% (± 1.36).

Enhancing EEG-based cross-day mental workload classification using periodic component of power spectrum

Objective. The day-to-day variability of electroencephalogram (EEG) poses a significant challenge to decode human brain activity in EEG-based passive brain-computer interfaces (pBCIs). Conventionally, a time-consuming calibration process is required to collect data from users on a new day to ensure the performance of the machine learning-based decoding model, which hinders the application of pBCIs to monitor mental workload (MWL) states in real-world settings.Approach. This study investigated the day-to-day stability of the raw power spectral density (PSD) and their periodic and aperiodic components decomposed by the Fitting Oscillations and One-Over-F algorithm. In addition, we validated the feasibility of using periodic components to improve cross-day MWL classification performance.Main results. Compared to the raw PSD (69.9% ± 18.5%) and the aperiodic component (69.4% ± 19.2%), the periodic component had better day-to-day stability and significantly higher cross-day classification accuracy (84.2% ± 11.0%).Significance. These findings indicate that periodic components of EEG have the potential to be applied in decoding brain states for more robust pBCIs.

Developing cognitive workload and performance evaluation models using functional brain network analysis

Cognition, defined as the ability to learn, remember, sustain attention, make decisions, and solve problems, is essential in daily activities and in learning new skills. The purpose of this study was to develop cognitive workload and performance evaluation models using features that were extracted from Electroencephalogram (EEG) data through functional brain network and spectral analyses. The EEG data were recorded from 124 brain areas of 26 healthy participants conducting two cognitive tasks on a robot simulator. The functional brain network and Power Spectral Density features were extracted from EEG data using coherence and spectral analyses, respectively. Participants reported their perceived cognitive workload using the SURG-TLX questionnaire after each exercise, and the simulator generated actual performance scores. The extracted features, actual performance scores, and subjectively assessed cognitive workload values were used to develop linear models for evaluating performance and cognitive workload. Furthermore, the Pearson correlation was used to find the correlation between participants' age, performance, and cognitive workload. The findings demonstrated that combined EEG features retrieved from spectral analysis and functional brain networks can be used to evaluate cognitive workload and performance. The cognitive workload in conducting only Matchboard level 3, which is more challenging than Matchboard level 2, was correlated with age (0.54, p-value = 0.01). This finding may suggest playing more challenging computer games are more helpful in identifying changes in cognitive workload caused by aging. The findings could open the door for a new era of objective evaluation and monitoring of cognitive workload and performance.

Neurophysiological Assessment of An Innovative Maritime Safety System in Terms of Ship Operators' Mental Workload, Stress, and Attention in the Full Mission Bridge Simulator

The current industrial environment relies heavily on maritime transportation. Despite the continuous technological advances for the development of innovative safety software and hardware systems, there is a consistent gap in the scientific literature regarding the objective evaluation of the performance of maritime operators. The human factor is profoundly affected by changes in human performance or psychological state. The difficulty lies in the fact that the technology, tools, and protocols for investigating human performance are not fully mature or suitable for experimental investigation. The present research aims to integrate these two concepts by (i) objectively characterizing the psychological state of mariners, i.e., mental workload, stress, and attention, through their electroencephalographic (EEG) signal analysis, and (ii) validating an innovative safety framework countermeasure, defined as Human Risk-Informed Design (HURID), through the aforementioned neurophysiological approach. The proposed study involved 26 mariners within a high-fidelity bridge simulator while encountering collision risk in congested waters with and without the HURID. Subjective, behavioral, and neurophysiological data, i.e., EEG, were collected throughout the experimental activities. The results showed that the participants experienced a statistically significant higher mental workload and stress while performing the maritime activities without the HURID, while their attention level was statistically lower compared to the condition in which they performed the experiments with the HURID (all p < 0.05). Therefore, the presented study confirmed the effectiveness of the HURID during maritime operations in critical scenarios and led the way to extend the neurophysiological evaluation of the HFs of maritime operators during the performance of critical and/or standard shipboard tasks.

Music in noise recognition: An EEG study of listening effort in cochlear implant users and normal hearing controls

Despite the plethora of studies investigating listening effort and the amount of research concerning music perception by cochlear implant (CI) users, the investigation of the influence of background noise on music processing has never been performed. Given the typical speech in noise recognition task for the listening effort assessment, the aim of the present study was to investigate the listening effort during an emotional categorization task on musical pieces with different levels of background noise. The listening effort was investigated, in addition to participants' ratings and performances, using EEG features known to be involved in such phenomenon, that is alpha activity in parietal areas and in the left inferior frontal gyrus (IFG), that includes the Broca's area. Results showed that CI users performed worse than normal hearing (NH) controls in the recognition of the emotional content of the stimuli. Furthermore, when considering the alpha activity corresponding to the listening to signal to noise ratio (SNR) 5 and SNR10 conditions subtracted of the activity while listening to the Quiet condition-ideally removing the emotional content of the music and isolating the difficulty level due to the SNRs- CI users reported higher levels of activity in the parietal alpha and in the homologous of the left IFG in the right hemisphere (F8 EEG channel), in comparison to NH. Finally, a novel suggestion of a particular sensitivity of F8 for SNR-related listening effort in music was provided.

Analysis of Relation between Brainwave Activity and Reaction Time of Short-Haul Pilots Based on EEG Data

The purpose of this research is to examine and assess the relation between a pilot's concentration and reaction time with specific brain activity during short-haul flights. Participants took part in one-hour long flight sessions performed on the FNPT II class flight simulator. Subjects were instructed to respond to unexpected events that occurred during the flight. The brainwaves of each participant were recorded with the Emotiv EPOC+ Scientific Contextual EEG device. The majority of participants showed a statistically significant, positive correlation between Theta Power in the frontal lobe and response time. Additionally, most subjects exhibited statistically significant, positive correlations between band-power and reaction times in the Theta range for the temporal and parietal lobes. Statistically significant event-related changes (ERC) were observed for the majority of subjects in the frontal lobe for Theta frequencies, Beta waves in the frontal lobe and in all lobes for the Gamma band. Notably, significant ERC was also observed for Theta and Beta frequencies in the temporal and occipital Lobes, Alpha waves in the frontal, parietal and occipital lobes for most participants. A difference in brain activity patterns was observed, depending on the performance in time-restricted tasks.

Evaluation of Single-Trial Classification to Control a Visual ERP-BCI under a Situation Awareness Scenario

An event-related potential (ERP)-based brain-computer interface (BCI) can be used to monitor a user's cognitive state during a surveillance task in a situational awareness context. The present study explores the use of an ERP-BCI for detecting new planes in an air traffic controller (ATC). Two experiments were conducted to evaluate the impact of different visual factors on target detection. Experiment 1 validated the type of stimulus used and the effect of not knowing its appearance location in an ERP-BCI scenario. Experiment 2 evaluated the effect of the size of the target stimulus appearance area and the stimulus salience in an ATC scenario. The main results demonstrate that the size of the plane appearance area had a negative impact on the detection performance and on the amplitude of the P300 component. Future studies should address this issue to improve the performance of an ATC in stimulus detection using an ERP-BCI.

EEG-based detection of modality-specific visual and auditory sensory processing

Objective.A passive brain-computer interface (pBCI) is a system that enhances a human-machine interaction by monitoring the mental state of the user and, based on this implicit information, making appropriate modifications to the interaction. Key to the development of such a system is the ability to reliably detect the mental state of interest via neural signals. Many different mental states have been investigated, including fatigue, attention and various emotions, however one of the most commonly studied states is mental workload, i.e. the amount of attentional resources required to perform a task. The emphasis of mental workload studies to date has been almost exclusively on detecting and predicting the 'level' of cognitive resources required (e.g. high vs. low), but we argue that having information regarding the specific 'type' of resources (e.g. visual or auditory) would allow the pBCI to apply more suitable adaption techniques than would be possible knowing just the overall workload level.Approach.15 participants performed carefully designed visual and auditory tasks while electroencephalography (EEG) data was recorded. The tasks were designed to be as similar as possible to one another except for the type of attentional resources required. The tasks were performed at two different levels of demand. Using traditional machine learning algorithms, we investigated, firstly, if EEG can be used to distinguish between auditory and visual processing tasks and, secondly, what effect level of sensory processing demand has on the ability to distinguish between auditory and visual processing tasks.Main results.The results show that at the high level of demand, the auditory vs. visual processing tasks could be distinguished with an accuracy of 77.1% on average. However, in the low demand condition in this experiment, the tasks were not classified with an accuracy exceeding chance.Significance.These results support the feasibility of developing a pBCI for detecting not only the level, but also the type, of attentional resources being required of the user at a given time. Further research is required to determine if there is a threshold of demand under which the type of sensory processing cannot be detected, but even if that is the case, these results are still promising since it is the high end of demand that is of most concern in safety critical scenarios. Such a BCI could help improve safety in high risk occupations by initiating the most effective and efficient possible adaptation strategies when high workload conditions are detected.

Domain- and task-analytic workload (DTAW) method: a methodology for predicting mental workload during severe accidents in nuclear power plants

Excessive mental workload reduces operators' performance and threatens the safety of nuclear power plants (NPPs) in severe accident management (SAM). Given the lack of suitable mental workload measurement methods for SAM tasks, we proposed a Domain- and Task-Analytic Workload (DTAW) method to predict SAM workload. The DTAW method is developed in three stages: scenario construction based on work domain analysis, task analysis, and workload estimation with eight workload components scored through task-analytic and projective methods. To demonstrate its utility, we applied the method to construct two SAM scenarios and predict the mental workload demand of operators in these scenarios as compared to two design basis accident scenarios. With statistical analysis, the DTAW method can predict the overall subjective workload rated by NPP operators, be used to identify high-load tasks, cluster tasks with similar workload patterns, and provide direct implications for improving SAM strategies and supporting systems.Practitioner summary: To predict mental workload in severe accident management (SAM) scenarios in nuclear power plants, we proposed an analytic method and applied it to estimate mental workload in two SAM scenarios and two design basis accident (DBA) scenarios. We found that the workload pattern in SAM scenarios is different from that in DBA scenarios.

Comparing online cognitive load on mobile versus PC-based devices

Navigating the web represents a complex cognitive activity that requires effective integration of different stimuli and the correct functioning of numerous cognitive abilities (including attention, perception, and working memory). Despite the potential relevance of the topic, numerous limitations are present throughout the literature about the cognitive load during online activities. The main aim of this study is to investigate cognitive load during comprehension and information-seeking tasks. In particular, we here focus on the comparison of the cognitive load required while performing those tasks using mobile or PC-based devices. This topic has become even more crucial due to the massive adoption of smart working and distance learning during the COVID-19 pandemic. A great effort is nowadays devoted to the detection and quantification of stressful states induced by working and learning activities. Continuous stress and excessive cognitive load are two of the main causes of mental and physical illnesses such as depression or anxiety. Cognitive load was measured through electroencephalography (EEG), acquired via a low-cost wireless EEG headset. Two different tasks were considered: reading comprehension (CO) of online text and online information-seeking (IS). Moreover, two experimental conditions were compared, administering the two tasks using mobile (MB) and desktop (PC) devices. Eleven participants were involved in each experimental condition, MB and PC, performing both the tasks on the same device, for a total of twenty-two people, recruited from students, researchers, and employees of the university. The following two research questions were investigated: Q1: Is there a difference in the cognitive load while performing the comprehension and the information-seeking tasks? Q2: Does the adopted device influence the cognitive load? The results obtained show that the baseline (BL) requires the lower cognitive load in both the conditions, while in IS task, the requirement reaches its highest value, especially using a mobile phone. In general, the power of all the brain wave bands increased in all conditions (MB and PC) during the two tasks (CO and IS), except for alpha, which is usually high in a state of relaxation and low cognitive load. People include website navigation into their daily routines, and for this, it is important to create an interaction that is as easy and barrier-free as possible. An effective design allows a user to focus on interesting information: many website architectures, instead, are an obstacle to be overcome; they impose a high cognitive load and poor user experience. All these aspects draw cognitive resources away from the user's primary task of finding and comprehending the site's information. Having information about how the cognitive load varies based on the device adopted and the considered task can provide useful indicators in this direction. This work suggests that using an EEG low-cost wearable device could be useful to quantify the cognitive load induced, allowing the development of new experiments to analyse these dependencies deeper, and to provide suggestions for better interaction with the web.

Aviation and neurophysiology: A systematic review

This paper systematically reviews 20 years of publications (N = 54) on aviation and neurophysiology. The main goal is to provide an account of neurophysiological changes associated with flight training with the aim of identifying neurometrics indicative of pilot's flight training level and task relevant mental states, as well as to capture the current state-of-art of (neuro)ergonomic design and practice in flight training. We identified multiple candidate neurometrics of training progress and workload, such as frontal theta power, the EEG Engagement Index and the Cognitive Stability Index. Furthermore, we discovered that several types of classifiers could be used to accurately detect mental states, such as the detection of drowsiness and mental fatigue. The paper advances practical guidelines on terminology usage, simulator fidelity, and multimodality, as well as future research ideas including the potential of Virtual Reality flight simulations for training, and a brain-computer interface for flight training.

Cognitive performance and electroencephalographic variations in air traffic controllers under various mental workload and time of day

The aim of this study was to investigate the effects of mental workload (MWL) and time of day on cognitive performance and electroencephalographic (EEG) parameters of air traffic controllers. EEG signals recorded while 20 professional air traffic controllers performed cognitive tasks [A-X Continuous Performance Test (AX-CPT) and 3-back working memory task] after they were exposed to two levels of task difficulty (high and low MWL) in the morning and afternoon. Significant decreases in cognitive performance were found when the levels of task difficulty increased in both tasks. The results confirmed the sensitivity of the theta and beta activities to levels of task difficulty in the 3-back task, while they were not affected in the AX-CPT. Theta and beta activities were influenced by time of day in the AX-CPT. The findings provide guidance for application of changes in EEG parameters when MWL level is manipulated during the day that could be implemented in future for the development of real-time monitoring systems to improve aviation safety.

Task difficulty and physiological measures of mental workload in air traffic control: a scoping review

This study provides a systematic synthesis of empirical research on mental workload (MWL) in air traffic control (ATC). MWL is a key concept in research on innovative technologies, because the assessment of MWL is crucial to the evaluation of such technologies. Our specific focus was on physiological measures of MWL. The used search strategy identified 39 peer-reviewed publications that analysed ATC tasks, examined different levels of difficulty of the ATC task, and considered at least one physiological measure of MWL. Positive relations between measures of MWL and task difficulty were observed most frequently, indicating that the measures indeed allowed the assessment of MWL. The most commonly used physiological measures were brain measures (EEG and fNIR) and heart rate measures. The review revealed a need for more precise descriptions of crucial experimental parameters in order to permit a transition of the field towards more interactive and dynamic types of analysis. Practitioner summary: Research on innovative technology in air traffic control (ATC) depends on assessments of mental workload (MWL). We reviewed empirical research on MWL in ATC. Brain and heart measures often allow assessments of MWL. Better descriptions of experiments are needed to allow comparisons among studies and more dynamic and interactive analyses.

Evaluation of a New Lightweight EEG Technology for Translational Applications of Passive Brain-Computer Interfaces

Technologies like passive brain-computer interfaces (BCI) can enhance human-machine interaction. Anyhow, there are still shortcomings in terms of easiness of use, reliability, and generalizability that prevent passive-BCI from entering real-life situations. The current work aimed to technologically and methodologically design a new gel-free passive-BCI system for out-of-the-lab employment. The choice of the water-based electrodes and the design of a new lightweight headset met the need for easy-to-wear, comfortable, and highly acceptable technology. The proposed system showed high reliability in both laboratory and realistic settings, performing not significantly different from the gold standard based on gel electrodes. In both cases, the proposed system allowed effective discrimination (AUC > 0.9) between low and high levels of workload, vigilance, and stress even for high temporal resolution (<10 s). Finally, the generalizability of the proposed system has been tested through a cross-task calibration. The system calibrated with the data recorded during the laboratory tasks was able to discriminate the targeted human factors during the realistic task reaching AUC values higher than 0.8 at 40 s of temporal resolution in case of vigilance and workload, and 20 s of temporal resolution for the stress monitoring. These results pave the way for ecologic use of the system, where calibration data of the realistic task are difficult to obtain.

Rescue Robots for the Urban Earthquake Environment

Robotics have important applications in the field of disaster medical rescue. The deployment of urban rescue robots at the earthquake site can help shorten response time, improve rescue efficiency and keep rescue personnel away from danger. This discussion introduces the performance of some robots in actual rescue scenarios, focuses on the current research status of robots that can provide medical assistance, and analyzes the merits and shortcomings of each system. Based on existing studies, the limitations and development directions of urban rescue robots are also discussed.

Human Mental Workload: A Survey and a Novel Inclusive Definition

Human mental workload is arguably the most invoked multidimensional construct in Human Factors and Ergonomics, getting momentum also in Neuroscience and Neuroergonomics. Uncertainties exist in its characterization, motivating the design and development of computational models, thus recently and actively receiving support from the discipline of Computer Science. However, its role in human performance prediction is assured. This work is aimed at providing a synthesis of the current state of the art in human mental workload assessment through considerations, definitions, measurement techniques as well as applications, Findings suggest that, despite an increasing number of associated research works, a single, reliable and generally applicable framework for mental workload research does not yet appear fully established. One reason for this gap is the existence of a wide swath of operational definitions, built upon different theoretical assumptions which are rarely examined collectively. A second reason is that the three main classes of measures, which are self-report, task performance, and physiological indices, have been used in isolation or in pairs, but more rarely in conjunction all together. Multiple definitions complement each another and we propose a novel inclusive definition of mental workload to support the next generation of empirical-based research. Similarly, by comprehensively employing physiological, task-performance, and self-report measures, more robust assessments of mental workload can be achieved.

EEG-based measurement system for monitoring student engagement in learning 4.0

A wearable system for the personalized EEG-based detection of engagement in learning 4.0 is proposed. In particular, the effectiveness of the proposed solution is assessed by means of the classification accuracy in predicting engagement. The system can be used to make an automated teaching platform adaptable to the user, by managing eventual drops in the cognitive and emotional engagement. The effectiveness of the learning process mainly depends on the engagement level of the learner. In case of distraction, lack of interest or superficial participation, the teaching strategy could be personalized by an automatic modulation of contents and communication strategies. The system is validated by an experimental case study on twenty-one students. The experimental task was to learn how a specific human-machine interface works. Both the cognitive and motor skills of participants were involved. De facto standard stimuli, namely (1) cognitive task (Continuous Performance Test), (2) music background (Music Emotion Recognition-MER database), and (3) social feedback (Hermans and De Houwer database), were employed to guarantee a metrologically founded reference. In within-subject approach, the proposed signal processing pipeline (Filter bank, Common Spatial Pattern, and Support Vector Machine), reaches almost 77% average accuracy, in detecting both cognitive and emotional engagement.

Degraded States of Engagement in Air Traffic Control

Safety studies have identified attention as a recurring cause of incidents and accidents in air traffic control. However, little is known of the precise attentional states that lead to degraded ATC performance. Therefore, we surveyed 150 French en route air traffic controllers on the causes of and impacts on perceived cooperation, safety, and performance of seven degraded attentional states from the literature: task-related and task-unrelated mind wandering, mental overload, inattentional deafness and blindness, attentional entropy, and perseveration. Our findings indicated that task-related and task-unrelated mind wandering were the most prevalent but had the least impact on perceived safety. Conversely, inattentional blindness and attentional entropy were less reported but were considered a significant safety concern, while inattentional deafness affected cooperation. Most states were experienced in workload levels consistent with the literature. However, no other factor such as shift work was identified as a cause of these states. Overall, these findings suggest that “attention” is not a specific enough subject for ATC, as attentional issues can occur in various conditions and have different impacts. As far as safety is concerned, inattentional blindness should be the prime target for further research. Neuroergonomics in particular could help develop dynamic countermeasures to mitigate its impact.

Cognitive demands and mental workload: A filed study of the mining control room operators

Cognitive demand and mental workload assessment are essential for the optimal interaction of human-machine systems. The aim of this study was to investigate the cognitive demands and mental workload as well as the relationship between them among the mining control room operators.

This cross-sectional study was performed on 63 control room operators of a large mining plant located in Iran. Cognitive demands and mental workload were assessed using cognitive task analysis (CTA) and NASA Task Load Index (NASA-TLX), respectively and the analysis was performed using SPSS version 21. Independent samples T-test, Mann-Whitney U test and multivariate linear regression were used for data analysis.

Twelve cognitive demands were extracted after observing the tasks and conducting semi-structured interviews with the control room staff. The mean scores of total cognitive demands and MWL were 6.60 and 72.89, respectively, and these two indicators showed a positive and significant correlation (r = 0.286; P = 0.023). The participants’ demographic characteristics such as age, education, and work experience did not affect mental workload, but the two cognitive demands (memory and defect detection) affected MWL.

High cognitive demands and mental workload indicate poor interaction between humans and machines. Due to the effect of memory load and defect detection on mental workload, it is recommended to assign cognitive tasks based on memory and defect detection to the machine to reduce the mental workload and improve human-machine interaction.

Recent Advances and Current Trends in Brain-Computer Interface (BCI) Research and Their Applications

Brain-Computer Interface (BCI) provides direct communication between the brain and an external device. BCI systems have become a trendy field of research in recent years. These systems can be used in a variety of applications to help both disabled and healthy people. Concerning significant BCI progress, we may assume that these systems are not very far from real-world applications. This review has taken into account current trends in BCI research. In this survey, one hundred most cited articles from the WOS database were selected over the last four years. This survey is divided into several sectors. These sectors are Medicine, Communication and Control, Entertainment, and Other BCI applications. The application area, recording method, signal acquisition types, and countries of origin have been identified in each article. This survey provides an overview of the BCI articles published from 2016 to 2020 and their current trends and advances in different application areas.

Impacts of Color Coding on Programming Learning in Multimedia Learning: Moving Toward a Multimodal Methodology

In the present study, we tested the effectiveness of color coding on the programming learning of students who were learning from video lectures. Effectiveness was measured using multimodal physiological measures, combining eye tracking and electroencephalography (EEG). Using a between-subjects design, 42 university students were randomly assigned to two video lecture conditions (color-coded vs. grayscale). The participants' eye tracking and EEG signals were recorded while watching the assigned video, and their learning performance was subsequently assessed. The results showed that the color-coded design was more beneficial than the grayscale design, as indicated by smaller pupil diameter, shorter fixation duration, higher EEG theta and alpha band power, lower EEG cognitive load, and better learning performance. The present findings have practical implications for designing slide-based programming learning video lectures; slides should highlight the format of the program code using color coding.

Measuring the Domain Specificity of Workload Using EEG: Auditory and Visual Domains in Rotary-Wing Simulated Flight

OBJECTIVE

The overarching objective was to evaluate whether workload sensory-domain specificity could be identified through electroencephalogram (EEG) recordings during simulated rotary-wing operations.

BACKGROUND

Rotary-wing aviators experience workload from different sensory domains, although predominantly through auditory and visual domains. Development of real-time monitoring tools using psychophysiological indices, such as EEG recordings, could enable identification of aviator overload in real time.

METHOD

Two studies were completed, both of which recorded EEG, task performance, and self-report data. In Study 1, 16 individuals completed a basic auditory and a basic visual laboratory task where workload was manipulated. In Study 2, 23 Army aviators completed simulated aviation flights where workload was manipulated within auditory and visual sensory domains.

RESULTS

Results from Study 1 found differences in frontal alpha activity during the auditory task, and that alpha and beta activities were associated with perceived workload. Frontal theta activity was found to differ during the visual task while frontal alpha was associated with perceived workload. Study 2 found support for frontal beta activity and the ratio of beta to alpha + theta to differentiate level of workload within the auditory domain.

CONCLUSION

There is likely a role of frontal alpha and beta activities in response to workload manipulations within the auditory domain; however, this role becomes more equivocal when examined in a multifaceted flight scenario.

APPLICATION

Results from this study provide a basis for understanding changes in EEG activity when workload is manipulated in sensory domains that can be used in furthering the development of real-time monitoring tools.

Sensitivity of the Spatial Distribution of Fixations to Variations in the Type of Task Demand and Its Relation to Visual Entropy

Ocular activity is known to be sensitive to variations in mental workload, and recent studies have successfully related the distribution of eye fixations to the mental load. This study aimed to verify the effectiveness of the spatial distribution of fixations as a measure of mental workload and its sensitivity to different types of demands imposed by the task: mental, temporal, and physical. To test the research hypothesis, two experimental studies were run: Experiment 1 evaluated the sensitivity of an index of spatial distribution (Nearest Neighbor Index; NNI) to changes in workload. A sample of 30 participants participated in a within-subject design with different types of task demands (mental, temporal, physical) applied to Tetris game; Experiment 2 investigated the accuracy of the index through the analysis of 1-min epochs during the execution of a visual-spatial task (the “spot the differences” puzzle game). Additionally, NNI was compared to a better-known ocular mental workload index, the entropy rate. The data analysis showed a relation between the NNI and the different workload levels imposed by the tasks. In particular: Experiment 1 demonstrated that increased difficulty, due to higher temporal demand, led to a more dispersed pattern with respect to the baseline, whereas the mental demand led to a more grouped pattern of fixations with respect to the baseline; Experiment 2 indicated that the entropy rate and the NNI show a similar pattern over time, indicating high mental workload after the first minute of activity. That suggests that NNI highlights the greater presence of fixation groups and, accordingly, the entropy indicates a more regular and orderly scanpath. Both indices are sensitive to changes in workload and they seem to anticipate the drop in performance. However, the entropy rate is limited by the use of the areas of interest, making it impossible to apply it in dynamic contexts. Conversely, NNI works with the entire scanpath and it shows sensitivity to different types of task demands. These results confirm the NNI as a measure applicable to different contexts and its potential use as a trigger in adaptive systems implemented in high-risk settings, such as control rooms and transportation systems.

Joint Analysis of Eye Blinks and Brain Activity to Investigate Attentional Demand during a Visual Search Task

In several fields, the need for a joint analysis of brain activity and eye activity to investigate the association between brain mechanisms and manifest behavior has been felt. In this work, two levels of attentional demand, elicited through a conjunction search task, have been modelled in terms of eye blinks, brain activity, and brain network features. Moreover, the association between endogenous neural mechanisms underlying attentional demand and eye blinks, without imposing a time-locked structure to the analysis, has been investigated. The analysis revealed statistically significant spatial and spectral modulations of the recorded brain activity according to the different levels of attentional demand, and a significant reduction in the number of eye blinks when a higher amount of attentional investment was required. Besides, the integration of information coming from high-density electroencephalography (EEG), brain source localization, and connectivity estimation allowed us to merge spectral and causal information between brain areas, characterizing a comprehensive model of neurophysiological processes behind attentional demand. The analysis of the association between eye and brain-related parameters revealed a statistically significant high correlation (R > 0.7) of eye blink rate with anterofrontal brain activity at 8 Hz, centroparietal brain activity at 12 Hz, and a significant moderate correlation with the participation of right Intra Parietal Sulcus in alpha band (R = -0.62). Due to these findings, this work suggests the possibility of using eye blinks measured from one sensor placed on the forehead as an unobtrusive measure correlating with neural mechanisms underpinning attentional demand.

The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence

Real-world experience is typically multimodal. Evidence indicates that the facilitation in the detection of multisensory stimuli is modulated by the perceptual load, the amount of information involved in the processing of the stimuli. Here, we used a realistic virtual reality environment while concomitantly acquiring Electroencephalography (EEG) and Galvanic Skin Response (GSR) to investigate how multisensory signals impact target detection in two conditions, high and low perceptual load. Different multimodal stimuli (auditory and vibrotactile) were presented, alone or in combination with the visual target. Results showed that only in the high load condition, multisensory stimuli significantly improve performance, compared to visual stimulation alone. Multisensory stimulation also decreases the EEG-based workload. Instead, the perceived workload, according to the "NASA Task Load Index" questionnaire, was reduced only by the trimodal condition (i.e., visual, auditory, tactile). This trimodal stimulation was more effective in enhancing the sense of presence, that is the feeling of being in the virtual environment, compared to the bimodal or unimodal stimulation. Also, we show that in the high load task, the GSR components are higher compared to the low load condition. Finally, the multimodal stimulation (Visual-Audio-Tactile-VAT and Visual-Audio-VA) induced a significant decrease in latency, and a significant increase in the amplitude of the P300 potentials with respect to the unimodal (visual) and visual and tactile bimodal stimulation, suggesting a faster and more effective processing and detection of stimuli if auditory stimulation is included. Overall, these findings provide insights into the relationship between multisensory integration and human behavior and cognition.

NeuroDante: Poetry Mentally Engages More Experts but Moves More Non-Experts, and for Both the Cerebral Approach Tendency Goes Hand in Hand with the Cerebral Effort

Neuroaesthetics, the science studying the biological underpinnings of aesthetic experience, recently extended its area of investigation to literary art; this was the humus where neurocognitive poetics blossomed. Divina Commedia represents one of the most important, famous and studied poems worldwide. Poetry stimuli are characterized by elements (meter and rhyme) promoting the processing fluency, a core aspect of neuroaesthetics theories. In addition, given the evidence of different neurophysiological reactions between experts and non-experts in response to artistic stimuli, the aim of the present study was to investigate, in poetry, a different neurophysiological cognitive and emotional reaction between Literature (L) and Non-Literature (NL) students. A further aim was to investigate whether neurophysiological underpinnings would support explanation of behavioral data. Investigation methods employed: self-report assessments (recognition, appreciation, content recall) and neurophysiological indexes (approach/withdrawal (AW), cerebral effort (CE) and galvanic skin response (GSR)). The main behavioral results, according to fluency theories in aesthetics, suggested in the NL but not in the L group that the appreciation/liking went hand by hand with the self-declared recognition and with the content recall. The main neurophysiological results were: (i) higher galvanic skin response in NL, whilst higher CE values in L; (ii) a positive correlation between AW and CE indexes in both groups. The present results extended previous evidence relative to figurative art also to auditory poetry stimuli, suggesting an emotional attenuation “expertise-specific” showed by experts, but increased cognitive processing in response to the stimuli.

Forefront Users' Experience Evaluation by Employing Together Virtual Reality and Electroencephalography: A Case Study on Cognitive Effects of Scents

Scents have the ability to affect peoples' mental states and task performance with to different extents. It has been widely demonstrated that the lemon scent, included in most all-purpose cleaners, elicits stimulation and activation, while the lavender scent elicits relaxation and sedative effects. The present study aimed at investigating and fostering a novel approach to evaluate users' experience with respect to scents' effects through the joint employment of Virtual Reality and users' neurophysiological monitoring, in particular Electroencephalography. In particular, this study, involving 42 participants, aimed to compare the effects of lemon and lavender scents on the deployment of cognitive resources during a daily life experience consisting in a train journey carried out in virtual reality. Our findings showed a significant higher request of cognitive resources during the processing of an informative message for subjects exposed to the lavender scent with respect to the lemon exposure. No differences were found between lemon and lavender conditions on the self-reported items of pleasantness and involvement; as this study demonstrated, the employment of the lavender scent preserves the quality of the customer experience to the same extent as the more widely used lemon scent.

A comprehensive assessment of Brain Computer Interfaces: Recent trends and challenges

BACKGROUND

An uninterrupted channel of communication and control between the human brain and electronic processing units has led to an increased use of Brain Computer Interfaces (BCIs). This article attempts to present an all-encompassing review on BCI and the scientific advancements associated with it. The ultimate goal of this review is to provide a general overview of the BCI technology and to shed light on different aspects of BCIs. This review also underscores the applications, practical challenges and opportunities associated with BCI technology, which can be used to accelerate future developments in this field.

METHODS

This review is based on a systematic literature search for tracking down the relevant research annals and proceedings. Using a methodical search strategy, the search was carried out across major technical databases. The retrieved records were screened for their relevance and a total of 369 research chronicles were engulfed in this review based on the inclusion criteria.

RESULTS

This review describes the present scenario and recent advancements in BCI technology. It also identifies several application areas of BCI technology. This comprehensive review provides evidence that, while we are getting ever closer, significant challenges still exist for the development of BCIs that can seamlessly integrate with the user's biological system.

CONCLUSION

The findings of this review confirm the importance of BCI technology in various applications. It is concluded that BCI technology, still in its sprouting phase, requires significant explorations for further development.

A Novel Mutual Information Based Feature Set for Drivers' Mental Workload Evaluation Using Machine Learning

Analysis of physiological signals, electroencephalography more specifically, is considered a very promising technique to obtain objective measures for mental workload evaluation, however, it requires a complex apparatus to record, and thus, with poor usability in monitoring in-vehicle drivers’ mental workload. This study proposes a methodology of constructing a novel mutual information-based feature set from the fusion of electroencephalography and vehicular signals acquired through a real driving experiment and deployed in evaluating drivers’ mental workload. Mutual information of electroencephalography and vehicular signals were used as the prime factor for the fusion of features. In order to assess the reliability of the developed feature set mental workload score prediction, classification and event classification tasks were performed using different machine learning models. Moreover, features extracted from electroencephalography were used to compare the performance. In the prediction of mental workload score, expert-defined scores were used as the target values. For classification tasks, true labels were set from contextual information of the experiment. An extensive evaluation of every prediction tasks was carried out using different validation methods. In predicting the mental workload score from the proposed feature set lowest mean absolute error was 0.09 and for classifying mental workload highest accuracy was 94%. According to the outcome of the study, it can be stated that the novel mutual information based features developed through the proposed approach can be employed to classify and monitor in-vehicle drivers’ mental workload.

Neurophysiological Vigilance Characterisation and Assessment: Laboratory and Realistic Validations Involving Professional Air Traffic Controllers

Vigilance degradation usually causes significant performance decrement. It is also considered the major factor causing the out-of-the-loop phenomenon (OOTL) occurrence. OOTL is strongly related to a high level of automation in operative contexts such as the Air Traffic Management (ATM), and it could lead to a negative impact on the Air Traffic Controllers’ (ATCOs) engagement. As a consequence, being able to monitor the ATCOs’ vigilance would be very important to prevent risky situations. In this context, the present study aimed to characterise and assess the vigilance level by using electroencephalographic (EEG) measures. The first study, involving 13 participants in laboratory settings allowed to find out the neurophysiological features mostly related to vigilance decrements. Those results were also confirmed under realistic ATM settings recruiting 10 professional ATCOs. The results demonstrated that (i) there was a significant performance decrement related to vigilance reduction; (ii) there were no substantial differences between the identified neurophysiological features in controlled and ecological settings, and the EEG-channel configuration defined in laboratory was able to discriminate and classify vigilance changes in ATCOs’ vigilance with high accuracy (up to 84%); (iii) the derived two EEG-channel configuration was able to assess vigilance variations reporting only slight accuracy reduction.

A review and experimental study on the application of classifiers and evolutionary algorithms in EEG-based brain-machine interface systems

OBJECTIVE

Considering the importance and the near-future development of noninvasive brain-machine interface (BMI) systems, this paper presents a comprehensive theoretical-experimental survey on the classification and evolutionary methods for BMI-based systems in which EEG signals are used.

APPROACH

The paper is divided into two main parts. In the first part, a wide range of different types of the base and combinatorial classifiers including boosting and bagging classifiers and evolutionary algorithms are reviewed and investigated. In the second part, these classifiers and evolutionary algorithms are assessed and compared based on two types of relatively widely used BMI systems, sensory motor rhythm-BMI and event-related potentials-BMI. Moreover, in the second part, some of the improved evolutionary algorithms as well as bi-objective algorithms are experimentally assessed and compared.

MAIN RESULTS

In this study two databases are used, and cross-validation accuracy (CVA) and stability to data volume (SDV) are considered as the evaluation criteria for the classifiers. According to the experimental results on both databases, regarding the base classifiers, linear discriminant analysis and support vector machines with respect to CVA evaluation metric, and naive Bayes with respect to SDV demonstrated the best performances. Among the combinatorial classifiers, four classifiers, Bagg-DT (bagging decision tree), LogitBoost, and GentleBoost with respect to CVA, and Bagging-LR (bagging logistic regression) and AdaBoost (adaptive boosting) with respect to SDV had the best performances. Finally, regarding the evolutionary algorithms, single-objective invasive weed optimization (IWO) and bi-objective nondominated sorting IWO algorithms demonstrated the best performances.

SIGNIFICANCE

We present a general survey on the base and the combinatorial classification methods for EEG signals (sensory motor rhythm and event-related potentials) as well as their optimization methods through the evolutionary algorithms. In addition, experimental and statistical significance tests are carried out to study the applicability and effectiveness of the reviewed methods.

Consumer Behaviour through the Eyes of Neurophysiological Measures: State-of-the-Art and Future Trends

The new technological advances achieved during the last decade allowed the scientific community to investigate and employ neurophysiological measures not only for research purposes but also for the study of human behaviour in real and daily life situations. The aim of this review is to understand how and whether neuroscientific technologies can be effectively employed to better understand the human behaviour in real decision-making contexts. To do so, firstly, we will describe the historical development of neuromarketing and its main applications in assessing the sensory perceptions of some marketing and advertising stimuli. Then, we will describe the main neuroscientific tools available for such kind of investigations (e.g., measuring the cerebral electrical or hemodynamic activity, the eye movements, and the psychometric responses). Also, this review will present different brain measurement techniques, along with their pros and cons, and the main cerebral indexes linked to the specific mental states of interest (used in most of the neuromarketing research). Such indexes have been supported by adequate validations from the scientific community and are largely employed in neuromarketing research. This review will also discuss a series of papers that present different neuromarketing applications, such us in-store choices and retail, services, pricing, brand perception, web usability, neuropolitics, evaluation of the food and wine taste, and aesthetic perception of artworks. Furthermore, this work will face the ethical issues arisen on the use of these tools for the evaluation of the human behaviour during decision-making tasks. In conclusion, the main challenges that neuromarketing is going to face, as well as future directions and possible scenarios that could be derived by the use of neuroscience in the marketing field, will be identified and discussed.

Brain-Computer Interface-Based Adaptive Automation to Prevent Out-Of-The-Loop Phenomenon in Air Traffic Controllers Dealing With Highly Automated Systems

Increasing the level of automation in air traffic management is seen as a measure to increase the performance of the service to satisfy the predicted future demand. This is expected to result in new roles for the human operator: he will mainly monitor highly automated systems and seldom intervene. Therefore, air traffic controllers (ATCos) would often work in a supervisory or control mode rather than in a direct operating mode. However, it has been demonstrated how human operators in such a role are affected by human performance issues, known as Out-Of-The-Loop (OOTL) phenomenon, consisting in lack of attention, loss of situational awareness and de-skilling. A countermeasure to this phenomenon has been identified in the adaptive automation (AA), i.e., a system able to allocate the operative tasks to the machine or to the operator depending on their needs. In this context, psychophysiological measures have been highlighted as powerful tool to provide a reliable, unobtrusive and real-time assessment of the ATCo's mental state to be used as control logic for AA-based systems. In this paper, it is presented the so-called "Vigilance and Attention Controller", a system based on electroencephalography (EEG) and eye-tracking (ET) techniques, aimed to assess in real time the vigilance level of an ATCo dealing with a highly automated human-machine interface and to use this measure to adapt the level of automation of the interface itself. The system has been tested on 14 professional ATCos performing two highly realistic scenarios, one with the system disabled and one with the system enabled. The results confirmed that (i) long high automated tasks induce vigilance decreasing and OOTL-related phenomena; (ii) EEG measures are sensitive to these kinds of mental impairments; and (iii) AA was able to counteract this negative effect by keeping the ATCo more involved within the operative task. The results were confirmed by EEG and ET measures as well as by performance and subjective ones, providing a clear example of potential applications and related benefits of AA.

How Neurophysiological Measures Can be Used to Enhance the Evaluation of Remote Tower Solutions

New solutions in operational environments are often, among objective measurements, evaluated by using subjective assessment and judgment from experts. Anyhow, it has been demonstrated that subjective measures suffer from poor resolution due to a high intra and inter-operator variability. Also, performance measures, if available, could provide just partial information, since an operator could achieve the same performance but experiencing a different workload. In this study, we aimed to demonstrate: (i) the higher resolution of neurophysiological measures in comparison to subjective ones; and (ii) how the simultaneous employment of neurophysiological measures and behavioral ones could allow a holistic assessment of operational tools. In this regard, we tested the effectiveness of an electroencephalography (EEG)-based neurophysiological index (W_EEG index) in comparing two different solutions (i.e., Normal and Augmented) in terms of experienced workload. In this regard, 16 professional air traffic controllers (ATCOs) have been asked to perform two operational scenarios. Galvanic Skin Response (GSR) has also been recorded to evaluate the level of arousal (i.e., operator involvement) during the two scenarios execution. NASA-TLX questionnaire has been used to evaluate the perceived workload, and an expert was asked to assess performance achieved by the ATCOs. Finally, reaction times on specific operational events relevant for the assessment of the two solutions, have also been collected. Results highlighted that the Augmented solution induced a local increase in subjects performance (Reaction times). At the same time, this solution induced an increase in the workload experienced by the participants (W_EEG). Anyhow, this increase is still acceptable, since it did not negatively impact the performance and has to be intended only as a consequence of the higher engagement of the ATCOs. This behavioral effect is totally in line with physiological results obtained in terms of arousal (GSR), that increased during the scenario with augmentation. Subjective measures (NASA-TLX) did not highlight any significant variation in perceived workload. These results suggest that neurophysiological measure provide additional information than behavioral and subjective ones, even at a level of few seconds, and its employment during the pre-operational activities (e.g., design process) could allow a more holistic and accurate evaluation of new solutions.

Individual-Specific Classification of Mental Workload Levels Via an Ensemble Heterogeneous Extreme Learning Machine for EEG Modeling

In a human–machine cooperation system, assessing the mental workload (MW) of the human operator is quite crucial to maintaining safe operation conditions. Among various MW indicators, electroencephalography (EEG) signals are particularly attractive because of their high temporal resolution and sensitivity to the occupation of working memory. However, the individual difference of the EEG feature distribution may impair the machine-learning based MW classifier. In this paper, we employed a fast-training neural network, extreme learning machine (ELM), as the basis to build an individual-specific classifier ensemble to recognize binary MW. To improve the diversity of the classification committee, heterogeneous member classifiers were adopted by fusing multiple ELMs and Bayesian models. Specifically, a deep network structure was applied in each weak model aiming at finding informative EEG feature representations. The structure of hyper-parameters of the proposed heterogeneous ensemble ELM (HE-ELM) was then identified and then its performance was compared against several competitive MW classifiers. We found that the HE-ELM model was superior for improving the individual-specific accuracy of MW assessments.

EEG-Based Prediction of Cognitive Load in Intelligence Tests

Measuring and assessing the cognitive load associated with different tasks is crucial for many applications, from the design of instructional materials to monitoring the mental well-being of aircraft pilots. The goal of this paper is to utilize EEG to infer the cognitive workload of subjects during intelligence tests. We chose the well established advanced progressive matrices test, an ideal framework because it presents problems at increasing levels of difficulty and has been rigorously validated in past experiments. We train classic machine learning models using basic EEG measures as well as measures of network connectivity and signal complexity. Our findings demonstrate that cognitive load can be well predicted using these features, even for a low number of channels. We show that by creating an individually tuned neural network for each subject, we can improve prediction compared to a generic model and that such models are robust to decreasing the number of available channels as well.

The Dry Revolution: Evaluation of Three Different EEG Dry Electrode Types in Terms of Signal Spectral Features, Mental States Classification and Usability

One century after the first recording of human electroencephalographic (EEG) signals, EEG has become one of the most used neuroimaging techniques. The medical devices industry is now able to produce small and reliable EEG systems, enabling a wide variety of applications also with no-clinical aims, providing a powerful tool to neuroscientific research. However, these systems still suffer from a critical limitation, consisting in the use of wet electrodes, that are uncomfortable and require expertise to install and time from the user. In this context, dozens of different concepts of EEG dry electrodes have been recently developed, and there is the common opinion that they are reaching traditional wet electrodes quality standards. However, although many papers have tried to validate them in terms of signal quality and usability, a comprehensive comparison of different dry electrode types from multiple points of view is still missing. The present work proposes a comparison of three different dry electrode types, selected among the main solutions at present, against wet electrodes, taking into account several aspects, both in terms of signal quality and usability. In particular, the three types consisted in gold-coated single pin, multiple pins and solid-gel electrodes. The results confirmed the great standards achieved by dry electrode industry, since it was possible to obtain results comparable to wet electrodes in terms of signals spectra and mental states classification, but at the same time drastically reducing the time of montage and enhancing the comfort. In particular, multiple-pins and solid-gel electrodes overcome gold-coated single-pin-based ones in terms of comfort.

Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects

Recent advances in neuroscience have paved the way to innovative applications that cognitively augment and enhance humans in a variety of contexts. This paper aims at providing a snapshot of the current state of the art and a motivated forecast of the most likely developments in the next two decades. Firstly, we survey the main neuroscience technologies for both observing and influencing brain activity, which are necessary ingredients for human cognitive augmentation. We also compare and contrast such technologies, as their individual characteristics (e.g., spatio-temporal resolution, invasiveness, portability, energy requirements, and cost) influence their current and future role in human cognitive augmentation. Secondly, we chart the state of the art on neurotechnologies for human cognitive augmentation, keeping an eye both on the applications that already exist and those that are emerging or are likely to emerge in the next two decades. Particularly, we consider applications in the areas of communication, cognitive enhancement, memory, attention monitoring/enhancement, situation awareness and complex problem solving, and we look at what fraction of the population might benefit from such technologies and at the demands they impose in terms of user training. Thirdly, we briefly review the ethical issues associated with current neuroscience technologies. These are important because they may differentially influence both present and future research on (and adoption of) neurotechnologies for human cognitive augmentation: an inferior technology with no significant ethical issues may thrive while a superior technology causing widespread ethical concerns may end up being outlawed. Finally, based on the lessons learned in our analysis, using past trends and considering other related forecasts, we attempt to forecast the most likely future developments of neuroscience technology for human cognitive augmentation and provide informed recommendations for promising future research and exploitation avenues.

Using Brain Activation to Evaluate Arrangements Aiding Hand-Eye Coordination in Surgical Robot Systems

GOAL

To realize intuitive, minimally invasive surgery, surgical robots are often controlled using master-slave systems. However, the surgical robot's structure often differs from that of the human body, so the arrangement between the monitor and master must reflect this physical difference. In this study, we validate the feasibility of an embodiment evaluation method that determines the arrangement between the monitor and master. In our constructed cognitive model, the brain's intraparietal sulcus activates significantly when somatic and visual feedback match. Using this model, we validate a cognitively appropriate arrangement between the monitor and master.

METHODS

In experiments, we measure participants' brain activation using an imaging device as they control the virtual surgical simulator. Two experiments are carried out that vary the monitor and hand positions.

CONCLUSION

There are two common arrangements of the monitor and master at the brain activation's peak: One is placing the monitor behind the master, so the user feels that the system is an extension of his arms into the monitor; the other arranges the monitor in front of the master, so the user feels the correspondence between his own arm and the virtual arm in the monitor.

SIGNIFICANCE

From these results, we conclude that the arrangement between the monitor and master impacts embodiment, enabling the participant to feel apparent posture matches in master-slave surgical robot systems.

EEG-Based Mental Workload Neurometric to Evaluate the Impact of Different Traffic and Road Conditions in Real Driving Settings

Car driving is considered a very complex activity, consisting of different concomitant tasks and subtasks, thus it is crucial to understand the impact of different factors, such as road complexity, traffic, dashboard devices, and external events on the driver's behavior and performance. For this reason, in particular situations the cognitive demand experienced by the driver could be very high, inducing an excessive experienced mental workload and consequently an increasing of error commission probability. In this regard, it has been demonstrated that human error is the main cause of the 57% of road accidents and a contributing factor in most of them. In this study, 20 young subjects have been involved in a real driving experiment, performed under different traffic conditions (rush hour and not) and along different road types (main and secondary streets). Moreover, during the driving tasks different specific events, in particular a pedestrian crossing the road and a car entering the traffic flow just ahead of the experimental subject, have been acted. A Workload Index based on the Electroencephalographic (EEG), i.e., brain activity, of the drivers has been employed to investigate the impact of the different factors on the driver's workload. Eye-Tracking (ET) technology and subjective measures have also been employed in order to have a comprehensive overview of the driver's perceived workload and to investigate the different insights obtainable from the employed methodologies. The employment of such EEG-based Workload index confirmed the significant impact of both traffic and road types on the drivers' behavior (increasing their workload), with the advantage of being under real settings. Also, it allowed to highlight the increased workload related to external events while driving, in particular with a significant effect during those situations when the traffic was low. Finally, the comparison between methodologies revealed the higher sensitivity of neurophysiological measures with respect to ET and subjective ones. In conclusion, such an EEG-based Workload index would allow to assess objectively the mental workload experienced by the driver, standing out as a powerful tool for research aimed to investigate drivers' behavior and providing additional and complementary insights with respect to traditional methodologies employed within road safety research.

Neurophysiological Responses to Different Product Experiences

It is well known that the evaluation of a product from the shelf considers the simultaneous cerebral and emotional evaluation of the different qualities of the product such as its colour, the eventual images shown, and the envelope's texture (hereafter all included in the term “product experience”). However, the measurement of cerebral and emotional reactions during the interaction with food products has not been investigated in depth in specialized literature. The aim of this paper was to investigate such reactions by the EEG and the autonomic activities, as elicited by the cross-sensory interaction (sight and touch) across several different products. In addition, we investigated whether (i) the brand (Major Brand or Private Label), (ii) the familiarity (Foreign or Local Brand), and (iii) the hedonic value of products (Comfort Food or Daily Food) influenced the reaction of a group of volunteers during their interaction with the products. Results showed statistically significantly higher tendency of cerebral approach (as indexed by EEG frontal alpha asymmetry) in response to comfort food during the visual exploration and the visual and tactile exploration phases. Furthermore, for the same index, a higher tendency of approach has been found toward foreign food products in comparison with local food products during the visual and tactile exploration phase. Finally, the same comparison performed on a different index (EEG frontal theta) showed higher mental effort during the interaction with foreign products during the visual exploration and the visual and tactile exploration phases. Results from the present study could deepen the knowledge on the neurophysiological response to food products characterized by different nature in terms of hedonic value familiarity; moreover, they could have implications for food marketers and finally lead to further study on how people make food choices through the interactions with their commercial envelope.

Neurophysiological Measures of the Perception of Antismoking Public Service Announcements Among Young Population

Tobacco constitutes a global emergency with totally preventable millions of deaths per year and smoking-related illnesses. Public service announcements (PSAs) are the main tool against smoking and by now their efficacy is still assessed through questionnaires and metrics, only months after their circulation. The present study focused on the young population, because at higher risk of developing tobacco addiction, investigating the reaction to the vision of Effective, Ineffective and Awarded antismoking PSAs through: electroencephalography (EEG), autonomic activity variation (Galvanic skin response—GSR- and Heart Rate—HR-) and Eye-Tracking (ET). The employed indices were: the EEG frontal alpha band asymmetry and the frontal theta; the Emotional Index (EI), deriving from the GSR and HR signals matching; the ET Visual Attention (VA) index, based on the ratio between the total time spent fixating an area of interest (AOI) and its area. Smokers expressed higher frontal alpha asymmetry values in comparison to non-smokers. Concerning frontal theta, Awarded PSAs reported the highest values in comparison to both Effective and Ineffective PSAs. EI results highlighted that lowest values were expressed by Heavy Smokers (HS), and Effective PSAs obtained the highest EI values. Finally, concerning the Effective PSAs, regression analysis highlighted a correlation between the number of cigarettes smoked by participants (independent variable) and frontal alpha asymmetry, frontal theta and EI values. ET results suggested that for the Ineffective PSAs the main focus were texts, while for the Effective and Awarded PSAs were the visual elements. Results support the use of methods aimed at assessing the physiological reaction for the evaluation of PSAs images, in particular when considering the smoking habits of target populations.

EEG-based Approach-Withdrawal index for the pleasantness evaluation during taste experience in realistic settings

The taste is a vital sense in humans, because of its active role in regulating nutrition or avoiding harmful substances. Several studies showed the important role of the brain Pre-Frontal Cortex in decoding information coming from the gustatory system. It is also widely known, in neuroscientific literature, that the asymmetry of Pre-Frontal Cortex Activity is closely linked to the feeling of pleasantness experienced by the subject during sensorial stimulation. In this regard, from the electroencephalographic (EEG) signal it is possible to estimate the Approach/Withdrawal (AW) index, which has been largely investigated and validated in scientific literature, regarding visual, acoustic and olfactory stimuli.

Evaluation of different cochlear implants in unilateral hearing patients during word listening tasks: A brain connectivity study

Advanced methodologies used for the biomedical signal interpretation allow using cerebral signals to assess important cognitive functions in humans. In the present study, as parameter of cerebral effort, has been employed the isolated effective coherence, in order to estimate the effective connectivity and network organization. The hypothesis was that the lower the number of inter-connections engaged, the lower the cerebral effort induced by the experimental condition. In the present research this index has been applied to test the reaction to the use of different cochlear implant processors (Freedom, CP810 and CP910 - Cochlear Ltd), with the aim to identify the most performing device during a word in noise recognition task. Results support the capability of identifying the device eliciting less brain area connections. In particular, the CP910 was the processor inducing the lower number of inter-connections among the tested ones. This investigation appeared to be worthy, since representing a tool to identify devices that would make available user's cognitive resources for additional tasks, a matter susceptible of generalization to various fields of application. The employment of the cerebral signals therefore open the way to the evaluation of the impact of different sensors and prosthetic devices, also using connectivity measures.

Human-Machine Interaction Assessment by Neurophysiological Measures: A Study on Professional Air Traffic Controllers

This study aims at investigating the possibility to employ neurophysiological measures to assess the humanmachine interaction effectiveness. Such a measure can be used to compare new technologies or solutions, with the final purpose to enhance operator's experience and increase safety. In the present work, two different interaction modalities (Normal and Augmented) related to Air Traffic Management field have been compared, by involving 10 professional air traffic controllers in a control tower simulated environment. Experimental task consisted in locating aircrafts in different airspace positions by using the sense of hearing. In one modality (i.e. "Normal"), all the sound sources (aircrafts) had the same amplification factor. In the "Augmented" modality, the amplification factor of the sound sources located along the participant head sagittal axis was increased, while the intensity of sound sources located outside this axis decreased. In other words, when the user oriented his head toward the aircraft position, the related sound was amplified. Performance data, subjective questionnaires (i.e. NASA-TLX) and neurophysiological measures (i.e. EEG-based) related to the experienced workload have been collected. Results showed higher significant performance achieved by the users during the "Augmented" modality with respect to the "Normal" one, supported by a significant decreasing in experienced workload, evaluated by using EEG-based index. In addition, Performance and EEG-based workload index showed a significant negative correlation. On the contrary, subjective workload analysis did not show any significant trend. This result is a demonstration of the higher effectiveness of neurophysiological measures with respect to subjective ones for Human-Computer Interaction assessment.

In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations.

Benchmarking Brain-Computer Interfaces Outside the Laboratory: The Cybathlon 2016

This paper presents a new approach to benchmarking brain-computer interfaces (BCIs) outside the lab. A computer game was created that mimics a real-world application of assistive BCIs, with the main outcome metric being the time needed to complete the game. This approach was used at the Cybathlon 2016, a competition for people with disabilities who use assistive technology to achieve tasks. The paper summarizes the technical challenges of BCIs, describes the design of the benchmarking game, then describes the rules for acceptable hardware, software and inclusion of human pilots in the BCI competition at the Cybathlon. The 11 participating teams, their approaches, and their results at the Cybathlon are presented. Though the benchmarking procedure has some limitations (for instance, we were unable to identify any factors that clearly contribute to BCI performance), it can be successfully used to analyze BCI performance in realistic, less structured conditions. In the future, the parameters of the benchmarking game could be modified to better mimic different applications (e.g., the need to use some commands more frequently than others). Furthermore, the Cybathlon has the potential to showcase such devices to the general public.

Brain Interaction during Cooperation: Evaluating Local Properties of Multiple-Brain Network

Subjects’ interaction is the core of most human activities. This is the reason why a lack of coordination is often the cause of missing goals, more than individual failure. While there are different subjective and objective measures to assess the level of mental effort required by subjects while facing a situation that is getting harder, that is, mental workload, to define an objective measure based on how and if team members are interacting is not so straightforward. In this study, behavioral, subjective and synchronized electroencephalographic data were collected from couples involved in a cooperative task to describe the relationship between task difficulty and team coordination, in the sense of interaction aimed at cooperatively performing the assignment. Multiple-brain connectivity analysis provided information about the whole interacting system. The results showed that averaged local properties of a brain network were affected by task difficulty. In particular, strength changed significantly with task difficulty and clustering coefficients strongly correlated with the workload itself. In particular, a higher workload corresponded to lower clustering values over the central and parietal brain areas. Such results has been interpreted as less efficient organization of the network when the subjects’ activities, due to high workload tendencies, were less coordinated.