Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG

Amputees but not healthy subjects optimally integrate non-spatially matched visuo-tactile stimuli

Our brain combines sensory inputs to create a univocal perception, enhanced when stimuli originate from the same location. Following amputation, distorted body representations may disrupt visuo-tactile integration at the amputated leg. We aim to unveil the principles guiding optimal and cognitive-efficient visuo-tactile integration at both intact and amputated legs. Hence, we designed a VR electro-stimulating platform to assess the functional and cognitive correlates of visuo-tactile integration in two amputees and sixteen healthy subjects performing a 2-alternative forced choice (2AFC) task. We showed that amputees optimally integrate non-spatially matched stimuli at the amputated leg but not the intact leg (tactile cue at the stump/thigh and visual cue under the virtual foot), while healthy controls only integrated spatially matched visuo-tactile stimuli. Optimal integration also reduced 2AFC task reaction times and was confirmed by cognitive EEG-based mental workload reduction. These findings offer insights into multisensory integration processes, opening new perspectives on amputees' brain plasticity.

Virtual reality as a treatment for chronic musculoskeletal pain syndromes

Chronic musculoskeletal pain syndromes, including fibromyalgia, are often resistant to conventional medications and invasive therapies. Central hypersensitization, neurotransmitter dysregulation, and autonomic nervous system abnormalities are key pathomechanisms, frequently resulting in widespread pain and a variety of psychosomatic symptoms. Virtual Reality (VR) applications have demonstrated effectiveness in reducing pain, both during and after interventions, and in chronic conditions such as fibromyalgia and back pain. The proposed mechanisms behind VR's effectiveness include distraction and immersion, coupled with cognitive behavioral therapy, which promote neuroplasticity and alter pain perceptions. Functional MRI studies have shown the impact of VR interventions on specific brain regions. Advances in hardware and software, potentially combined with treatments like biofeedback, could enhance VR's role in managing chronic pain. Currently, VR for musculoskeletal pain syndromes is primarily used within multimodal programs, but it is also available for home use as a standalone health application. Future research should focus on the 'drug-like' effects of VR, requiring controlled trials with comparable study populations and appropriate sham interventions.

HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes

We present HOIMotion - a novel approach for human motion forecasting during human-object interactions that integrates information about past body poses and egocentric 3D object bounding boxes. Human motion forecasting is important in many augmented reality applications but most existing methods have only used past body poses to predict future motion. HOIMotion first uses an encoder-residual graph convolutional network (GCN) and multi-layer perceptrons to extract features from body poses and egocentric 3D object bounding boxes, respectively. Our method then fuses pose and object features into a novel pose-object graph and uses a residual-decoder GCN to forecast future body motion. We extensively evaluate our method on the Aria digital twin (ADT) and MoGaze datasets and show that HOIMotion consistently outperforms state-of-the-art methods by a large margin of up to 8.7% on ADT and 7.2% on MoGaze in terms of mean per joint position error. Complementing these evaluations, we report a human study (N=20) that shows that the improvements achieved by our method result in forecasted poses being perceived as both more precise and more realistic than those of existing methods. Taken together, these results reveal the significant information content available in egocentric 3D object bounding boxes for human motion forecasting and the effectiveness of our method in exploiting this information.

Bridging minds and machines in Industry 5.0: neurobiological approach

Introduction

In transitioning from Industry 4.0 to the forthcoming Industry 5.0, this research explores the fusion of the humanistic view and technological developments to redefine Continuing Engineering Education (CEE). Industry 5.0 introduces concepts like biomanufacturing and human-centricity, embodying the integration of sustainability and resiliency principles in CEE, thereby shaping the upskilling and reskilling initiatives for the future workforce. The interaction of sophisticated concepts such as Human-Machine Interface and Brain-Computer Interface (BCI) forms a conceptual bridge toward the approaching Fifth Industrial Revolution, allowing one to understand human beings and the impact of their biological development across diverse and changing workplace settings.

Methods

Our research is based on recent studies into Knowledge, Skills, and Abilities taxonomies, linking these elements with dynamic labor market profiles. This work intends to integrate a biometric perspective to conceptualize and describe how cognitive abilities could be represented by linking a Neuropsychological test and a biometric assessment. We administered the brief Neuropsychological Battery in Spanish (Neuropsi Breve). At the same time, 15 engineering students used the Emotiv insight device that allowed the EEG recollection to measure performance metrics such as attention, stress, engagement, and excitement.

Results

The findings of this research illustrate a methodology that allowed the first approach to the cognitive abilities of engineering students to be from neurobiological and behavioral perspectives. Additionally, two profiles were extracted from the results. The first illustrates the Neuropsi test areas, its most common mistakes, and its performance ratings regarding the students' sample. The second profile shows the interaction between the EEG and Neuropsi test, showing engineering students' cognitive and emotional states based on biometric levels.

Discussions

The study demonstrates the potential of integrating neurobiological assessment into engineering education, highlighting a significant advancement in addressing the skills requirements of Industry 5.0. The results suggest that obtaining a comprehensive understanding of students' cognitive abilities is possible, and educational interventions can be adapted by combining neuropsychological approaches with EEG data collection. In the future, it is essential to refine these evaluation methods further and explore their applicability in different engineering disciplines. Additionally, it is necessary to investigate the long-term impact of these methods on workforce preparation and performance.

Towards electrophysiological measurement of presence in virtual reality through auditory oddball stimuli

Objective.Presence is an important aspect of user experience in virtual reality (VR). It corresponds to the illusion of being physically located in a virtual environment (VE). This feeling is usually measured through questionnaires that disrupt presence, are subjective and do not allow for real-time measurement. Electroencephalography (EEG), which measures brain activity, is increasingly used to monitor the state of users, especially while immersed in VR.Approach.In this paper, we present a way of evaluating presence, through the measure of the attention dedicated to the real environment via an EEG oddball paradigm. Using breaks in presence, this experimental protocol constitutes an ecological method for the study of presence, as different levels of presence are experienced in an identical VE.Main results.Through analysing the EEG data of 18 participants, a significant increase in the neurophysiological reaction to the oddball, i.e. the P300 amplitude, was found in low presence condition compared to high presence condition. This amplitude was significantly correlated with the self-reported measure of presence. Using Riemannian geometry to perform single-trial classification, we present a classification algorithm with 79% accuracy in detecting between two presence conditions.Significance.Taken together our results promote the use of EEG and oddball stimuli to monitor presence offline or in real-time without interrupting the user in the VE.

A Dual-Adversarial Model for Cross-Time and Cross-Subject Cognitive Workload Decoding

Electroencephalogram (EEG) signals are widely utilized in the field of cognitive workload decoding (CWD). However, when the recognition scenario is shifted from subject-dependent to subject-independent or spans a long period, the accuracy of CWD deteriorates significantly. Current solutions are either dependent on extensive training datasets or fail to maintain clear distinctions between categories, additionally lacking a robust feature extraction mechanism. In this paper, we tackle these issues by proposing a Bi-Classifier Joint Domain Adaptation (BCJDA) model for EEG-based cross-time and cross-subject CWD. Specifically, the model consists of a feature extractor, a domain discriminator, and a Bi-Classifier, containing two sets of adversarial processes for domain-wise alignment and class-wise alignment. In the adversarial domain adaptation, the feature extractor is forced to learn the common domain features deliberately. The Bi-Classifier also fosters the feature extractor to retain the category discrepancies of the unlabeled domain, so that its classification boundary is consistent with the labeled domain. Furthermore, different adversarial distance functions of the Bi-Classifier are adopted and evaluated in this model. We conduct classification experiments on a publicly available BCI competition dataset for recognizing low, medium, and high cognitive workload levels. The experimental results demonstrate that our proposed BCJDA model based on cross-gradient difference maximization achieves the best performance.

PLUME: Record, Replay, Analyze and Share User Behavior in 6DoF XR Experiences

From education to medicine to entertainment, a wide range of industrial and academic fields now utilize eXtended Reality (XR) technologies. This diversity and growing use are boosting research and leading to an increasing number of XR experiments involving human subjects. The main aim of these studies is to understand the user experience in the broadest sense, such as the user cognitive and emotional states. Behavioral data collected during XR experiments, such as user movements, gaze, actions, and physiological signals constitute precious assets for analyzing and understanding the user experience. While they contribute to overcome the intrinsic flaws of explicit data such as post-experiment questionnaires, the required acquisition and analysis tools are costly and challenging to develop, especially for 6DoF (Degrees of Freedom) XR experiments. Moreover, there is no common format for XR behavioral data, which restrains data-sharing, and thus hinders wide usages across the community, replicability of studies, and the constitution of large datasets or meta-analysis. In this context, we present PLUME, an open-source software toolbox (PLUME Recorder, PLUME Viewer, PLUME Python) that allows for the exhaustive record of XR behavioral data (including synchronous physiological signals), their offline interactive replay and analysis (with a standalone application), and their easy sharing due to our compact and interoperable data format. We believe that PLUME can greatly benefit the scientific community by making the use of behavioral and physiological data available for the greatest, contributing to the reproducibility and replicability of XR user studies, enabling the creation of large datasets, and contributing to a deeper understanding of user experience.

iVR-fNIRS: studying brain functions in a fully immersive virtual environment

Immersive virtual reality (iVR) employs head-mounted displays or cave-like environments to create a sensory-rich virtual experience that simulates the physical presence of a user in a digital space. The technology holds immense promise in neuroscience research and therapy. In particular, virtual reality (VR) technologies facilitate the development of diverse tasks and scenarios closely mirroring real-life situations to stimulate the brain within a controlled and secure setting. It also offers a cost-effective solution in providing a similar sense of interaction to users when conventional stimulation methods are limited or unfeasible. Although combining iVR with traditional brain imaging techniques may be difficult due to signal interference or instrumental issues, recent work has proposed the use of functional near infrared spectroscopy (fNIRS) in conjunction with iVR for versatile brain stimulation paradigms and flexible examination of brain responses. We present a comprehensive review of current research studies employing an iVR-fNIRS setup, covering device types, stimulation approaches, data analysis methods, and major scientific findings. The literature demonstrates a high potential for iVR-fNIRS to explore various types of cognitive, behavioral, and motor functions in a fully immersive VR (iVR) environment. Such studies should set a foundation for adaptive iVR programs for both training (e.g., in novel environments) and clinical therapeutics (e.g., pain, motor and sensory disorders and other psychiatric conditions).

Assessing Cognitive Workload in Motor Decision-Making through Functional Connectivity Analysis: Towards Early Detection and Monitoring of Neurodegenerative Diseases

Neurodegenerative diseases (NDs), such as Alzheimer's, Parkinson's, amyotrophic lateral sclerosis, and frontotemporal dementia, among others, are increasingly prevalent in the global population. The clinical diagnosis of these NDs is based on the detection and characterization of motor and non-motor symptoms. However, when these diagnoses are made, the subjects are often in advanced stages where neuromuscular alterations are frequently irreversible. In this context, we propose a methodology to evaluate the cognitive workload (CWL) of motor tasks involving decision-making processes. CWL is a concept widely used to address the balance between task demand and the subject's available resources to complete that task. In this study, multiple models for motor planning during a motor decision-making task were developed by recording EEG and EMG signals in n=17 healthy volunteers (9 males, 8 females, age 28.66±8.8 years). In the proposed test, volunteers have to make decisions about which hand should be moved based on the onset of a visual stimulus. We computed functional connectivity between the cortex and muscles, as well as among muscles using both corticomuscular and intermuscular coherence. Despite three models being generated, just one of them had strong performance. The results showed two types of motor decision-making processes depending on the hand to move. Moreover, the central processing of decision-making for the left hand movement can be accurately estimated using behavioral measures such as planning time combined with peripheral recordings like EMG signals. The models provided in this study could be considered as a methodological foundation to detect neuromuscular alterations in asymptomatic patients, as well as to monitor the process of a degenerative disease.

Decoding Eye Blink and Related EEG Activity in Realistic Working Environments

Accurately evaluating cognitive load during work-related tasks in complex real-world environments is challenging, leading researchers to investigate the use of eye blinking as a fundamental pacing mechanism for segmenting EEG data and understanding the neural mechanisms associated with cognitive workload. Yet, little is known about the temporal dynamics of eye blinks and related visual processing in relation to the representation of task-specific information. Therefore, we analyzed EEG responses from two experiments involving simulated driving (re-active and pro-active) with three levels of task load for each, as well as operating a steam engine (active vs. passive), to decode the temporal dynamics of eye blink activity and the subsequent neural activity that follows blinking. As a result, we successfully decoded the binary representation of difficulty levels for pro-active driving using multivariate pattern analysis. However, the decoding level varied for different re-active driving conditions, which could be attributed to the required level of alertness. Furthermore, our study revealed that it was possible to decode both driving types as well as steam engine operating conditions, with the most significant decoding activity observed approximately 200 ms after a blink. Additionally, our findings suggest that eye blinks have considerable potential for decoding various cognitive states that may not be discernible through neural activity, particularly near the peak of the blink. The findings demonstrate the potential of blink-related measures alongside EEG data to decode cognitive states during complex tasks, with implications for improving evaluations of cognitive and behavioral states during tasks, such as driving and operating machinery.

Raw Electroencephalogram-Based Cognitive Workload Classification Using Directed and Nondirected Functional Connectivity Analysis and Deep Learning

With the phenomenal rise in internet-of-things devices, the use of electroencephalogram (EEG) based brain-computer interfaces (BCIs) can empower individuals to control equipment with thoughts. These allow BCI to be used and pave the way for pro-active health management and the development of internet-of-medical-things architecture. However, EEG-based BCIs have low fidelity, high variance, and EEG signals are very noisy. These challenges compel researchers to design algorithms that can process big data in real-time while being robust to temporal variations and other variations in the data. Another issue in designing a passive BCI is the regular change in user's cognitive state (measured through cognitive workload). Though considerable amount of research has been conducted on this front, methods that could withstand high variability in EEG data and still reflect the neuronal dynamics of cognitive state variations are lacking and much needed in literature. In this research, we evaluate the efficacy of a combination of functional connectivity algorithms and state-of-the-art deep learning algorithms for the classification of three different levels of cognitive workload. We acquire 64-channel EEG data from 23 participants executing the n-back task at three different levels; 1-back (low-workload condition), 2-back (medium-workload condition), and 3-back (high-workload condition). We compared two different functional connectivity algorithms, namely phase transfer entropy (PTE) and mutual information (MI). PTE is a directed functional connectivity algorithm, whereas MI is non-directed. Both methods are suitable for extracting functional connectivity matrices in real-time, which could eventually be used for rapid, robust, and efficient classification. For classification, we use the recently proposed BrainNetCNN deep learning model, designed specifically to classify functional connectivity matrices. Results reveal a classification accuracy of 92.81% with MI and BrainNetCNN and a staggering 99.50% with PTE and BrainNetCNN on test data. PTE can yield a higher classification accuracy due to its robustness to linear mixing of the data and its ability to detect functional connectivity across a range of analysis lags.

Analysis of Physiological Response of Attention and Stress States in Teleoperation Performance of Social Tasks

Some studies addressed monitoring mental states by physiological responses analysis in robots' teleoperation in traditional applications such as inspection and exploration; however, no study analyzed the physiological response during teleoperated social tasks to the best of our knowledge. We analyzed the physiological response of attention and stress mental states by computing the correlation between multimodal biomarkers and performance, pleasure-arousal scale, and workload. Physiological data were recorded during simulated teleoperated social tasks to induce mental states, such as normal, attention, and stress. The results showed that task performance and workload subscales achieved moderate correlations with some multimodal biomarkers. The correlations depended on the induced state. The cognitive workload was related to brain biomarkers of attention in the frontal and frontal-central regions. These regions were close to the frontopolar region, which is commonly reported in attentional studies. Thus, some multimodal biomarkers of attention and stress mental states could monitor or predict metrics related to the performance in teleoperation of social tasks.

A Novel OpenBCI Framework for EEG-Based Neurophysiological Experiments

An Open Brain-Computer Interface (OpenBCI) provides unparalleled freedom and flexibility through open-source hardware and firmware at a low-cost implementation. It exploits robust hardware platforms and powerful software development kits to create customized drivers with advanced capabilities. Still, several restrictions may significantly reduce the performance of OpenBCI. These limitations include the need for more effective communication between computers and peripheral devices and more flexibility for fast settings under specific protocols for neurophysiological data. This paper describes a flexible and scalable OpenBCI framework for electroencephalographic (EEG) data experiments using the Cyton acquisition board with updated drivers to maximize the hardware benefits of ADS1299 platforms. The framework handles distributed computing tasks and supports multiple sampling rates, communication protocols, free electrode placement, and single marker synchronization. As a result, the OpenBCI system delivers real-time feedback and controlled execution of EEG-based clinical protocols for implementing the steps of neural recording, decoding, stimulation, and real-time analysis. In addition, the system incorporates automatic background configuration and user-friendly widgets for stimuli delivery. Motor imagery tests the closed-loop BCI designed to enable real-time streaming within the required latency and jitter ranges. Therefore, the presented framework offers a promising solution for tailored neurophysiological data processing.

EHTask: Recognizing User Tasks From Eye and Head Movements in Immersive Virtual Reality

Understanding human visual attention in immersive virtual reality (VR) is crucial for many important applications, including gaze prediction, gaze guidance, and gaze-contingent rendering. However, previous works on visual attention analysis typically only explored one specific VR task and paid less attention to the differences between different tasks. Moreover, existing task recognition methods typically focused on 2D viewing conditions and only explored the effectiveness of human eye movements. We first collect eye and head movements of 30 participants performing four tasks, i.e., Free viewing, Visual search, Saliency, and Track, in 15 360-degree VR videos. Using this dataset, we analyze the patterns of human eye and head movements and reveal significant differences across different tasks in terms of fixation duration, saccade amplitude, head rotation velocity, and eye-head coordination. We then propose EHTask - a novel learning-based method that employs eye and head movements to recognize user tasks in VR. We show that our method significantly outperforms the state-of-the-art methods derived from 2D viewing conditions both on our dataset (accuracy of 84.4% versus 62.8%) and on a real-world dataset ( 61.9% versus 44.1%). As such, our work provides meaningful insights into human visual attention under different VR tasks and guides future work on recognizing user tasks in VR.

Monitoring Brain Activity in VR: EEG and Neuroimaging

Virtual reality (VR) is increasingly used in neuroscientific research to increase ecological validity without sacrificing experimental control, to provide a richer visual and multisensory experience, and to foster immersion and presence in study participants, which leads to increased motivation and affective experience. But the use of VR, particularly when coupled with neuroimaging or neurostimulation techniques such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), or transcranial magnetic stimulation (TMS), also yields some challenges. These include intricacies of the technical setup, increased noise in the data due to movement, and a lack of standard protocols for data collection and analysis. This chapter examines current approaches to recording, pre-processing, and analyzing electrophysiological (stationary and mobile EEG), as well as neuroimaging data recorded during VR engagement. It also discusses approaches to synchronizing these data with other data streams. In general, previous research has used a range of different approaches to technical setup and data processing, and detailed reporting of procedures is urgently needed in future studies to ensure comparability and replicability. More support for open-source VR software as well as the development of consensus and best practice papers on issues such as the handling of movement artifacts in mobile EEG-VR will be essential steps in ensuring the continued success of this exciting and powerful technique in neuroscientific research.

A Survey on Affective and Cognitive VR

In Virtual Reality (VR), users can be immersed in emotionally intense and cognitively engaging experiences. Yet, despite strong interest from scholars and a large amount of work associating VR and Affective and Cognitive States (ACS), there is a clear lack of structured and systematic form in which this research can be classified. We define "Affective and Cognitive VR" to relate to works which (1) induce ACS, (2) recognize ACS, or (3) exploit ACS by adapting virtual environments based on ACS measures. This survey clarifies the different models of ACS, presents the methods for measuring them with their respective advantages and drawbacks in VR, and showcases Affective and Cognitive VR studies done in an Immersive Virtual Environment (IVE) in a non-clinical context. Our article covers the main research lines in Affective and Cognitive VR. We provide a comprehensive list of references with the analysis of 63 research articles and summarize future works directions.

Predictions of task using neural modeling

Introduction

A well-designed brain-computer interface (BCI) can make accurate and reliable predictions of a user's state through the passive assessment of their brain activity; in turn, BCI can inform an adaptive system (such as artificial intelligence, or AI) to intelligently and optimally aid the user to maximize the human-machine team (HMT) performance. Various groupings of spectro-temporal neural features have shown to predict the same underlying cognitive state (e.g., workload) but vary in their accuracy to generalize across contexts, experimental manipulations, and beyond a single session. In our work we address an outstanding challenge in neuroergonomic research: we quantify if (how) identified neural features and a chosen modeling approach will generalize to various manipulations defined by the same underlying psychological construct, (multi)task cognitive workload.

Methods

To do this, we train and test 20 different support vector machine (SVM) models, each given a subset of neural features as recommended from previous research or matching the capabilities of commercial devices. We compute each model's accuracy to predict which (monitoring, communications, tracking) and how many (one, two, or three) task(s) were completed simultaneously. Additionally, we investigate machine learning model accuracy to predict task(s) within- vs. between-sessions, all at the individual-level.

Results

Our results indicate gamma activity across all recording locations consistently outperformed all other subsets from the full model. Our work demonstrates that modelers must consider multiple types of manipulations which may each influence a common underlying psychological construct.

Discussion

We offer a novel and practical modeling solution for system designers to predict task through brain activity and suggest next steps in expanding our framework to further contribute to research and development in the neuroergonomics community. Further, we quantified the cost in model accuracy should one choose to deploy our BCI approach using a mobile EEG-systems with fewer electrodes-a practical recommendation from our work.

Effects of different virtual reality technology driven dual-tasking paradigms on posture and saccadic eye movements in healthy older adults

Postural sway and eye movements are potential biomarkers for dementia screening. Assessing the two movements comprehensively could improve the understanding of complicated syndrome for more accurate screening. The purpose of this research is to evaluate the effects of comprehensive assessment in healthy older adults (OA), using a novel concurrent comprehensive assessment system consisting of stabilometer and virtual reality headset. 20 healthy OA (70.4 ± 4.9 years) were recruited. Using a cross-sectional study design, this study investigated the effects of various dual-tasking paradigms with integrated tasks of visuospatial memory (VM), spatial orientation (SO), and visual challenge on posture and saccades. Dual-task paradigms with VM and SO affected the saccadic eye movements significantly. Two highly intensive tests of anti-saccade with VM task and pro-saccade with SO task also influenced postural sway significantly. Strong associations were seen between postural sway and eye movements for the conditions where the two movements theoretically shared common neural pathways in the brain, and vice versa. This study suggests that assessing posture and saccades with the integrated tasks comprehensively and simultaneously could be useful to explain different functions of the brain. The results warrant a cross-sectional study in OA with and without dementia to explore differences between these groups.

Assessment of instantaneous cognitive load imposed by educational multimedia using electroencephalography signals

The use of multimedia learning is increasing in modern education. On the other hand, it is crucial to design multimedia contents that impose an optimal amount of cognitive load, which leads to efficient learning. Objective assessment of instantaneous cognitive load plays a critical role in educational design quality evaluation. Electroencephalography (EEG) has been considered a potential candidate for cognitive load assessment among neurophysiological methods. In this study, we experiment to collect EEG signals during a multimedia learning task and then build a model for instantaneous cognitive load measurement. In the experiment, we designed four educational multimedia in two categories to impose different levels of cognitive load by intentionally applying/violating Mayer's multimedia design principles. Thirty university students with homogenous English language proficiency participated in our experiment. We divided them randomly into two groups, and each watched a version of the multimedia followed by a recall test task and filling out a NASA-TLX questionnaire. EEG signals are collected during these tasks. To construct the load assessment model, at first, power spectral density (PSD) based features are extracted from EEG signals. Using the minimum redundancy - maximum relevance (MRMR) feature selection approach, the best features are selected. In this way, the selected features consist of only about 12% of the total number of features. In the next step, we propose a scoring model using a support vector machine (SVM) for instantaneous cognitive load assessment in 3s segments of multimedia. Our experiments indicate that the selected feature set can classify the instantaneous cognitive load with an accuracy of 84.5 ± 2.1%. The findings of this study indicate that EEG signals can be used as an appropriate tool for measuring the cognitive load introduced by educational videos. This can be help instructional designers to develop more effective content.

A meta-learning BCI for estimating decision confidence

OBJECTIVE

We investigated whether a recently introduced transfer-learning technique based on meta-learning could improve the performance of Brain-Computer Interfaces (BCIs) for decision-confidence prediction with respect to more traditional machine learning methods.

APPROACH

We adapted the meta-learning by biased regularisation algorithm to the problem of predicting decision confidence from EEG and EOG data on a decision-by-decision basis in a difficult target discrimination task based on video feeds. The method exploits previous participants' data to produce a prediction algorithm that is then quickly tuned to new participants. We compared it with with the traditional single-subject training almost universally adopted in BCIs, a state-of-the-art transfer learning technique called Domain Adversarial Neural Networks (DANN), a transfer-learning adaptation of a zero-training method we used recently for a similar task, and with a simple baseline algorithm.

MAIN RESULTS

The meta-learning approach was significantly better than other approaches in most conditions, and much better in situations where limited data from a new participant are available for training/tuning. Meta-learning by biased regularisation allowed our BCI to seamlessly integrate information from past participants with data from a specific user to produce high-performance predictors. Its robustness in the presence of small training sets is a real-plus in BCI applications, as new users need to train the BCI for a much shorter period.

SIGNIFICANCE

Due to the variability and noise of EEG/EOG data, BCIs need to be normally trained with data from a specific participant. This work shows that even better performance can be obtained using our version of meta-learning by biased regularisation.

Improved Cognitive Vigilance Assessment after Artifact Reduction with Wavelet Independent Component Analysis

Vigilance level assessment is of prime importance to avoid life-threatening human error. Critical working environments such as air traffic control, driving, or military surveillance require the operator to be alert the whole time. The electroencephalogram (EEG) is a very common modality that can be used in assessing vigilance. Unfortunately, EEG signals are prone to artifacts due to eye movement, muscle contraction, and electrical noise. Mitigating these artifacts is important for an accurate vigilance level assessment. Independent Component Analysis (ICA) is an effective method and has been extensively used in the suppression of EEG artifacts. However, in vigilance assessment applications, it was found to suffer from leakage of the cerebral activity into artifacts. In this work, we show that the wavelet ICA (wICA) method provides an alternative for artifact reduction, leading to improved vigilance level assessment results. We conducted an experiment in nine human subjects to induce two vigilance states, alert and vigilance decrement, while performing a Stroop Color-Word Test for approximately 45 min. We then compared the performance of the ICA and wICA preprocessing methods using five classifiers. Our classification results showed that in terms of features extraction, the wICA method outperformed the existing ICA method. In the delta, theta, and alpha bands, we obtained a mean classification accuracy of 84.66% using the ICA method, whereas the mean accuracy using the wICA methodwas 96.9%. However, no significant improvement was observed in the beta band. In addition, we compared the topographical map to show the changes in power spectral density across the brain regions for the two vigilance states. The proposed method showed that the frontal and central regions were most sensitive to vigilance decrement. However, in this application, the proposed wICA shows a marginal improvement compared to the Fast-ICA.

EEG-Based Identification of Emotional Neural State Evoked by Virtual Environment Interaction

Classifying emotional states is critical for brain–computer interfaces and psychology-related domains. In previous studies, researchers have tried to identify emotions using neural data such as electroencephalography (EEG) signals or brain functional magnetic resonance imaging (fMRI). In this study, we propose a machine learning framework for emotion state classification using EEG signals in virtual reality (VR) environments. To arouse emotional neural states in brain signals, we provided three VR stimuli scenarios to 15 participants. Fifty-four features were extracted from the collected EEG signals under each scenario. To find the optimal classification in our research design, three machine learning algorithms (XGBoost classifier, support vector classifier, and logistic regression) were applied. Additionally, various class conditions were used in machine learning classifiers to validate the performance of our framework. To evaluate the classification performance, we utilized five evaluation metrics (precision, recall, f1-score, accuracy, and AUROC). Among the three classifiers, the XGBoost classifiers showed the best performance under all experimental conditions. Furthermore, the usability of features, including differential asymmetry and frequency band pass categories, were checked from the feature importance of XGBoost classifiers. We expect that our framework can be applied widely not only to psychological research but also to mental health-related issues.

The challenges of entering the metaverse: An experiment on the effect of extended reality on workload

Information technologies exist to enable us to either do things we have not done before or do familiar things more efficiently. Metaverse (i.e. extended reality: XR) enables novel forms of engrossing telepresence, but it also may make mundate tasks more effortless. Such technologies increasingly facilitate our work, education, healthcare, consumption and entertainment; however, at the same time, metaverse bring a host of challenges. Therefore, we pose the question whether XR technologies, specifically Augmented Reality (AR) and Virtual Reality (VR), either increase or decrease the difficulties of carrying out everyday tasks. In the current study we conducted a 2 (AR: with vs. without) × 2 (VR: with vs. without) between-subject experiment where participants faced a shopping-related task (including navigating, movement, hand-interaction, information processing, information searching, storing, decision making, and simple calculation) to examine a proposed series of hypotheses. The NASA Task Load Index (NASA-TLX) was used to measure subjective workload when using an XR-mediated information system including six sub-dimensions of frustration, performance, effort, physical, mental, and temporal demand. The findings indicate that AR was significantly associated with overall workload, especially mental demand and effort, while VR had no significant effect on any workload sub-dimensions. There was a significant interaction effect between AR and VR on physical demand, effort, and overall workload. The results imply that the resources and cost of operating XR-mediated realities are different and higher than physical reality.

Cognitive Workload of Tugboat Captains in Realistic Scenarios: Adaptive Spatial Filtering for Transfer Between Conditions

Changing and often class-dependent non-stationarities of signals are a big challenge in the transfer of common findings in cognitive workload estimation using Electroencephalography (EEG) from laboratory experiments to realistic scenarios or other experiments. Additionally, it often remains an open question whether actual cognitive workload reflected by brain signals was the main contribution to the estimation or discriminative and class-dependent muscle and eye activity, which can be secondary effects of changing workload levels. Within this study, we investigated a novel approach to spatial filtering based on beamforming adapted to changing settings. We compare it to no spatial filtering and Common Spatial Patterns (CSP). We used a realistic maneuvering task, as well as an auditory n-back secondary task on a tugboat simulator as two different conditions to induce workload changes on professional tugboat captains. Apart from the typical within condition classification, we investigated the ability of the different classification methods to transfer between the n-back condition and the maneuvering task. The results show a clear advantage of the proposed approach over the others in the challenging transfer setting. While no filtering leads to lowest within-condition normalized classification loss on average in two scenarios (22 and 10%), our approach using adaptive beamforming (30 and 18%) performs comparably to CSP (33 and 15%). Importantly, in the transfer from one to another setting, no filtering and CSP lead to performance around chance level (45 to 53%), while our approach in contrast is the only one capable of classifying in all other scenarios (34 and 35%) with a significant difference from chance level. The changing signal composition over the scenarios leads to a need to adapt the spatial filtering in order to be transferable. With our approach, the transfer is successful due to filtering being optimized for the extraction of neural components and additional investigation of their scalp patterns revealed mainly neural origin. Interesting findings are that rather the patterns slightly change between conditions. We conclude that the approaches with low normalized loss depend on eye and muscle activity which is successful for classification within conditions, but fail in the classifier transfer since eye and muscle contributions are highly condition-specific.

Ecological validity in exercise neuroscience research: A systematic investigation

The contribution of cortical processes to adaptive motor behaviour is of great interest in the field of exercise neuroscience. Next to established criteria of objectivity, reliability and validity, ecological validity refers to the concerns of whether measurements and behaviour in research settings are representative of the real world. Because exercise neuroscience investigations using mobile electroencephalography are oftentimes conducted in laboratory settings under controlled environments, methodological approaches may interfere with the idea of ecological validity. This review utilizes an original ecological validity tool to assess the degree of ecological validity in current exercise neuroscience research. A systematic literature search was conducted to identify articles investigating cortical dynamics during goal-directed sports movement. To assess ecological validity, five elements (environment, stimulus, response, body and mind) were assessed on a continuum of artificiality-naturality and simplicity-complexity. Forty-seven studies were included in the present review. Results indicate lowest average ratings for the element of environment. The elements stimulus, body and mind had mediocre ratings, and the element of response had the highest overall ratings. In terms of the type of sport, studies that assessed closed-skill indoor sports had the highest ratings, whereas closed-skill outdoor sports had the lowest overall rating. Our findings identify specific elements that are lacking in ecological validity and areas of improvement in current exercise neuroscience literature. Future studies may potentially increase ecological validity by moving from reductionist, artificial environments towards complex, natural environments and incorporating real-world sport elements such as adaptive responses and competition.

Association between the total amount of electromagnetic cortical neuronal activity and a decline in motivation

In situations involving fatigue, the increase in fatigue levels and the apparent decrease in motivation levels are thought to suppress mental and physical performance to avoid disrupting homeostasis and aid recovery; however, the ultimate source of information on which the brain depends to perceive fatigue and/or a loss of motivation for protection remains unknown. In this study, we found that, as assessed by magnetoencephalography, electromagnetic cortical neuronal activity while performing cognitive tasks was associated with a decrease in motivation caused by the tasks in healthy participants, suggesting the possibility that the brain utilizes information that reflects the invested amount of neural activity to suppress performance. To our knowledge, this is the first report to provide clues for the missing link between neural investments and the resulting activation of the biological alarms that suppress performance.

A comparative experimental study of visual brain event-related potentials to a working memory task: virtual reality head-mounted display versus a desktop computer screen

Virtual reality head mounted display (VR HMD) systems are increasingly utilised in combination with electroencephalography (EEG) in the experimental study of cognitive tasks. The aim of our investigation was to determine the similarities/differences between VR HMD and the computer screen (CS) in response to an n-back working memory task by comparing visual electrophysiological event-related potential (ERP) waveforms (N1/P1/P3 components). The same protocol was undertaken for VR HMD and CS with participants wearing the same EEG headcap. ERP waveforms obtained with the VR HMD environment followed a similar time course to those acquired in CS. The P3 mean and peak amplitudes obtained in VR HMD were not significantly different to those obtained in CS. In contrast, the N1 component was significantly higher in mean and peak amplitudes for the VR HMD environment compared to CS at the frontal electrodes. Significantly higher P1 mean and peak amplitudes were found at the occipital region compared to the temporal for VR HMD. Our results show that successful acquisition of ERP components to a working memory task is achievable by combining VR HMD with EEG. In addition, the higher amplitude N1/P1 components seen in VR HMD indicates the potential utility of this VR modality in the investigation of early ERPs. In conclusion, the combination of VR HMD with EEG/ERP would be a useful approach to advance the study of cognitive function in experimental brain research.

Workplace Mental State Monitoring during VR-Based Training for Offshore Environment

Adults are constantly exposed to stressful conditions at their workplace, and this can lead to decreased job performance followed by detrimental clinical health problems. Advancement of sensor technologies has allowed the electroencephalography (EEG) devices to be portable and used in real-time to monitor mental health. However, real-time monitoring is not often practical in workplace environments with complex operations such as kindergarten, firefighting and offshore facilities. Integrating the EEG with virtual reality (VR) that emulates workplace conditions can be a tool to assess and monitor mental health of adults within their working environment. This paper evaluates the mental states induced when performing a stressful task in a VR-based offshore environment. The theta, alpha and beta frequency bands are analysed to assess changes in mental states due to physical discomfort, stress and concentration. During the VR trials, mental states of discomfort and disorientation are observed with the drop of theta activity, whilst the stress induced from the conditional tasks is reflected in the changes of low-alpha and high-beta activities. The deflection of frontal alpha asymmetry from negative to positive direction reflects the learning effects from emotion-focus to problem-solving strategies adopted to accomplish the VR task. This study highlights the need for an integrated VR-EEG system in workplace settings as a tool to monitor and assess mental health of working adults.

Real-Time EEG-Based Cognitive Workload Monitoring on Wearable Devices

OBJECTIVE

Cognitive workload monitoring (CWM) can enhance human-machine interaction by supporting task execution assistance considering the operators cognitive state. Therefore, we propose a machine learning design methodology and a data processing strategy to enable CWM on resource-constrained wearable devices.

METHODS

Our CWM solution is built upon edge computing on a simple wearable system, with only four peripheral channels of electroencephalography (EEG). We assess our solution on experimental data from 24 volunteers. Moreover, to overcome the system's memory constraints, we adopt an optimization strategy for model size reduction and a multi-batch data processing scheme for optimizing RAM memory footprint. Finally, we implement our data processing strategy on a state-of-the-art wearable platform and assess its execution and system battery life.

RESULTS

We achieve an accuracy of 74.5% and a 74.0% geometric mean between sensitivity and specificity for CWM classification on unseen data. Besides, the proposed model optimization strategy generates a 27.5x smaller model compared to the one generated with default parameters, and the multi-batch data processing scheme reduces RAM memory footprint by 14x compared to a single batch data processing. Finally, our algorithm uses only 1.28% of the available processing time, thus, allowing our system to achieve 28.5 hours of battery life.

CONCLUSION

We provide a reliable and optimized CWM solution using wearable devices, enabling more than a day of operation on a single battery charge.

SIGNIFICANCE

The proposed methodology enables real-time data processing on resource-constrained devices and supports real-time wearable monitoring based on EEG for applications as CWM in human-machine interaction.

Subject- and task-independent neural correlates and prediction of decision confidence in perceptual decision making

Objective.In many real-world decision tasks, the information available to the decision maker is incomplete. To account for this uncertainty, we associate a degree of confidence to every decision, representing the likelihood of that decision being correct. In this study, we analyse electroencephalography (EEG) data from 68 participants undertaking eight different perceptual decision-making experiments. Our goals are to investigate (1) whether subject- and task-independent neural correlates of decision confidence exist, and (2) to what degree it is possible to build brain computer interfaces that can estimate confidence on a trial-by-trial basis. The experiments cover a wide range of perceptual tasks, which allowed to separate the task-related, decision-making features from the task-independent ones.Approach.Our systems train artificial neural networks to predict the confidence in each decision from EEG data and response times. We compare the decoding performance with three training approaches: (1) single subject, where both training and testing data were acquired from the same person; (2) multi-subject, where all the data pertained to the same task, but the training and testing data came from different users; and (3) multi-task, where the training and testing data came from different tasks and subjects. Finally, we validated our multi-task approach using data from two additional experiments, in which confidence was not reported.Main results.We found significant differences in the EEG data for different confidence levels in both stimulus-locked and response-locked epochs. All our approaches were able to predict the confidence between 15% and 35% better than the corresponding reference baselines.Significance.Our results suggest that confidence in perceptual decision making tasks could be reconstructed from neural signals even when using transfer learning approaches. These confidence estimates are based on the decision-making process rather than just the confidence-reporting process.

Progress in Brain Computer Interface: Challenges and Opportunities

Brain computer interfaces (BCI) provide a direct communication link between the brain and a computer or other external devices. They offer an extended degree of freedom either by strengthening or by substituting human peripheral working capacity and have potential applications in various fields such as rehabilitation, affective computing, robotics, gaming, and neuroscience. Significant research efforts on a global scale have delivered common platforms for technology standardization and help tackle highly complex and non-linear brain dynamics and related feature extraction and classification challenges. Time-variant psycho-neurophysiological fluctuations and their impact on brain signals impose another challenge for BCI researchers to transform the technology from laboratory experiments to plug-and-play daily life. This review summarizes state-of-the-art progress in the BCI field over the last decades and highlights critical challenges.

Neurophysiological Correlates of Cognition as Revealed by Virtual Reality: Delving the Brain with a Synergistic Approach

The synergy of perceptual psychology, technology, and neuroscience can be used to comprehend how virtual reality affects cognition of human brain. Numerous studies have used neuroimaging modalities to assess the cognitive state and response of the brain with various external stimulations. The virtual reality-based devices are well known to incur visual, auditory, and haptic induced perceptions. Neurophysiological recordings together with virtual stimulations can assist in correlating humans’ physiological perception with response in the environment designed virtually. The effective combination of these two has been utilized to study human behavior, spatial navigation performance, and spatial presence, to name a few. Moreover, virtual reality-based devices can be evaluated for the neurophysiological correlates of cognition through neurophysiological recordings. Challenges exist in the integration of real-time neuronal signals with virtual reality-based devices, and enhancing the experience together with real-time feedback and control through neuronal signals. This article provides an overview of neurophysiological correlates of cognition as revealed by virtual reality experience, together with a description of perception and virtual reality-based neuromodulation, various applications, and existing challenges in this field of research.

The Current Research of Combining Multi-Modal Brain-Computer Interfaces With Virtual Reality

Combing brain-computer interfaces (BCI) and virtual reality (VR) is a novel technique in the field of medical rehabilitation and game entertainment. However, the limitations of BCI such as a limited number of action commands and low accuracy hinder the widespread use of BCI-VR. Recent studies have used hybrid BCIs that combine multiple BCI paradigms and/or the multi-modal biosensors to alleviate these issues, which may become the mainstream of BCIs in the future. The main purpose of this review is to discuss the current status of multi-modal BCI-VR. This study first reviewed the development of the BCI-VR, and explored the advantages and disadvantages of incorporating eye tracking, motor capture, and myoelectric sensing into the BCI-VR system. Then, this study discussed the development trend of the multi-modal BCI-VR, hoping to provide a pathway for further research in this field.