Passive BCI based on drowsiness detection: an fNIRS study

Mental fatigue causes significant activation of the prefrontal cortex: A systematic review and meta-analysis of fNIRS studies

Mental fatigue, a psychobiological prevalent and underestimated condition, is defined by increased lethargy and impaired concentration. This condition is not restricted by age and is exacerbated by various predisposing factors. Prolonged mental fatigue in occupational environments raises the probability of accidents or fatalities. Its fundamental mechanism is largely obscure and inherently subjective, thus there is no universally accepted parameter for its detection. Recently, there has been an increase in research that focuses on the use of functional near-infrared spectroscopy (fNIRS) to observe changes in brain hemoglobin during mental fatigue. Thus, this study assessed the reliability of oxygenhemoglobin and deoxyhemoglobin as fatigue biomarkers and conducted a systematic review and meta-analysis of studies which used fNIRS to monitor mental fatigue. The findings revealed significant activation of the prefrontal lobe under mental fatigue, and its activation level is intricately associated with the monitoring of diverse states during mental fatigue. Importantly, the type of induced mental fatigue and whether pre-trial training was provided to subjects were independent of the prefrontal lobe activation level. Overall, fNIRS proves to be an effective tool in tracking brain activity during mental fatigue, with a highly active prefrontal cortex acting as a dependable indicator for early identification of mental fatigue.

Investigating the relationship between CSAT scores and prefrontal fNIRS signals during cognitive tasks using a quantum annealing algorithm

Academic achievement is a critical measure of intellectual ability, prompting extensive research into cognitive tasks as potential predictors. Neuroimaging technologies, such as functional near-infrared spectroscopy (fNIRS), offer insights into brain hemodynamics, allowing understanding of the link between cognitive performance and academic achievement. Herein, we explored the association between cognitive tasks and academic achievement by analyzing prefrontal fNIRS signals. A novel quantum annealer (QA) feature selection algorithm was applied to fNIRS data to identify cognitive tasks correlated with CSAT scores. Twelve features (signal mean, median, variance, peak, number of peaks, sum of peaks, range, minimum, kurtosis, skewness, standard deviation, and root mean square) were extracted from fNIRS signals at two time windows (10- and 60-s) to compare results from various feature variable conditions. The feature selection results from the QA-based and XGBoost regressor algorithms were compared to validate the former's performance. In a two-step validation process using multiple linear regression models, model fitness (adjusted R2) and model prediction error (RMSE) values were calculated. The quantum annealer demonstrated comparable performance to classical machine learning models, and specific cognitive tasks, including verbal fluency, recognition, and the Corsi block tapping task, were correlated with academic achievement. Group analyses revealed stronger associations between Tower of London and N-back tasks with higher CSAT scores. Quantum annealing algorithms have significant potential in feature selection using fNIRS data, and represents a novel research approach. Future studies should explore predictors of academic achievement and cognitive ability.

Working memory load recognition with deep learning time series classification

Working memory load (WML) is one of the widely applied signals in the areas of human-machine interaction. The precise evaluation of the WML is crucial for this kind of application. This study aims to propose a deep learning (DL) time series classification (TSC) model for inter-subject WML decoding. We used fNIRS to record the hemodynamic signals of 27 participants during visual working memory tasks. Traditional machine learning and deep time series classification algorithms were respectively used for intra-subject and inter-subject WML decoding from the collected blood oxygen signals. The intra-subject classification accuracy of LDA and SVM were 94.6% and 79.1%. Our proposed TAResnet-BiLSTM model had the highest inter-subject WML decoding accuracy, reaching 92.4%. This study provides a new idea and method for the brain-computer interface application of fNIRS in real-time WML detection.

Trust in Shared-Space Collaborative Robots: Shedding Light on the Human Brain

BACKGROUND

Industry 4.0 is currently underway allowing for improved manufacturing processes that leverage the collective advantages of human and robot agents. Consideration of trust can improve the quality and safety in such shared-space human-robot collaboration environments.

OBJECTIVE

The use of physiological response to monitor and understand trust is currently limited due to a lack of knowledge on physiological indicators of trust. This study examines neural responses to trust within a shared-workcell human-robot collaboration task as well as discusses the use of granular and multimodal perspectives to study trust.

METHODS

Sixteen sex-balanced participants completed a surface finishing task in collaboration with a UR10 collaborative robot. All participants underwent robot reliability conditions and robot assistance level conditions. Brain activation and connectivity using functional near infrared spectroscopy, subjective responses, and performance were measured throughout the study.

RESULTS

Significantly, increased neural activation was observed in response to faulty robot behavior within the medial and right dorsolateral prefrontal cortex (PFC). A similar trend was observed for the anterior PFC, primary motor cortex, and primary visual cortex. Faulty robot behavior also resulted in reduced functional connectivity strengths throughout the brain.

DISCUSSION

These findings implicate regions in the prefrontal cortex along with specific connectivity patterns as signifiers of distrusting conditions. The neural response may be indicative of how trust is influenced, measured, and manifested for human-robot collaboration that requires active teaming.

APPLICATION

Neuroergonomic response metrics can reveal new perspectives on trust in automation that subjective responses alone are not able to provide.

SSVEP-based brain-computer interface for music using a low-density EEG system

In this paper, we present a bespoke brain-computer interface (BCI), which was developed for a person with severe motor-impairments, who was previously a Violinist, to allow performing and composing music at home. It uses steady-state visually evoked potential (SSVEP) and adopts a dry, low-density, and wireless electroencephalogram (EEG) headset. In this study, we investigated two parameters: (1) placement of the EEG headset and (2) inter-stimulus distance and found that the former significantly improved the information transfer rate (ITR). To analyze EEG, we adopted canonical correlation analysis (CCA) without weight-calibration. The BCI for musical performance realized a high ITR of 37.59 ± 9.86 bits min-1 and a mean accuracy of 88.89 ± 10.09%. The BCI for musical composition obtained an ITR of 14.91 ± 2.87 bits min-1 and a mean accuracy of 95.83 ± 6.97%. The BCI was successfully deployed to the person with severe motor-impairments. She regularly uses it for musical composition at home, demonstrating how BCIs can be translated from laboratories to real-world scenarios.

CGAN-rIRN: a data-augmented deep learning approach to accurate classification of mental tasks for a fNIRS-based brain-computer interface

Functional near-infrared spectroscopy (fNIRS) is increasingly used to investigate different mental tasks for brain-computer interface (BCI) control due to its excellent environmental and motion robustness. Feature extraction and classification strategy for fNIRS signal are essential to enhance the classification accuracy of voluntarily controlled BCI systems. The limitation of traditional machine learning classifiers (MLCs) lies in manual feature engineering, which is considered as one of the drawbacks that reduce accuracy. Since the fNIRS signal is a typical multivariate time series with multi-dimensionality and complexity, it makes the deep learning classifier (DLC) ideal for classifying neural activation patterns. However, the inherent bottleneck of DLCs is the requirement of substantial-scale, high-quality labeled training data and expensive computational resources to train deep networks. The existing DLCs for classifying mental tasks do not fully consider the temporal and spatial properties of fNIRS signals. Therefore, a specifically-designed DLC is desired to classify multi-tasks with high accuracy in fNIRS-BCI. To this end, we herein propose a novel data-augmented DLC to accurately classify mental tasks, which employs a convolution-based conditional generative adversarial network (CGAN) for data augmentation and a revised Inception-ResNet (rIRN) based DLC. The CGAN is utilized to generate class-specific synthetic fNIRS signals to augment the training dataset. The network architecture of rIRN is elaborately designed in accordance with the characteristics of the fNIRS signal, with serial multiple spatial and temporal feature extraction modules (FEMs), where each FEM performs deep and multi-scale feature extraction and fusion. The results of the paradigm experiments show that the proposed CGAN-rIRN approach improves the single-trial accuracy for mental arithmetic and mental singing tasks in both the data augmentation and classifier, as compared to the traditional MLCs and the commonly used DLCs. The proposed fully data-driven hybrid deep learning approach paves a promising way to improve the classification performance of volitional control fNIRS-BCI.

Metaheuristic Optimization-Based Feature Selection for Imagery and Arithmetic Tasks: An fNIRS Study

In recent decades, the brain-computer interface (BCI) has emerged as a leading area of research. The feature selection is vital to reduce the dataset's dimensionality, increase the computing effectiveness, and enhance the BCI's performance. Using activity-related features leads to a high classification rate among the desired tasks. This study presents a wrapper-based metaheuristic feature selection framework for BCI applications using functional near-infrared spectroscopy (fNIRS). Here, the temporal statistical features (i.e., the mean, slope, maximum, skewness, and kurtosis) were computed from all the available channels to form a training vector. Seven metaheuristic optimization algorithms were tested for their classification performance using a k-nearest neighbor-based cost function: particle swarm optimization, cuckoo search optimization, the firefly algorithm, the bat algorithm, flower pollination optimization, whale optimization, and grey wolf optimization (GWO). The presented approach was validated based on an available online dataset of motor imagery (MI) and mental arithmetic (MA) tasks from 29 healthy subjects. The results showed that the classification accuracy was significantly improved by utilizing the features selected from the metaheuristic optimization algorithms relative to those obtained from the full set of features. All of the abovementioned metaheuristic algorithms improved the classification accuracy and reduced the feature vector size. The GWO yielded the highest average classification rates (p < 0.01) of 94.83 ± 5.5%, 92.57 ± 6.9%, and 85.66 ± 7.3% for the MA, MI, and four-class (left- and right-hand MI, MA, and baseline) tasks, respectively. The presented framework may be helpful in the training phase for selecting the appropriate features for robust fNIRS-based BCI applications.

A Hybrid GCN and Filter-Based Framework for Channel and Feature Selection: An fNIRS-BCI Study

In this study, a channel and feature selection methodology is devised for brain-computer interface (BCI) applications using functional near-infrared spectroscopy (fNIRS). A graph convolutional network (GCN) is employed to select the appropriate and correlated fNIRS channels. Furthermore, in the feature extraction phase, the performance of two filter-based feature selection algorithms, (i) the minimum redundancy maximum relevance (mRMR) and (ii) ReliefF, is investigated. The five most commonly used temporal statistical features (i.e., mean, slope, maximum, skewness, and kurtosis) are used, whereas the conventional support vector machine (SVM) is utilized as a classifier for training and testing. The proposed methodology is validated using an available online dataset of motor imagery (left- and right-hand), mental arithmetic, and baseline tasks. First, the efficacy of the proposed methodology is shown for two-class BCI applications (i.e., left- vs. right-hand motor imagery and mental arithmetic vs. baseline). Second, the proposed framework is applied to four-class BCI applications (i.e., left- vs. right-hand motor imagery vs. mental arithmetic vs. baseline). The results show that the number of appropriate channels and features was significantly reduced, resulting in a significant increase in classification accuracy for both two-class and four-class BCI applications, respectively. Furthermore, both mRMR (i.e., 87.8% for motor imagery, 87.1% for mental arithmetic, and 78.7% for four-class) and ReliefF (i.e., 90.7% for motor imagery, 93.7% for mental arithmetic, and 81.6% for four-class) yielded high average classification accuracy $\left(p<0.05\right)$ . However, the results of the ReliefF algorithm are more stable and significant.

Assessing the development of mental fatigue during simulated flights with concurrent EEG-fNIRS measurement

Mental fatigue (MF) can impair pilots' performance and reactions to unforeseen events and is therefore an important concept within aviation. The physiological measurement of MF, especially with EEG and, in recent years, fNIRS, has gained much attention. However, a systematic investigation and comparison of the measurements is seldomly done. We induced MF via time on task during a 90-min simulated flight task and collected concurrent EEG-fNIRS, performance and self-report data from 31 participants. While their subjective MF increased linearly, the participants were able to keep their performance stable over the course of the experiment. EEG data showed an early increase and levelling in parietal alpha power and a slower, but steady increase in frontal theta power. No consistent trend could be observed in the fNIRS data. Thus, more research on fNIRS is needed to understand its possibilities and limits for MF assessment, and a combination with EEG is advisable to compare and validate results. Until then, EEG remains the better choice for continuous MF assessment in cockpit applications because of its high sensitivity to a transition from alert to fatigued, even before performance is impaired.

EEG-based detection of modality-specific visual and auditory sensory processing

Objective.A passive brain-computer interface (pBCI) is a system that enhances a human-machine interaction by monitoring the mental state of the user and, based on this implicit information, making appropriate modifications to the interaction. Key to the development of such a system is the ability to reliably detect the mental state of interest via neural signals. Many different mental states have been investigated, including fatigue, attention and various emotions, however one of the most commonly studied states is mental workload, i.e. the amount of attentional resources required to perform a task. The emphasis of mental workload studies to date has been almost exclusively on detecting and predicting the 'level' of cognitive resources required (e.g. high vs. low), but we argue that having information regarding the specific 'type' of resources (e.g. visual or auditory) would allow the pBCI to apply more suitable adaption techniques than would be possible knowing just the overall workload level.Approach.15 participants performed carefully designed visual and auditory tasks while electroencephalography (EEG) data was recorded. The tasks were designed to be as similar as possible to one another except for the type of attentional resources required. The tasks were performed at two different levels of demand. Using traditional machine learning algorithms, we investigated, firstly, if EEG can be used to distinguish between auditory and visual processing tasks and, secondly, what effect level of sensory processing demand has on the ability to distinguish between auditory and visual processing tasks.Main results.The results show that at the high level of demand, the auditory vs. visual processing tasks could be distinguished with an accuracy of 77.1% on average. However, in the low demand condition in this experiment, the tasks were not classified with an accuracy exceeding chance.Significance.These results support the feasibility of developing a pBCI for detecting not only the level, but also the type, of attentional resources being required of the user at a given time. Further research is required to determine if there is a threshold of demand under which the type of sensory processing cannot be detected, but even if that is the case, these results are still promising since it is the high end of demand that is of most concern in safety critical scenarios. Such a BCI could help improve safety in high risk occupations by initiating the most effective and efficient possible adaptation strategies when high workload conditions are detected.

Scalable Gamma-Driven Multilayer Network for Brain Workload Detection Through Functional Near-Infrared Spectroscopy

This work proposes a scalable gamma non-negative matrix network (SGNMN), which uses a Poisson randomized Gamma factor analysis to obtain the neurons of the first layer of a network. These neurons obey Gamma distribution whose shape parameter infers the neurons of the next layer of the network and their related weights. Upsampling the connection weights follows a Dirichlet distribution. Downsampling hidden units obey Gamma distribution. This work performs up-down sampling on each layer to learn the parameters of SGNMN. Experimental results indicate that the width and depth of SGNMN are closely related, and a reasonable network structure for accurately detecting brain fatigue through functional near-infrared spectroscopy can be obtained by considering network width, depth, and parameters.

Motion artifacts removal and evaluation techniques for functional near-infrared spectroscopy signals: A review

With the emergence of an increasing number of functional near-infrared spectroscopy (fNIRS) devices, the significant deterioration in measurement caused by motion artifacts has become an essential research topic for fNIRS applications. However, a high requirement for mathematics and programming limits the number of related researches. Therefore, here we provide the first comprehensive review for motion artifact removal in fNIRS aiming to (i) summarize the latest achievements, (ii) present the significant solutions and evaluation metrics from the perspective of application and reproduction, and (iii) predict future topics in the field. The present review synthesizes information from fifty-one journal articles (screened according to three criteria). Three hardware-based solutions and nine algorithmic solutions are summarized, and their application requirements (compatible signal types, the availability for online applications, and limitations) and extensions are discussed. Five metrics for noise suppression and two metrics for signal distortion were synthesized to evaluate the motion artifact removal methods. Moreover, we highlight three deficiencies in the existing research: (i) The balance between the use of auxiliary hardware and that of an algorithmic solution is not clarified; (ii) few studies mention the filtering delay of the solutions, and (iii) the robustness and stability of the solution under extreme application conditions are not discussed.

Riemannian distance based channel selection and feature extraction combining discriminative time-frequency bands and Riemannian tangent space for MI-BCIs

OBJECTIVE

Motor imagery-based brain computer interfaces (MI-BCIs) have been widely researched because they do not demand external stimuli and have a high degree of maneuverability. In most scenarios, superabundant selected channels, fixed time windows, and frequency bands would certainly affect the performance of MI-BCIs due to the neurophysiological diversities between different individuals. In this study, we tended to effectively use the Riemannian geometry of spatial covariance matrix to extract more robust features and thus enhance the decoding efficiency.

APPROACH

First, we propose a Riemannian distance-based EEG channel selection method (RDCS), which preliminarily reduces the information redundancy in the first stage. Second, we extract discriminative Riemannian Tangent Space features of EEG signals of selected channels from the most discriminant time-frequency bands (DTFRTS) to further enhance decoding accuracy for MI-BCIs. Finally, we trained a support vector machine (SVM) model with a linear kernel to classify our extracted discriminative Riemannian features and evaluated our proposed method using publicly available BCI Competition Ⅳ dataset Ⅰ (DS1) and Competition Ⅲ dataset Ⅲa (DS2).

MAIN RESULTS

The experimental results showed that the average classification accuracy with the selected 10-channel EEG signals of our method is 88.1% and 91.6% in DS1 and DS2 respectively. The average improvements are 24.3% & 27.1% on DS1 and 4.4% & 14.2% on DS2 for 10 & 20 selected channels, respectively.

SIGNIFICANCE

These results showed that our proposed method is a promising candidate for performance improvement of MI-BCIs.

Migraine classification by machine learning with functional near-infrared spectroscopy during the mental arithmetic task

Migraine is a common and complex neurovascular disorder. Clinically, the diagnosis of migraine mainly relies on scales, but the degree of pain is too subjective to be a reliable indicator. It is even more difficult to diagnose the medication-overuse headache, which can only be evaluated by whether the symptom is improved after the medication adjustment. Therefore, an objective migraine classification system to assist doctors in making a more accurate diagnosis is needed. In this research, 13 healthy subjects (HC), 9 chronic migraine subjects (CM), and 12 medication-overuse headache subjects (MOH) were measured by functional near-infrared spectroscopy (fNIRS) to observe the change of the hemoglobin in the prefrontal cortex (PFC) during the mental arithmetic task (MAT). Our model shows the sensitivity and specificity of CM are 100% and 75%, and that of MOH is 75% and 100%.The results of the classification of the three groups prove that fNIRS combines with machine learning is feasible for the migraine classification.

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model's fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals.

Motion Artifacts Correction from Single-Channel EEG and fNIRS Signals Using Novel Wavelet Packet Decomposition in Combination with Canonical Correlation Analysis

The electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals, highly non-stationary in nature, greatly suffers from motion artifacts while recorded using wearable sensors. Since successful detection of various neurological and neuromuscular disorders is greatly dependent upon clean EEG and fNIRS signals, it is a matter of utmost importance to remove/reduce motion artifacts from EEG and fNIRS signals using reliable and robust methods. In this regard, this paper proposes two robust methods: (i) Wavelet packet decomposition (WPD) and (ii) WPD in combination with canonical correlation analysis (WPD-CCA), for motion artifact correction from single-channel EEG and fNIRS signals. The efficacy of these proposed techniques is tested using a benchmark dataset and the performance of the proposed methods is measured using two well-established performance matrices: (i) difference in the signal to noise ratio ( ) and (ii) percentage reduction in motion artifacts ( ). The proposed WPD-based single-stage motion artifacts correction technique produces the highest average (29.44 dB) when db2 wavelet packet is incorporated whereas the greatest average (53.48%) is obtained using db1 wavelet packet for all the available 23 EEG recordings. Our proposed two-stage motion artifacts correction technique, i.e., the WPD-CCA method utilizing db1 wavelet packet has shown the best denoising performance producing an average and values of 30.76 dB and 59.51%, respectively, for all the EEG recordings. On the other hand, for the available 16 fNIRS recordings, the two-stage motion artifacts removal technique, i.e., WPD-CCA has produced the best average (16.55 dB, utilizing db1 wavelet packet) and largest average (41.40%, using fk8 wavelet packet). The highest average and using single-stage artifacts removal techniques (WPD) are found as 16.11 dB and 26.40%, respectively, for all the fNIRS signals using fk4 wavelet packet. In both EEG and fNIRS modalities, the percentage reduction in motion artifacts increases by 11.28% and 56.82%, respectively when two-stage WPD-CCA techniques are employed in comparison with the single-stage WPD method. In addition, the average also increases when WPD-CCA techniques are used instead of single-stage WPD for both EEG and fNIRS signals. The increment in both and values is a clear indication that two-stage WPD-CCA performs relatively better compared to single-stage WPD. The results reported using the proposed methods outperform most of the existing state-of-the-art techniques.

Motion Artifacts Correction from Single-Channel EEG and fNIRS Signals Using Novel Wavelet Packet Decomposition in Combination with Canonical Correlation Analysis

The electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals, highly non-stationary in nature, greatly suffers from motion artifacts while recorded using wearable sensors. Since successful detection of various neurological and neuromuscular disorders is greatly dependent upon clean EEG and fNIRS signals, it is a matter of utmost importance to remove/reduce motion artifacts from EEG and fNIRS signals using reliable and robust methods. In this regard, this paper proposes two robust methods: (i) Wavelet packet decomposition (WPD) and (ii) WPD in combination with canonical correlation analysis (WPD-CCA), for motion artifact correction from single-channel EEG and fNIRS signals. The efficacy of these proposed techniques is tested using a benchmark dataset and the performance of the proposed methods is measured using two well-established performance matrices: (i) difference in the signal to noise ratio ( ) and (ii) percentage reduction in motion artifacts ( ). The proposed WPD-based single-stage motion artifacts correction technique produces the highest average (29.44 dB) when db2 wavelet packet is incorporated whereas the greatest average (53.48%) is obtained using db1 wavelet packet for all the available 23 EEG recordings. Our proposed two-stage motion artifacts correction technique, i.e., the WPD-CCA method utilizing db1 wavelet packet has shown the best denoising performance producing an average and values of 30.76 dB and 59.51%, respectively, for all the EEG recordings. On the other hand, for the available 16 fNIRS recordings, the two-stage motion artifacts removal technique, i.e., WPD-CCA has produced the best average (16.55 dB, utilizing db1 wavelet packet) and largest average (41.40%, using fk8 wavelet packet). The highest average and using single-stage artifacts removal techniques (WPD) are found as 16.11 dB and 26.40%, respectively, for all the fNIRS signals using fk4 wavelet packet. In both EEG and fNIRS modalities, the percentage reduction in motion artifacts increases by 11.28% and 56.82%, respectively when two-stage WPD-CCA techniques are employed in comparison with the single-stage WPD method. In addition, the average also increases when WPD-CCA techniques are used instead of single-stage WPD for both EEG and fNIRS signals. The increment in both and values is a clear indication that two-stage WPD-CCA performs relatively better compared to single-stage WPD. The results reported using the proposed methods outperform most of the existing state-of-the-art techniques.

A Safe and Compliant Noncontact Interactive Approach for Wheeled Walking Aid Robot

Aiming at promptly and accurately detecting falls and drag-to gaits induced by asynchronous human-robot movement speed during assisted walking, a noncontact interactive approach with generality, compliance and safety is proposed in this paper, and is applied to a wheeled walking aid robot. Firstly, the structure and the functions of the wheeled walking aid robot, including gait rehabilitation robot (GRR) and walking aid robot (WAR) are illustrated, and the characteristic futures of falls and the drag-to gait are shown by experiments. To obtain gait information, a multichannel proximity sensor array is developed, and a two-dimensional gait information detection system is established by combining four proximity sensors groups which are installed in the robot chassis. Additionally, a node-iterative fuzzy Petri net algorithm for abnormal gait recognition is proposed by generating the network trigger mechanism using the fuzzy membership function. It integrates the walking intention direction vector by taking gait deviation, frequency, and torso angle as input parameters of the system. Finally, to improve the compliance of the robot during human-robot interaction, a PID_SC controller is designed by integrating the gait speed compensation, which enables the WAR to track human gait closely. Abnormal gait recognition and assisted walking experiments are carried out respectively. Experimental results show that the proposed algorithm can accurately identify abnormal gaits of different groups of users with different walking habits, and the recognition rate of abnormal gait reaches 91.2%. Results also show that the developed method can guarantee safety in human robot interaction because of user gate follow-up accuracy and compliant movements. The noncontact interactive approach can be applied to robots with similar structure for usage in walking assistance and gait rehabilitation.

Research on the Identification of Pilots’ Fatigue Status Based on Functional Near-Infrared Spectroscopy

Fatigue can lead to sluggish responses, misjudgments, flight illusions and other problems for pilots, which could easily bring about serious flight accidents. In this paper, a wearable functional near-infrared spectroscopy (fNIRS) device was used to record the changes of hemoglobin concentration of pilots during flight missions. The data was pre-processed, and 1080 valid samples were determined. Then, mean value, variance, standard deviation, kurtosis, skewness, coefficient of variation, peak value, and range of oxyhemoglobin (HbO2) in each channel were extracted. These indexes were regarded as the input of a stacked denoising autoencoder (SDAE) and were used to train the identification model of pilots’ fatigue state. The identification model of pilots’ fatigue status was established. The identification accuracy of the SDAE model was 91.32%, which was 23.26% and 15.97% higher than that of linear discriminant analysis (LDA) models and support vector machines (SVM) models, respectively. Results show that the SDAE model established in our study has high identification accuracy, which can accurately identify different fatigue states of pilots. Identification of pilots’ fatigue status based on fNIRS has important practical significance for reducing flight accidents caused by pilot fatigue.

Detecting Fear of Heights Response to a Virtual Reality Environment Using Functional Near-Infrared Spectroscopy

To enable virtual reality exposure therapy (VRET) that treats anxiety disorders by gradually exposing the patient to fear using virtual reality (VR), it is important to monitor the patient's fear levels during the exposure. Despite the evidence of a fear circuit in the brain as reflected by functional near-infrared spectroscopy (fNIRS), the measurement of fear response in highly immersive VR using fNIRS is limited, especially in combination with a head-mounted display (HMD). In particular, it is unclear to what extent fNIRS can differentiate users with and without anxiety disorders and detect fear response in a highly ecological setting using an HMD. In this study, we investigated fNIRS signals captured from participants with and without a fear of height response. To examine the extent to which fNIRS signals of both groups differ, we conducted an experiment during which participants with moderate fear of heights and participants without it were exposed to VR scenarios involving heights and no heights. The between-group statistical analysis shows that the fNIRS data of the control group and the experimental group are significantly different only in the channel located close to right frontotemporal lobe, where the grand average oxygenated hemoglobin Δ[HbO] contrast signal of the experimental group exceeds that of the control group. The within-group statistical analysis shows significant differences between the grand average Δ[HbO] contrast values during fear responses and those during no-fear responses, where the Δ[HbO] contrast values of the fear responses were significantly higher than those of the no-fear responses in the channels located towards the frontal part of the prefrontal cortex. Also, the channel located close to frontocentral lobe was found to show significant difference for the grand average deoxygenated hemoglobin contrast signals. Support vector machine-based classifier could detect fear responses at an accuracy up to 70% and 74% in subject-dependent and subject-independent classifications, respectively. The results demonstrate that cortical hemodynamic responses of a control group and an experimental group are different to a considerable extent, exhibiting the feasibility and ecological validity of the combination of VR-HMD and fNIRS to elicit and detect fear responses. This research thus paves a way toward the a brain-computer interface to effectively manipulate and control VRET.

Is There a Difference in Brain Functional Connectivity between Chinese Coal Mine Workers Who Have Engaged in Unsafe Behavior and Those Who Have Not?

(1) Background: As a world-recognized high-risk occupation, coal mine workers need various cognitive functions to process the surrounding information to cope with a large number of perceived hazards or risks. Therefore, it is necessary to explore the connection between coal mine workers’ neural activity and unsafe behavior from the perspective of cognitive neuroscience. This study explored the functional brain connectivity of coal mine workers who have engaged in unsafe behaviors (EUB) and those who have not (NUB). (2) Methods: Based on functional near-infrared spectroscopy (fNIRS), a total of 106 workers from the Hongliulin coal mine of Shaanxi North Mining Group, one of the largest modern coal mines in China, completed the test. Pearson’s Correlation Coefficient (COR) analysis, brain network analysis, and two-sample t-test were used to investigate the difference in brain functional connectivity between the two groups. (3) Results: The results showed that there were significant differences in functional brain connectivity between EUB and NUB among the frontopolar area (p = 0.002325), orbitofrontal area (p = 0.02102), and pars triangularis Broca’s area (p = 0.02888). Small-world properties existed in the brain networks of both groups, and the dorsolateral prefrontal cortex had significant differences in clustering coefficient (p = 0.0004), nodal efficiency (p = 0.0384), and nodal local efficiency (p = 0.0004). (4) Conclusions: This study is the first application of fNIRS to the field of coal mine safety. The fNIRS brain functional connectivity analysis is a feasible method to investigate the neuropsychological mechanism of unsafe behavior in coal mine workers in the view of brain science.

Brain–Computer Interfacing Using Functional Near-Infrared Spectroscopy (fNIRS)

Functional Near-Infrared Spectroscopy (fNIRS) is a wearable optical spectroscopy system originally developed for continuous and non-invasive monitoring of brain function by measuring blood oxygen concentration. Recent advancements in brain–computer interfacing allow us to control the neuron function of the brain by combining it with fNIRS to regulate cognitive function. In this review manuscript, we provide information regarding current advancement in fNIRS and how it provides advantages in developing brain–computer interfacing to enable neuron function. We also briefly discuss about how we can use this technology for further applications.

Recognition of Flexion and Extension Imagery Involving the Right and Left Arms Based on Deep Belief Network and Functional Near-Infrared Spectroscopy

Brain-computer interaction based on motor imagery (MI) is an important brain-computer interface (BCI). Most methods for MI classification are based on electroencephalogram (EEG), and few studies have investigated signal processing based on MI-Functional Near-Infrared Spectroscopy (fNIRS). In addition, there is a need to improve the classification accuracy for MI fNIRS methods. In this study, a deep belief network (DBN) based on a restricted Boltzmann machine (RBM) was used to classify fNIRS signals of flexion and extension imagery involving the left and right arms. fNIRS signals from 16 channels covering the motor cortex area were recorded for each of 10 subjects executing or imagining flexion and extension involving the left and right arms. Oxygenated hemoglobin (HbO) concentration was used as a feature to train two RBMs that were subsequently stacked with an additional softmax regression output layer to construct DBN. We also explored the DBN model classification accuracy for the test dataset from one subject using training dataset from other subjects. The average DBN classification accuracy for flexion and extension movement and imagery involving the left and right arms was 84.35 ± 3.86% and 78.19 ± 3.73%, respectively. For a given DBN model, better classification results are obtained for test datasets for a given subject when the model is trained using dataset from the same subject than when the model is trained using datasets from other subjects. The results show that the DBN algorithm can effectively identify flexion and extension imagery involving the right and left arms using fNIRS. This study is expected to serve as a reference for constructing online MI-BCI systems based on DBN and fNIRS.

Decoding different working memory states during an operation span task from prefrontal fNIRS signals

We propose an effective and practical decoding method of different mental states for potential applications for the design of brain-computer interfaces, prediction of cognitive behaviour, and investigation of cognitive mechanism. Functional near infrared spectroscopy (fNIRS) signals that interrogated the prefrontal and parietal cortices and were evaluated by generalized linear model were recorded when nineteen healthy adults performed the operation span (OSPAN) task. The oxygenated hemoglobin changes during OSPAN, response, and rest periods were classified with a support vector machine (SVM). The relevance vector regression algorithm was utilized for prediction of cognitive performance based on multidomain features of fNIRS signals from the OSPAN task. We acquired decent classification accuracies for OSPAN vs. response (above 91.2%) and for OSPAN vs. rest (above 94.7%). Eight of the ten cognitive testing scores could be predicted from the combination of OSPAN and response features, which indicated the brain hemodynamic responses contain meaningful information suitable for predicting cognitive performance.

Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain-Computer Interface

A passive brain-computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects' data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.

Decoding Multiple Sound-Categories in the Auditory Cortex by Neural Networks: An fNIRS Study

This study aims to decode the hemodynamic responses (HRs) evoked by multiple sound-categories using functional near-infrared spectroscopy (fNIRS). The six different sounds were given as stimuli (English, non-English, annoying, nature, music, and gunshot). The oxy-hemoglobin (HbO) concentration changes are measured in both hemispheres of the auditory cortex while 18 healthy subjects listen to 10-s blocks of six sound-categories. Long short-term memory (LSTM) networks were used as a classifier. The classification accuracy was 20.38 ± 4.63% with six class classification. Though LSTM networks' performance was a little higher than chance levels, it is noteworthy that we could classify the data subject-wise without feature selections.

A Methodological Review of fNIRS in Driving Research: Relevance to the Future of Autonomous Vehicles

As automobile manufacturers have begun to design, engineer, and test autonomous driving systems of the future, brain imaging with functional near-infrared spectroscopy (fNIRS) can provide unique insights about cognitive processes associated with evolving levels of autonomy implemented in the automobile. Modern fNIRS devices provide a portable, relatively affordable, and robust form of functional neuroimaging that allows researchers to investigate brain function in real-world environments. The trend toward "naturalistic neuroscience" is evident in the growing number of studies that leverage the methodological flexibility of fNIRS, and in doing so, significantly expand the scope of cognitive function that is accessible to observation via functional brain imaging (i.e., from the simulator to on-road scenarios). While more than a decade's worth of study in this field of fNIRS driving research has led to many interesting findings, the number of studies applying fNIRS during autonomous modes of operation is limited. To support future research that directly addresses this lack in autonomous driving research with fNIRS, we argue that a cogent distillation of the methods used to date will help facilitate and streamline this research of tomorrow. To that end, here we provide a methodological review of the existing fNIRS driving research, with the overarching goal of highlighting the current diversity in methodological approaches. We argue that standardization of these approaches will facilitate greater overlap of methods by researchers from all disciplines, which will, in-turn, allow for meta-analysis of future results. We conclude by providing recommendations for advancing the use of such fNIRS technology in furthering understanding the adoption of safe autonomous vehicle technology.

Performance Improvement of Near-Infrared Spectroscopy-Based Brain-Computer Interfaces Using Transcranial Near-Infrared Photobiomodulation With the Same Device

Transcranial near-infrared photobiomodulation (tNIR-PBM) can modulate physiological characteristics of the human brain, such as the cerebral blood flow and oxidative metabolism. Here, we investigated whether the performance of near-infrared spectroscopy (NIRS)-based brain-computer interfaces (BCIs) can be improved by tNIR-PBM applied to the prefrontal cortex with the same NIRS device. A total of 14 healthy individuals participated in the NIRS-based BCI study where the aim was to distinguish the mental arithmetic task from the idle state (IS) task either after tNIR-PBM or after sham stimulation, with the two experiments being conducted at least two days apart. The tNIR-PBM was applied by simply turning on the NIRS recording equipment for 20 min. To evaluate the degree of performance improvement obtained after tNIR-PBM, the average BCI classification accuracy obtained under the tNIR-PBM condition was compared with that obtained under the sham stimulation condition. The classification accuracy of NIRS-based BCI was significantly improved upon conduction of tNIR-PBM (82.74%) as compared to that in the sham stimulation condition (76.07%, p < 0.005). Thus, our results suggest that simply turning on the NIRS recording equipment before the BCI experiment can improve the performance of the NIRS-based BCI system.

Shedding light on the prefrontal correlates of mental workload in simulated driving: a functional near-infrared spectroscopy study

Optimal mental workload plays a key role in driving performance. Thus, driver-assisting systems that automatically adapt to a drivers current mental workload via brain-computer interfacing might greatly contribute to traffic safety. To design economic brain computer interfaces that do not compromise driver comfort, it is necessary to identify brain areas that are most sensitive to mental workload changes. In this study, we used functional near-infrared spectroscopy and subjective ratings to measure mental workload in two virtual driving environments with distinct demands. We found that demanding city environments induced both higher subjective workload ratings as well as higher bilateral middle frontal gyrus activation than less demanding country environments. A further analysis with higher spatial resolution revealed a center of activation in the right anterior dorsolateral prefrontal cortex. The area is highly involved in spatial working memory processing. Thus, a main component of drivers' mental workload in complex surroundings might stem from the fact that large amounts of spatial information about the course of the road as well as other road users has to constantly be upheld, processed and updated. We propose that the right middle frontal gyrus might be a suitable region for the application of powerful small-area brain computer interfaces.

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010-2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Frontal lobe oxyhemoglobin levels in patients with lower extremity burns assessed using a functional near-Infrared spectroscopy device during usual walking: a pilot study

BACKGROUND

Understanding the mechanisms associated with locomotor networks may be of benefit for rehabilitation of burn victims with neurological locomotor deficits. A wearable functional near-infrared spectroscopy (fNIRS) device has been developed for studying cortical hemodynamics.

OBJECTIVES

To investigate cortical brain activity during usual walking, we examined patterns of cortical activation using fNIRS device (NIRSIT®; OBELAB Inc., Seoul, Korea), in patients with neurological injury caused by lower extremity burns.

METHODS

This cross-sectional study assessed 15 patients with lower extremity burns, 10 patients with upper extremity burns, and 11 healthy controls. We measured walking-related cortical activity using an fNIRS device at baseline and during usual walking.

RESULTS

There was no significant difference between the burns groups in terms of age (43.50 ± 14.08 and 44.67 ± 6.92 years, P = 1.00), pain score of NRS (Numeric rating scale) (5.83 ± 1.19 and 6.67 ± 1.21, P = 0.18) or the mean time since injury (228.50 ± 83.43 and 199.33 ± 68.84 days, P = 0.78). Measures showed increased cortical activation in the prefrontal cortex in patients with lower extremity burns than in patients with healthy controls(P = 0.015). The measured HbO₂ datas of the regions during usual walking in patients with lower extremity burn were insignificantly higher compared with the datas in patient with upper extremity burn (P = 0.302).

CONCLUSIONS

The patients with neurological injury due to lower extremity burns significantly rely more on cognitive resources even when performing a usual walking task.

Sensor Location Optimization of Wireless Wearable fNIRS System for Cognitive Workload Monitoring Using a Data-Driven Approach for Improved Wearability

Functional Near-Infrared Spectroscopy (fNIRS) is a hemodynamic modality in human cognitive workload assessment receiving popularity due to its easier implementation, non-invasiveness, low cost and other benefits from the signal-processing point of view. Wearable wireless fNIRS systems used in research have promisingly shown that fNIRS could be used in cognitive workload assessment in out-of-the-lab scenarios, such as in operators' cognitive workload monitoring. In such a scenario, the wearability of the system is a significant factor affecting user comfort. In this respect, the wearability of the system can be improved if it is possible to minimize an fNIRS system without much compromise of the cognitive workload detection accuracy. In this study, cognitive workload-related hemodynamic changes were acquired using an fNIRS system covering the whole forehead, which is the region of interest in most cognitive workload-monitoring studies. A machine learning approach was applied to explore how the mean accuracy of the cognitive workload classification accuracy varied across various sensing locations on the forehead such as the Left, Mid, Right, Left-Mid, Right-Mid and Whole forehead. The statistical significance analysis result showed that the Mid location could result in significant cognitive workload classification accuracy compared to Whole forehead sensing, with a statistically insignificant difference in the mean accuracy. Thus, the wearable fNIRS system can be improved in terms of wearability by optimizing the sensor location, considering the sensing of the Mid location on the forehead for cognitive workload monitoring.

Enhanced Performance of a Brain Switch by Simultaneous Use of EEG and NIRS Data for Asynchronous Brain-Computer Interface

Previous studies have shown the superior performance of hybrid electroencephalography (EEG)/ near-infrared spectroscopy (NIRS) brain-computer interfaces (BCIs). However, it has been veiled whether the use of a hybrid EEG/NIRS modality can provide better performance for a brain switch that can detect the onset of the intention to turn on a BCI. In this study, we developed such a hybrid EEG/NIRS brain switch and compared its performance with single modality EEG- and NIRS-based brain switch respectively, in terms of true positive rate (TPR), false positive rate (FPR), onset detection time (ODT), and information transfer rate (ITR). In an offline analysis, the performance of a hybrid EEG/NIRS brain switch was significantly improved over that of EEG- and NIRS-based brain switches in general, and in particular a significantly lower FPR was observed for the hybrid EEG/NIRS brain switch. A pseudo-online analysis was additionally performed to confirm the feasibility of implementing an online BCI system with our hybrid EEG/NIRS brain switch. The overall trend of pseudo-online analysis results generally coincided with that of the offline analysis results. No significant difference in all performance measures was also found between offline and pseudo online analysis schemes when the amount of training data was same, with one exception for the ITRs of an EEG brain switch. These offline and pseudo-online results demonstrate that a hybrid EEG/NIRS brain switch can be used to provide a better onset detection performance than that of a single neuroimaging modality.

Random Subspace Ensemble Learning for Functional Near-Infrared Spectroscopy Brain-Computer Interfaces

The feasibility of the random subspace ensemble learning method was explored to improve the performance of functional near-infrared spectroscopy-based brain-computer interfaces (fNIRS-BCIs). Feature vectors have been constructed using the temporal characteristics of concentration changes in fNIRS chromophores such as mean, slope, and variance to implement fNIRS-BCIs systems. The mean and slope, which are the most popular features in fNIRS-BCIs, were adopted. Linear support vector machine and linear discriminant analysis were employed, respectively, as a single strong learner and multiple weak learners. All features in every channel and available time window were employed to train the strong learner, and the feature subsets were selected at random to train multiple weak learners. It was determined that random subspace ensemble learning is beneficial to enhance the performance of fNIRS-BCIs.

Enhanced Accuracy for Multiclass Mental Workload Detection Using Long Short-Term Memory for Brain-Computer Interface

Cognitive workload is one of the widely invoked human factors in the areas of human-machine interaction (HMI) and neuroergonomics. The precise assessment of cognitive and mental workload (MWL) is vital and requires accurate neuroimaging to monitor and evaluate the cognitive states of the brain. In this study, we have decoded four classes of MWL using long short-term memory (LSTM) with 89.31% average accuracy for brain-computer interface (BCI). The brain activity signals are acquired using functional near-infrared spectroscopy (fNIRS) from the prefrontal cortex (PFC) region of the brain. We performed a supervised MWL experimentation with four varying MWL levels on 15 participants (both male and female) and 10 trials of each MWL per participant. Real-time four-level MWL states are assessed using fNIRS system, and initial classification is performed using three strong machine learning (ML) techniques, support vector machine (SVM), k-nearest neighbor (k-NN), and artificial neural network (ANN) with obtained average accuracies of 54.33, 54.31, and 69.36%, respectively. In this study, novel deep learning (DL) frameworks are proposed, which utilizes convolutional neural network (CNN) and LSTM with 87.45 and 89.31% average accuracies, respectively, to solve high-dimensional four-level cognitive states classification problem. Statistical analysis, t-test, and one-way F-test (ANOVA) are also performed on accuracies obtained through ML and DL algorithms. Results show that the proposed DL (LSTM and CNN) algorithms significantly improve classification performance as compared with ML (SVM, ANN, and k-NN) algorithms.

Decoding Mental Workload in Virtual Environments: A fNIRS Study using an Immersive n-back Task

Virtual Reality (VR) has emerged as a novel paradigm for immersive applications in training, entertainment, rehabilitation, and other domains. In this paper, we investigate the automatic classification of mental workload from brain activity measured through functional near-infrared spectroscopy (fNIRS) in VR. We present results from a study which implements the established n-back task in an immersive visual scene, including physical interaction. Our results show that user workload can be detected from fNIRS signals in immersive VR tasks both person-dependently and -adaptively.

Evaluating advanced driver-assistance system trainings using driver performance, attention allocation, and neural efficiency measures

There are about 44 million licensed older drivers in the U.S. Older adults have higher crash rates and fatalities as compared to middle-aged and young drivers, which might be associated with degradations in sensory, cognitive, and physical capabilities. Advanced driver-assistance systems (ADAS) have the potential to substantially improve safety by removing some of driver vehicle control responsibilities. However, a critical aspect of providing ADAS is educating drivers on their operational characteristics and continued use. Twenty older adults participated in a driving simulation study assessing the effectiveness of video-based and demonstration-based training protocols in learning ADAS considering gender differences. The findings revealed video-based training to be more effective than demonstration-based training in improving driver performance and reducing off-road visual attention allocation and mental workload. In addition, female drivers required lower investment of mental effort (higher neural efficiency) to maintain the performance relative to males and they were less distracted by ADAS. However, male drivers were faster in activating ADAS as compared to females since they were monitoring the status of ADAS features more frequently while driving. The findings of this study provided an empirical support for using video-based approach for learning ADAS in older adults to improve driver safety and supported previous findings on older adults' learning that as age increases there is a tendency to prefer more passive and observational learning methods.

Reduction of Onset Delay in Functional Near-Infrared Spectroscopy: Prediction of HbO/HbR Signals

An intrinsic problem when using hemodynamic responses for the brain-machine interface is the slow nature of the physiological process. In this paper, a novel method that estimates the oxyhemoglobin changes caused by neuronal activations is proposed and validated. In monitoring the time responses of blood-oxygen-level-dependent signals with functional near-infrared spectroscopy (fNIRS), the early trajectories of both oxy- and deoxy-hemoglobins in their phase space are scrutinized. Furthermore, to reduce the detection time, a prediction method based upon a kernel-based recursive least squares (KRLS) algorithm is implemented. In validating the proposed approach, the fNIRS signals of finger tapping tasks measured from the left motor cortex are examined. The results show that the KRLS algorithm using the Gaussian kernel yields the best fitting for both ΔHbO (i.e., 87.5%) and ΔHbR (i.e., 85.2%) at q = 15 steps ahead (i.e., 1.63 s ahead at a sampling frequency of 9.19 Hz). This concludes that a neuronal activation can be concluded in about 0.1 s with fNIRS using prediction, which enables an almost real-time practice if combined with EEG.

A Between-Subject fNIRS-BCI Study on Detecting Self-Regulated Intention During Walking

OBJECTIVE

Most BCI (brain-computer interface) studies have focused on detecting motion intention from a resting state. However, the dynamic regulation of two motion states, which usually happens in real life, is rarely studied. Besides, popular within-subject methods also require an extensive and time-consuming learning stage when testing on a new subject. This paper proposed a method to discriminate dynamic gait- adjustment intention with strong adaptability for different subjects.

METHODS

Cerebral hemoglobin signals obtained from 30 subjects were studied to decode gait-adjustment intention. Cerebral hemoglobin information was recorded by using fNIRS (functional near infrared spectroscopy) technology. Mathematical morphology filtering was applied to remove zero drift and EWM (Entropy Weight Method) was used to calculate the average hemoglobin values over Regions of Interest (ROIs). The gradient boosting decision tree (GBDT) was utilized to detect the onset of self-regulated intention. A 2-layer-GA-SVM (Genetic Algorithm-Support Vector Machine) model based on stacking algorithm was further proposed to identify the four types of self-regulated intention (speed increase, speed reduction, step increase, and step reduction).

RESULTS

It was found that GBDT had a good performance to detect the onset intention with an average AUC (Area Under Curve) of 0.894. The 2-layer-GA-SVM model boosted the average ACC (accuracy) of four types of intention from 70.6% to 84.4% ( p = 0.005 ) from the single GA-SVM model. Furthermore, the proposed method passed pseudo-online test with the average results as following: AUC = 0.883, TPR (True Positive Rate) = 97.5%, FPR (False Positive Rate) = 0.11%, and LAY (Detection Latency) = -0.52 ± 2.57 seconds for the recognition of gait-adjustment intention; ACC = 80% for the recognition of adjusted gait.

CONCLUSION

The results indicate that it is feasible to decode dynamic gait-adjustment intentions from a motion state for different subjects based on fNIRS technology. It has a potential to realize the practical application of fNIRS-based brain-computer interface technology in controlling walking-assistive devices.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Evaluation of Neural Degeneration Biomarkers in the Prefrontal Cortex for Early Identification of Patients With Mild Cognitive Impairment: An fNIRS Study

Mild cognitive impairment (MCI), a condition characterizing poor cognition, is associated with aging and depicts early symptoms of severe cognitive impairment, known as Alzheimer's disease (AD). Meanwhile, early detection of MCI can prevent progression to AD. A great deal of research has been performed in the past decade on MCI detection. However, availability of biomarkers for MCI detection requires greater attention. In our study, we evaluated putative and reliable biomarkers for diagnosing MCI by performing different mental tasks (i.e., N-back task, Stroop task, and verbal fluency task) using functional near-infrared spectroscopy (fNIRS) signals on a group of 15 MCI patients and 9 healthy control (HC). The 15 digital biomarkers (i.e., five means, seven slopes, peak, skewness, and kurtosis) and two image biomarkers (t-map, correlation map) in the prefrontal cortex (PFC) (i.e., left PFC, middle PFC, and right PFC) between the MCI and HC groups were investigated by the statistical analysis, linear discriminant analysis (LDA), and convolutional neural network (CNN) individually. The results reveal that the statistical analysis using digital biomarkers (with a p-value < 0.05) could not distinguish the MCI patients from the HC over 60% accuracy. Therefore, the current statistical analysis needs to be improved to be used for diagnosing the MCI patients. The best accuracy with LDA was 76.67% with the N-back and Stroop tasks. However, the CNN classification results trained by image biomarkers showed a high accuracy. In particular, the CNN results trained via t-maps revealed the best accuracy (90.62%) with the N-back task, whereas the CNN result trained by the correlation maps was 85.58% with the N-back task. Also, the results illustrated that investigating the sub-regions (i.e., right, middle, left) of the PFC for detecting MCI would be better than examining the whole PFC. The t-map (or/and the correlation map) is conclusively recommended as an image biomarker for early detection of AD. The combination of CNN and image biomarkers can provide a reliable clinical tool for diagnosing MCI patients.

Exploring the Brain Responses to Driving Fatigue Through Simultaneous EEG and fNIRS Measurements

Fatigue is one problem with driving as it can lead to difficulties with sustaining attention, behavioral lapses, and a tendency to ignore vital information or operations. In this research, we explore multimodal physiological phenomena in response to driving fatigue through simultaneous functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) recordings with the aim of investigating the relationships between hemodynamic and electrical features and driving performance. Sixteen subjects participated in an event-related lane-deviation driving task while measuring their brain dynamics through fNIRS and EEGs. Three performance groups, classified as Optimal, Suboptimal, and Poor, were defined for comparison. From our analysis, we find that tonic variations occur before a deviation, and phasic variations occur afterward. The tonic results show an increased concentration of oxygenated hemoglobin (HbO₂) and power changes in the EEG theta, alpha, and beta bands. Both dynamics are significantly correlated with deteriorated driving performance. The phasic EEG results demonstrate event-related desynchronization associated with the onset of steering vehicle in all power bands. The concentration of phasic HbO₂ decreased as performance worsened. Further, the negative correlations between tonic EEG delta and alpha power and HbO₂ oscillations suggest that activations in HbO₂ are related to mental fatigue. In summary, combined hemodynamic and electrodynamic activities can provide complete knowledge of the brain's responses as evidence of state changes during fatigue driving.

Sleep Is a Refreshing Process: An fNIRS Study

Sleep is a very critical process that constitutes up to one third of daytime of a healthy adult. It is known to be an active period where body and brain is refreshed for the next day. It is both part of a larger cycle, i.e., circadian rhythm, and has subcsycles in it, i.e., sleep stages. Although hemodynamics of these stages have been investigated especially in the last two decades, there are still points in the hemodynamics to be illuminated especially in terms of refreshment. This study aims to investigate refreshing property of sleep in terms of sleep stages using functional near-infrared spectroscopy (fNIRS) for measuring prefrontal cortex (PFC) hemodynamics. Nine healthy subjects slept in sleep laboratories, monitored by polysomnography and fNIRS before, during, and after night sleep. REM stage had lower oxyhemoglobin (HbO) and total hemoglobin (HbT) than the other sleep stages and wakefulness. Deoxyhemoglobin (HbR) did not differ between any stages. All sleep stages and wakefulness stage at the end of the sleep had higher HbO and lower HbR than the beginning of the sleep. HbT levels did not differ between the beginning and the end of the sleep for any stages. During REM sleep, PFC seems to get lower blood supply, possibly due to increased demand in other brain regions. Regardless of the stage, PFC has higher oxygenation toward the end of sleep, indicating refreshment. Overall, our brain seems to be on duty during sleep throughout the night for "cleaning" and "refreshing" itself. Hemodynamic changes from the beginning to end of sleep might be the indicator of this work. Thus, accordingly REM stage seems to be at a central point for this work.

Using Polygraph to Detect Passengers Carrying Illegal Items

The present study examined the effectiveness of a Modified-Comparison Questions Technique, used in conjunction with the polygraph, to differentiate between common travelers, drug traffickers, and terrorists at transportation hubs. Two experiments were conducted using a mock crime paradigm. In Experiment 1, we randomly assigned 78 participants to either a drug condition, where they packed and lied about illicit drugs in their luggage, or a control condition, where they did not pack or lie about any illegal items. In Experiment 2, we randomly assigned 164 participants to one of the two conditions in Experiment 1 or an additional bomb condition, where they packed and lied about a bomb in their luggage. For both experiments, we assessed participants’ RR interval, heart rate, peak-to-peak amplitude of Galvanic Skin Response (GSR) and all three combined, using Discriminant Analyses to determine the classification accuracy of participants in each condition. In both experiments, we found decelerated heart rates and increased peak-to-peak amplitude of GSR in guilty participants when lying in response to questions regarding their crime. We also found accurate classifications of participants, in both Experiment 1 (drug vs. control: 84.2% vs. 82.5%) and Experiment 2 (drug vs. control: 82:1% vs. 95.1%; bomb vs. control: 93.2% vs. 95.1%; drug vs. bomb: 92.3% vs. 90.9%), above chance level. These findings indicate that Modified-CQT, combined with a polygraph test, is a viable method for investigating suspects of drug trafficking and terrorism at transportation hubs such as train stations and airports.