Brain Dynamics in Predicting Driving Fatigue Using a Recurrent Self-Evolving Fuzzy Neural Network

Self-Organizing Robust Fuzzy Neural Network for Nonlinear System Modeling

Fuzzy neural network (FNN) is a structured learning technique that has been successfully adopted in nonlinear system modeling. However, since there exist uncertain external disturbances arising from mismatched model errors, sensor noises, or unknown environments, FNN generally fails to achieve the desirable performance of modeling results. To overcome this problem, a self-organization robust FNN (SOR-FNN) is developed in this article. First, an information integration mechanism (IIM), consisting of partition information and individual information, is introduced to dynamically adjust the structure of SOR-FNN. The proposed mechanism can make itself adapt to uncertain environments. Second, a dynamic learning algorithm based on the -divergence loss function ( -DLA) is designed to update the parameters of SOR-FNN. Then, this learning algorithm is able to reduce the sensibility of disturbances and improve the robustness of Third, the convergence of SOR-FNN is given by the Lyapunov theorem. Then, the theoretical analysis can ensure the successful application of SOR-FNN. Finally, the proposed SOR-FNN is tested on several benchmark datasets and a practical application to validate its merits. The experimental results indicate that the proposed SOR-FNN can obtain superior performance in terms of model accuracy and robustness.

Technologies for detecting and monitoring drivers' states: A systematic review

Driver fatigue or drowsiness detection techniques can significantly enhance road safety measures and reduce traffic accidents. These approaches used different sensor technologies to acquire the human physiological and behavioral characteristics to investigate the driver's vigilance state. Although the driver's vigilance detection technique has attracted significant interest recently, few studies have been conducted to review it systematically. These studies provide a thorough overview of the most advanced driver vigilance detection method available today in terms of sensor technology for scholars and specialists. This research is geared towards achieving three main objectives. Firstly, it aims to systematically gather, evaluate, and synthesize information from previous research published between 2014 and May 2024 on driver's state and driving sensors and their implementation on detection algorithms. It aims to provide a thorough review of the present state of research on wearable and unwearable sensor technology for driver fatigue detection, focusing on reporting experimental results in this field. This information will be necessary for experts and scientists seeking to advance their knowledge in this field. Lastly, the research aims to identify gaps in knowledge that require further investigation and recommend future research directions to help address these gaps. This way, it will contribute to the advancement of the field and provide beneficial insights for future researchers.

Systematic review of experimental paradigms and deep neural networks for electroencephalography-based cognitive workload detection

This article summarizes a systematic literature review of deep neural network-based cognitive workload (CWL) estimation from electroencephalographic (EEG) signals. The focus of this article can be delineated into two main elements: first is the identification of experimental paradigms prevalently employed for CWL induction, and second, is an inquiry about the data structure and input formulations commonly utilized in deep neural networks (DNN)-based CWL detection. The survey revealed several experimental paradigms that can reliably induce either graded levels of CWL or a desired cognitive state due to sustained induction of CWL. This article has characterized them with respect to the number of distinct CWL levels, cognitive states, experimental environment, and agents in focus. Further, this literature analysis found that DNNs can successfully detect distinct levels of CWL despite the inter-subject and inter-session variability typically observed in EEG signals. Several methodologies were found using EEG signals in its native representation of a two-dimensional matrix as input to the classification algorithm, bypassing traditional feature selection steps. More often than not, researchers used DNNs as black-box type models, and only a few studies employed interpretable or explainable DNNs for CWL detection. However, these algorithms were mostly post hoc data analysis and classification schemes, and only a few studies adopted real-time CWL estimation methodologies. Further, it has been suggested that using interpretable deep learning methodologies may shed light on EEG correlates of CWL, but this remains mostly an unexplored area. This systematic review suggests using networks sensitive to temporal dependencies and appropriate input formulations for each type of DNN architecture to achieve robust classification performance. An additional suggestion is to utilize transfer learning methods to achieve high generalizability across tasks (task-independent classifiers), while simple cross-subject data pooling may achieve the same for subject-independent classifiers.

Extended reality in the automotive sector: A bibliometric analysis of publications from 2012 to 2022

The present study aims to present a bibliometric analysis of publications related to "Extended Reality" (XR) in the automotive sector. XR is revolutionizing the industry in all fields, and the automotive is one of the sectors that has had much to gain from this technology and its components (Virtual Reality, Augmented Reality, Mixed Reality). Articles on XR in the automotive field that were published from 2012 to 2022 were retrieved from the Scopus database. Extracted items were analysed in terms of the document type, document language, year of publication, country, authors, affiliations, sources, citations, keywords, and research domains. The open-source tool VOSviewer was used to visualize trends in research on XR applied to automotive. The analyses of 1584 documents revealed that the total number of publications has continually increased over the last 11 years. The country producing most of the articles in this field was Germany, followed by the United States and China. The most productive journal is Transportation Research Part F: Traffic Psychology and Behaviour and the institution that issued most of the articles is Technical University of Munich. From the analysis of author keywords, the prominent research areas currently involving the use of XR technologies in automotive can be highlighted: virtual prototyping, design, manufacturing, sales, training, driver or pedestrian behaviour analysis, and ergonomics. More recently, terms like artificial intelligence and autonomous vehicles have started to be used more frequently in studies in the field. The current study reveals an expanding corpus of literature on XR-based applications for the automotive sector using bibliometric methods. Researchers and stakeholders can use this study as a useful reference to comprehend the big picture and the state-of-the-art in this area.

Bounded Antisynchronization of Multiple Neural Networks via Multilevel Hybrid Control

The bounded antisynchronization (AS) problem of multiple discrete-time neural networks (NNs) based on the fuzzy model is studied, in consideration of the differences in quantity and communication among different NN groups, the variabilities of dynamics, and communication topological affected by environments. To reduce the energy consumption of communication, a cluster pinning communication mechanism is proposed, and an impulsive observer is designed to estimate the state of target NN. Then, a multilevel hybrid controller based on the impulsive observer is built including the AS controller and the bounded synchronization (BS) controller. Sufficient conditions for bounded AS are obtained by analyzing the stability of the BS augmented error (BSAE) and the AS augmented error (ASAE) based on the fuzzy-based Lyapunov functional (FBLF). Finally, a numerical example and an application example are given to verify the validity of the obtained results.

A regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based cross-subject fatigue detection

Fatigue, one of the most important factors affecting road safety, has attracted many researchers' attention. Most existing fatigue detection methods are based on feature engineering and classification models. The feature engineering is greatly influenced by researchers' domain knowledge, which will lead to a poor performance in fatigue detection, especially in cross-subject experiment design. In addition, fatigue detection is often simplified as a classification problem of several discrete states. Models based on deep learning can realize automatic feature extraction without the limitation of researcher's domain knowledge. Therefore, this paper proposes a regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based (EEG-based) cross-subject fatigue detection. At the same time, a twofold random-offset zero-overlapping sampling method is proposed to train a bigger model and reduce overfitting. Compared with existing results, the proposed method achieves a much better result of 0.94 correlation coefficient (COR) and 0.09 root mean square error (RMSE) in a within-subject experiment design. What is more, there is no misclassification between awake and drowsy states. For cross-subject experiment design, the COR and RMSE are 0.79 and 0.15, respectively, which are close to the existing within-subject results and better than similar cross-subject results. The cross-subject regression model is very important for fatigue detection application since the fatigue indication is more precise than several discrete states and no model calibration is required for a new user. The twofold random-offset zero-overlapping sampling method can also be used as a reference by other EEG-based deep learning research.

Neuromorphic processor-oriented hybrid Q-format multiplication with adaptive quantization for tiny YOLO3

Deep neural networks (DNNs) have delivered unprecedented achievements in the modern Internet of Everything society, encompassing autonomous driving, expert diagnosis, unmanned supermarkets, etc. It continues to be challenging for researchers and engineers to develop a high-performance neuromorphic processor for deployment in edge devices or embedded hardware. DNNs’ superpower derives from their enormous and complex network architecture, which is computation-intensive, time-consuming, and energy-heavy. Due to the limited perceptual capacity of humans, accurate processing results from DNNs require a substantial amount of computing time, making them redundant in some applications. Utilizing adaptive quantization technology to compress the DNN model with sufficient accuracy is crucial for facilitating the deployment of neuromorphic processors in emerging edge applications. This study proposes a method to boost the development of neuromorphic processors by conducting fixed-point multiplication in a hybrid Q-format using an adaptive quantization technique on the convolution of tiny YOLO3. In particular, this work integrates the sign-bit check and bit roundoff techniques into the arithmetic of fixed-point multiplications to address overflow and roundoff issues within the convolution’s adding and multiplying operations. In addition, a hybrid Q-format multiplication module is developed to assess the proposed method from a hardware perspective. The experimental results prove that the hybrid multiplication with adaptive quantization on the tiny YOLO3’s weights and feature maps possesses a lower error rate than alternative fixed-point representation formats while sustaining the same object detection accuracy. Moreover, the fixed-point numbers represented by Q(6.9) have a suboptimal error rate, which can be utilized as an alternative representation form for the tiny YOLO3 algorithm-based neuromorphic processor design. In addition, the 8-bit hybrid Q-format multiplication module exhibits low power consumption and low latency in contrast to benchmark multipliers.

Predicting collision cases at unsignalized intersections using EEG metrics and driving simulator platform

Unsignalized intersection collision has been one of the most dangerous accidents in the world. How to identify road hazards and predict the potential intersection collision ahead are challenging problems in traffic safety. This paper studies the feasibility of EEG metrics to forecast road hazards and presents an improved neural network model to predict intersection collision based on EEG metrics and driving behavior. It is demonstrated that EEG metrics show significant differences between collision and non-collision cases. It indicates that EEG metrics can serve as effective indicators to predict the collision probability. The drivers with higher relative power in fast frequency band (alpha and beta), lower relative power in slow frequency band (delta and theta) are more likely to have conflicts. The prediction using three machine learning models (Multi-layer perceptron (MLP), Logistic regression (LR) and Random forest (RF)) based on three input datasets (only EEG metrics, only driving behavior and combined EEG metrics with driving behavior) are compared. The results show that for single time point prediction, MLP model has the highest accuracy among three machine learning models. The model solely based on EEG metrics datasets has higher accuracy than driving behavior as well as combined datasets. However, for multi-time point prediction, the accuracy of MLP is only 73.9%, worse than LR and RF. We improved the MLP model by adding attention mechanism layer and using random forest model to select important features. As a consequence, the accuracy is greatly improved and reaches 88%. This study demonstrates the importance and feasibility of EEG signals to identify unsafe drivers ahead. The improved neural network model can be helpful to reduce intersection accidents and improve traffic safety.

Autonomous learning for fuzzy systems: a review

As one of the three pillars in computational intelligence, fuzzy systems are a powerful mathematical tool widely used for modelling nonlinear problems with uncertainties. Fuzzy systems take the form of linguistic IF-THEN fuzzy rules that are easy to understand for human. In this sense, fuzzy inference mechanisms have been developed to mimic human reasoning and decision-making. From a data analytic perspective, fuzzy systems provide an effective solution to build precise predictive models from imprecise data with great transparency and interpretability, thus facilitating a wide range of real-world applications. This paper presents a systematic review of modern methods for autonomously learning fuzzy systems from data, with an emphasis on the structure and parameter learning schemes of mainstream evolving, evolutionary, reinforcement learning-based fuzzy systems. The main purpose of this paper is to introduce the underlying concepts, underpinning methodologies, as well as outstanding performances of the state-of-the-art methods. It serves as a one-stop guide for readers learning the representative methodologies and foundations of fuzzy systems or who desire to apply fuzzy-based autonomous learning in other scientific disciplines and applied fields.

A comparative analysis of masking empirical mode decomposition and a neural network with feed-forward and back propagation along with masking empirical mode decomposition to improve the classification performance for a reliable brain-computer interface

In general, extraction and classification are used in various fields like image processing, pattern recognition, signal processing, and so on. Extracting effective characteristics from raw electroencephalogram (EEG) signals is a crucial role of the brain-computer interface for motor imagery. Recently, there has been a great deal of focus on motor imagery in the EEG signals since they encode a person's intent to do an action. Researchers have been using MI signals to assist paralyzed people and even move them on their own with certain equipment, like wheelchairs. As a result, proper decoding is an important step required for the interconnection of the brain and the computer. EEG decoding is a challenging process because of poor SNR, complexity, and other reasons. However, choosing an appropriate method to extract the features to improve the performance of motor imagery recognition is still a research hotspot. To extract the features of the EEG signal in the classification task, this paper proposes a Masking Empirical Mode Decomposition (MEMD) based Feed Forward Back Propagation Neural Network (MEMD-FFBPNN). The dataset consists of EEG signals which are first normalized using the minimax method and given as input to the MEMD to extract the features and then given to the FFBPNN to classify the tasks. The accuracy of the proposed method MEMD-FFBPNN has been measured using the confusion matrix, mean square error and which has been recorded up to 99.9%. Thus, the proposed method gives better accuracy than the other conventional methods.

Fatigue Detection of Pilots' Brain Through Brains Cognitive Map and Multilayer Latent Incremental Learning Model

This work proposes a nonparametric prior induced deep sum-logarithmic-multinomial mixture (DSLMM) model to detect pilots' cognitive states through the developed brain power map. DSLMM uses multinormal distribution to infer the latent variable of each neuron in the first layer of the network. These latent variables obeyed a sum-logarithmic distribution that is backpropagated to its observation vector and the number of neurons in the next layer. Multinormal distribution is used to segment the extended observation vector to form a matrix associated with the width of the next layer. This work also proposes an adaptive topic-layer stochastic gradient Riemann (ATL-SGR) Markov chain Monte Carlo (MCMC) inference method to learn its global parameters without heuristic assumptions. The experimental results indicate that DSLMM can extract more probability distribution contained in the brain power map layer by layer, and achieve higher pilot cognition detection accuracy.

Real-time regulation of room temperature based on individual thermal sensation using an online brain-computer interface

Regulation of indoor temperature based on neurophysiological and psychological signals is one of the most promising technologies for intelligent buildings. In this study, we developed a system for closed-loop control of indoor temperature based on brain-computer interface (BCI) technology for the first time. Electroencephalogram (EEG) signals were collected from subjects for two room temperature categories (cool comfortable and hot uncomfortable) and used to build a thermal-sensation discrimination model (TSDM) with an ensemble learning method. Then, an online BCI system was developed based on the TSDM. In the online room temperature control experiment, when the TSDM detected that the subjects felt hot and uncomfortable, BCI would automatically turn on the air conditioner, and when the TSDM detected that the subjects felt cool and comfortable, BCI would automatically turn off the air conditioner. The results of online experiments in a hot environment showed that a BCI could significantly improve the thermal comfort of subjects (the subjective thermal comfort score decreased from 2.45 (hot uncomfortable) to 0.55 (cool comfortable), p < 0.001). A parallel experiment further showed that if the subjects wore thicker clothes during the experiment, the BCI would turn on the air conditioner for a longer time to ensure the thermal comfort of the subjects. This has further confirmed the effectiveness of TSDM model in evaluating thermal sensation under the dynamic change of room temperature and showed the model's good robustness. This study proposed a new paradigm of human-building interaction, which is expected to play a promising role in the development of human-centered intelligent buildings.

Past, Present, and Future of EEG-Based BCI Applications

An electroencephalography (EEG)-based brain-computer interface (BCI) is a system that provides a pathway between the brain and external devices by interpreting EEG. EEG-based BCI applications have initially been developed for medical purposes, with the aim of facilitating the return of patients to normal life. In addition to the initial aim, EEG-based BCI applications have also gained increasing significance in the non-medical domain, improving the life of healthy people, for instance, by making it more efficient, collaborative and helping develop themselves. The objective of this review is to give a systematic overview of the literature on EEG-based BCI applications from the period of 2009 until 2019. The systematic literature review has been prepared based on three databases PubMed, Web of Science and Scopus. This review was conducted following the PRISMA model. In this review, 202 publications were selected based on specific eligibility criteria. The distribution of the research between the medical and non-medical domain has been analyzed and further categorized into fields of research within the reviewed domains. In this review, the equipment used for gathering EEG data and signal processing methods have also been reviewed. Additionally, current challenges in the field and possibilities for the future have been analyzed.

An EEG-based systematic explainable detection framework for probing and localizing abnormal patterns in Alzheimer's disease

OBJECTIVE

Electroencephalography (EEG) is a potential source of downstream biomarkers for the early diagnosis of Alzheimer's disease (AD) due to its low-cost, non-invasive, and portable advantages. Accurately detecting AD-induced patterns from EEG signals is essential for understanding AD-related neurodegeneration at the EEG level and further evaluating the risk of AD at an early stage. This paper proposes a deep learning-based, functional explanatory framework that probes AD abnormalities from short-sequence EEG data.

APPROACH

The framework is a learning-based automatic detection system consisting of three encoding pathways that analyze EEG signals in frequency, complexity, and synchronous domains. We integrated the proposed EEG descriptors with the neural network components into one learning system to detect AD patterns. A transfer learning-based model was used to learn the deep representations, and a modified generative adversarial module was attached to the model to overcome feature sparsity. Furthermore, we utilized activation mapping to obtain the AD-related neurodegeneration at brain rhythm, dynamic complexity, and functional connectivity levels.

MAIN RESULTS

The proposed framework can accurately (100%) detect AD patterns based on our raw EEG recordings without delicate preprocessing. Meanwhile, the system indicates that 1) the power of different brain rhythms exhibits abnormal in the frontal lobes of AD patients, and such abnormality spreads to central lobes in the alpha and beta rhythms, 2) the difference in nonlinear complexity varies with the temporal scales, and 3) all the connections of pair-wise brain regions except bilateral temporal connectivity are weak in AD patterns. The proposed method outperforms other related methods in detection performance.

SIGNIFICANCE

We provide a new method for revealing abnormalities and corresponding localizations in different feature domains of EEG from AD patients. This study is a significant foundation for our future work on identifying individuals at high risk of AD at an early stage.

A Design of CGK-Based Granular Model Using Hierarchical Structure

In this paper, we propose context-based GK clustering and design a CGK-based granular model and a hierarchical CGK-based granular model. Existing fuzzy clustering generates clusters using Euclidean distances. However, there is a problem in that performance decreases when a cluster is created from data with strong nonlinearity. To improve this problem, GK clustering is used. GK clustering creates clusters using Mahalanobis distance. In this paper, we propose context-based GK (CGK) clustering, which adds a method that considers the output space in the existing GK clustering, to create a cluster that considers not only the input space but also the output space. there is. Based on the proposed CGK clustering, a CGK-based granular model and a hierarchical CGK-based granular model were designed. Since the output of the CGK-based granular model is in the form of a context, it has the advantage of verbally expressing the prediction result, and the CGK-based granular model with a hierarchical structure can generate high-dimensional information granules, so meaningful information with high abstraction value granules can be created. In order to verify the validity of the method proposed in this paper, as a result of conducting an experiment using the concrete compressive strength database, it was confirmed that the proposed methods showed superior performance than the existing granular models.

Recent Advances and Current Trends in Brain-Computer Interface (BCI) Research and Their Applications

Brain-Computer Interface (BCI) provides direct communication between the brain and an external device. BCI systems have become a trendy field of research in recent years. These systems can be used in a variety of applications to help both disabled and healthy people. Concerning significant BCI progress, we may assume that these systems are not very far from real-world applications. This review has taken into account current trends in BCI research. In this survey, one hundred most cited articles from the WOS database were selected over the last four years. This survey is divided into several sectors. These sectors are Medicine, Communication and Control, Entertainment, and Other BCI applications. The application area, recording method, signal acquisition types, and countries of origin have been identified in each article. This survey provides an overview of the BCI articles published from 2016 to 2020 and their current trends and advances in different application areas.

A gradient-based automatic optimization CNN framework for EEG state recognition

The Electroencephalogram (EEG) signal, as a data carrier that can contain a large amount of information about the human brain in different states, is one of the most widely used metrics for assessing human psychophysiological states. Among a variety of analysis methods, deep learning, especially convolutional neural network (CNN), has achieved remarkable results in recent years as a method to effectively extract features from EEG signals. Although deep learning has the advantages of automatic feature extraction and effective classification, it also faces difficulties in network structure design and requires an army of prior knowledge. Automating the design of these hyperparameters can therefore save experts' time and manpower. Neural architecture search techniques have thus emerged. In this paper, based on an existing gradient-based NAS algorithm, PC-DARTS, with targeted improvements and optimizations for the characteristics of EEG signals. Specifically, we establish the model architecture step by step based on the manually designed deep learning models for EEG discrimination by retaining the framework of the search algorithm and performing targeted optimization of the model search space. Corresponding features are extracted separately according to the frequency domain, time domain characteristics of the EEG signal and the spatial position of the EEG electrode. The architecture was applied to EEG-based emotion recognition and driver drowsiness assessment tasks. The results illustrate that compared with the existing methods, the model architecture obtained in this paper can achieve competitive overall accuracy and better standard deviation in both tasks. Therefore, this approach is an effective migration of NAS technology into the field of EEG analysis and has great potential to provide high-performance results for other types of classification and prediction tasks. This can effectively reduce the time cost for researchers and facilitate the application of CNN in more areas.

Analysis of Relationship between Electroencephalograms and Subjective Measurements for In-Vehicle Information System: A Preliminary Study

In this study, we explored the relationship between objective and subjective measures for usability evaluation in in-vehicle infotainment systems (IVISs). As a case study, four displays were evaluated based on cluster location and display orientation (that is, front–horizontal, front–vertical, right–horizontal, and right–vertical). Thirty-six participants performed tasks to manipulate the functions of the IVISs and data were collected through an electroencephalogram (EEG) sensor and questionnaire items. We analysed a model that estimated EEG-based objective indicators from subjective indicators. As a result, the objective indicators reflected the subjective indicators and were considered to explain the driver’s cognitive state. Although EEG data were collected from only four participants, this study proposed an experimental design that could be applied to the analysis of the relationship between the subject’s evaluation and EEG signals, as a preliminary study. We expect the experimental design and results of this study to be useful in analysing objective and subjective measures of usability evaluation.

Exploring Covert States of Brain Dynamics via Fuzzy Inference Encoding

Human brain inherently exhibits latent mental processes which are likely to change rapidly over time. A framework that adopts a fuzzy inference system is proposed to model the dynamics of the human brain. The fuzzy inference system is used to encode real-world data to represent the salient features of the EEG signals. Then, an unsupervised clustering is conducted on the extracted feature space to identify the brain (external and covert) states that respond to different cognitive demands. To understand the human state change, a state transition diagram is introduced, allowing visualization of connectivity patterns between every pair of states. We compute the transition probability between every pair of states to represent the relationships between the states. This state transition diagram is named as the Fuzzy Covert State Transition Diagram (FCOSTD), which helps the understanding of human states and human performance. We then apply FCOSTD on distracted driving experiments. FCOSTD successfully discovers the external and covert states, faithfully reveals the transition of the brain between states, and the route of the state change when humans are distracted during a driving task. The experimental results demonstrate that different subjects have similar states and inter-state transition behaviour (establishing the consistency of the system) but different ways to allocate brain resources as different actions are being taken.

EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications

Brain-Computer interfaces (BCIs) enhance the capability of human brain activities to interact with the environment. Recent advancements in technology and machine learning algorithms have increased interest in electroencephalographic (EEG)-based BCI applications. EEG-based intelligent BCI systems can facilitate continuous monitoring of fluctuations in human cognitive states under monotonous tasks, which is both beneficial for people in need of healthcare support and general researchers in different domain areas. In this review, we survey the recent literature on EEG signal sensing technologies and computational intelligence approaches in BCI applications, compensating for the gaps in the systematic summary of the past five years. Specifically, we first review the current status of BCI and signal sensing technologies for collecting reliable EEG signals. Then, we demonstrate state-of-the-art computational intelligence techniques, including fuzzy models and transfer learning in machine learning and deep learning algorithms, to detect, monitor, and maintain human cognitive states and task performance in prevalent applications. Finally, we present a couple of innovative BCI-inspired healthcare applications and discuss future research directions in EEG-based BCI research.

Noninvasive Electroencephalography Equipment for Assistive, Adaptive, and Rehabilitative Brain-Computer Interfaces: A Systematic Literature Review

Humans interact with computers through various devices. Such interactions may not require any physical movement, thus aiding people with severe motor disabilities in communicating with external devices. The brain-computer interface (BCI) has turned into a field involving new elements for assistive and rehabilitative technologies. This systematic literature review (SLR) aims to help BCI investigator and investors to decide which devices to select or which studies to support based on the current market examination. This examination of noninvasive EEG devices is based on published BCI studies in different research areas. In this SLR, the research area of noninvasive BCIs using electroencephalography (EEG) was analyzed by examining the types of equipment used for assistive, adaptive, and rehabilitative BCIs. For this SLR, candidate studies were selected from the IEEE digital library, PubMed, Scopus, and ScienceDirect. The inclusion criteria (IC) were limited to studies focusing on applications and devices of the BCI technology. The data used herein were selected using IC and exclusion criteria to ensure quality assessment. The selected articles were divided into four main research areas: education, engineering, entertainment, and medicine. Overall, 238 papers were selected based on IC. Moreover, 28 companies were identified that developed wired and wireless equipment as means of BCI assistive technology. The findings of this review indicate that the implications of using BCIs for assistive, adaptive, and rehabilitative technologies are encouraging for people with severe motor disabilities and healthy people. With an increasing number of healthy people using BCIs, other research areas, such as the motivation of players when participating in games or the security of soldiers when observing certain areas, can be studied and collaborated using the BCI technology. However, such BCI systems must be simple (wearable), convenient (sensor fabrics and self-adjusting abilities), and inexpensive.

Cross-Subject Zero Calibration Driver's Drowsiness Detection: Exploring Spatiotemporal Image Encoding of EEG Signals for Convolutional Neural Network Classification

This paper explores two methodologies for drowsiness detection using EEG signals in a sustained-attention driving task considering pre-event time windows, and focusing on cross-subject zero calibration. Driving accidents are a major cause of injuries and deaths on the road. A considerable portion of those are due to fatigue and drowsiness. Advanced driver assistance systems that could detect mental states which are associated with hazardous situations, such as drowsiness, are of critical importance. EEG signals are used widely for brain-computer interfaces, as well as mental state recognition. However, these systems are still difficult to design due to very low signal-to-noise ratios and cross-subject disparities, requiring individual calibration cycles. To tackle this research domain, here, we explore drowsiness detection based on EEG signals' spatiotemporal image encoding representations in the form of either recurrence plots or gramian angular fields for deep convolutional neural network (CNN) classification. Results comparing both techniques using a public dataset of 27 subjects show a superior balanced accuracy of up to 75.87% for leave-one-out cross-validation, using both techniques, against works in the literature, demonstrating the possibility to pursue cross-subject zero calibration design.

A CNN identified by reinforcement learning-based optimization framework for EEG-based state evaluation

Objective.Electroencephalogram (EEG) data, as a kind of complex time-series, is one of the most widely-used information measurements for evaluating human psychophysiological states. Recently, numerous works applied deep learning techniques, especially the convolutional neural network (CNN), into EEG-based research. The design of the hyper-parameters of the CNN model has a great influence on the performance of the model. Therefore, automatically designing these hyper-parameters can save the time and labor of experts. This leads to the appearance of the neural architecture search technique. In this paper, we propose a reinforcement learning (RL)-based step-by-step framework to efficiently search for CNN models.Approach.Specifically, the deep Q network in RL is first used to determine the depth of convolutional layers and the connection modes among layers. Then particle swarm optimization algorithm is used to fine-tune the number and size of convolution kernels. Through this step-by-step strategy, the search space can be narrowed in each step for saving the overall time cost. This framework is employed for both EEG-based sleep stage classification and driver drowsiness evaluation tasks.Main results.The results show that compared with state-of-the-art methods, the high-performance CNN models identified by the proposed optimization framework, can achieve high overall accuracy and better root mean squared error in the two tasks.Significance.Therefore, the proposed optimization framework has a great potential to provide high-performance results for other kinds of classification and prediction tasks. In this way, it can greatly save researchers' time cost and promote broader applications of CNNs.

Online Mental Fatigue Monitoring via Indirect Brain Dynamics Evaluation

Driver mental fatigue leads to thousands of traffic accidents. The increasing quality and availability of low-cost electroencephalogram (EEG) systems offer possibilities for practical fatigue monitoring. However, non-data-driven methods, designed for practical, complex situations, usually rely on handcrafted data statistics of EEG signals. To reduce human involvement, we introduce a data-driven methodology for online mental fatigue detection: self-weight ordinal regression (SWORE). Reaction time (RT), referring to the length of time people take to react to an emergency, is widely considered an objective behavioral measure for mental fatigue state. Since regression methods are sensitive to extreme RTs, we propose an indirect RT estimation based on preferences to explore the relationship between EEG and RT, which generalizes to any scenario when an objective fatigue indicator is available. In particular, SWORE evaluates the noisy EEG signals from multiple channels in terms of two states: shaking state and steady state. Modeling the shaking state can discriminate the reliable channels from the uninformative ones, while modeling the steady state can suppress the task-nonrelevant fluctuation within each channel. In addition, an online generalized Bayesian moment matching (online GBMM) algorithm is proposed to online-calibrate SWORE efficiently per participant. Experimental results with 40 participants show that SWORE can maximally achieve consistent with RT, demonstrating the feasibility and adaptability of our proposed framework in practical mental fatigue estimation.

A Unified Analytical Framework With Multiple fNIRS Features for Mental Workload Assessment in the Prefrontal Cortex

Knowing the actual level of mental workload is important to ensure the efficacy of brain-computer interface (BCI) based cognitive training. Extracting signals from limited area of a brain region might not reveal the actual information. In this study, a functional near-infrared spectroscopy (fNIRS) device equipped with multi-channel and multi-distance measurement capability was employed for the development of an analytical framework to assess mental workload in the prefrontal cortex (PFC). In addition to the conventional features, e.g. hemodynamic slope, we introduced a new feature - deep contribution ratio which is the proportion of cerebral hemodynamics to the fNIRS signals. Multiple sets of features were examined by a simple logical operator to suppress the false detection rate in identifying the activated channels. Using the number of activated channels as input to a linear support vector machine (SVM), the performance of the proposed analytical framework was assessed in classifying three levels of mental workload. The best set of features involves the combination of hemodynamic slope and deep contribution ratio, where the identified number of activated channels returned an average accuracy of 80.6% in predicting mental workload, compared to a single conventional feature (accuracy: 59.8%). This suggests the feasibility of the proposed analytical framework with multiple features as a means towards a more accurate assessment of mental workload in fNIRS-based BCI applications.

Nonlinear adaptive NeuroFuzzy feedback linearization based MPPT control schemes for photovoltaic system in microgrid

Renewable energy resources connected to a single utility grid system require highly nonlinear control algorithms to maintain efficient operation concerning power output and stability under varying operating conditions. This research work presents a comparative analysis of different adaptive Feedback Linearization (FBL) embedded Full Recurrent Adaptive NeuroFuzzy (FRANF) control schemes for maximum power point tracking (MPPT) of PV subsystem tied to a smart microgrid hybrid power system (SMG-HPS). The proposed schemes are differentiated based on structure and mathematical functions used in FRANF embedded in the FBL model. The comparative analysis is carried out based on efficiency and performance indexes obtained using the power error between the reference and the tracked power for three cases; a) step change in solar irradiation and temperature, b) partial shading condition (PSC), and c) daily field data. The proposed schemes offer enhanced convergence compared to existing techniques in terms of complexity and stability. The overall performance of all the proposed schemes is evaluated by a spider chart of multivariate comparable parameters. Adaptive PID is used for the comparison of results produced by proposed control schemes. The performance of Mexican hat wavelet-based FRANF embedded FBL is superior to the other proposed schemes as well as to aPID based MPPT scheme. However, all proposed schemes produce better results as compared to conventional MPPT control in all cases. Matlab/Simulink is used to carry out the simulations.

An iterative cross-subject negative-unlabeled learning algorithm for quantifying passive fatigue

OBJECTIVE

This paper proposes an iterative negative-unlabeled (NU) learning algorithm for cross-subject detection of passive fatigue from labelled alert (negative) and unlabeled driving EEG data.

APPROACH

Unlike other studies which used manual labeling of the fatigue state, the proposed algorithm (PA) first iteratively uses 29 subjects' alert data and unlabeled driving data to identify the most fatigued block of EEG data in each subject in a cross-subject manner. Subsequently, the PA computes subjects' driving fatigue score. Repeated measures correlations of the score to EEG band powers are then performed.

MAIN RESULTS

The PA yields an averaged accuracy of 93.77%  ±  8.15% across subjects in detecting fatigue, which is significantly better than the various baselines. The fatigue scores obtained are also significantly positively correlated with theta band power and negatively correlated with beta band power that are known to respectively increase and decrease in presence of passive fatigue. There is a strong negative correlation with alpha band power as well.

SIGNIFICANCE

The proposed iterative NU learning algorithm is capable of labelling and quantifying the most fatigued block in a cross-subject manner despite the lack of ground truth in the fatigue levels of unlabeled driving EEG data. Together with the significant correlations with theta, alpha and beta band power, the results show promise in the application of the proposed algorithm to detect fatigue from EEG.

Deep Learning Intervention for Health Care Challenges: Some Biomedical Domain Considerations

The use of deep learning (DL) for the analysis and diagnosis of biomedical and health care problems has received unprecedented attention in the last decade. The technique has recorded a number of achievements for unearthing meaningful features and accomplishing tasks that were hitherto difficult to solve by other methods and human experts. Currently, biological and medical devices, treatment, and applications are capable of generating large volumes of data in the form of images, sounds, text, graphs, and signals creating the concept of big data. The innovation of DL is a developing trend in the wake of big data for data representation and analysis. DL is a type of machine learning algorithm that has deeper (or more) hidden layers of similar function cascaded into the network and has the capability to make meaning from medical big data. Current transformation drivers to achieve personalized health care delivery will be possible with the use of mobile health (mHealth). DL can provide the analysis for the deluge of data generated from mHealth apps. This paper reviews the fundamentals of DL methods and presents a general view of the trends in DL by capturing literature from PubMed and the Institute of Electrical and Electronics Engineers database publications that implement different variants of DL. We highlight the implementation of DL in health care, which we categorize into biological system, electronic health record, medical image, and physiological signals. In addition, we discuss some inherent challenges of DL affecting biomedical and health domain, as well as prospective research directions that focus on improving health management by promoting the application of physiological signals and modern internet technology.

Multi-channel EEG recordings during a sustained-attention driving task

We describe driver behaviour and brain dynamics acquired from a 90-minute sustained-attention task in an immersive driving simulator. The data included 62 sessions of 32-channel electroencephalography (EEG) data for 27 subjects driving on a four-lane highway who were instructed to keep the car cruising in the centre of the lane. Lane-departure events were randomly induced to cause the car to drift from the original cruising lane towards the left or right lane. A complete trial included events with deviation onset, response onset, and response offset. The next trial, in which the subject was instructed to drive back to the original cruising lane, began 5-10 seconds after finishing the previous trial. We believe that this dataset will lead to the development of novel neural processing methodology that can be used to index brain cortical dynamics and detect driving fatigue and drowsiness. This publicly available dataset will be beneficial to the neuroscience and brain-computer interface communities.

An Efficient Second-Order Algorithm for Self-Organizing Fuzzy Neural Networks

Intelligent computing technologies are useful and important for online data modeling, where system dynamics may be nonstationary with some uncertainties. In this paper, an efficient learning mechanism is developed for building self-organizing fuzzy neural networks (SOFNNs), where a second-order algorithm (SOA) with adaptive learning rate is employed, the network size and the parameters can be determined simultaneously in the learning process. First, all parameters of SOFNN are adjusted by using the SOA strategy to achieve fast convergence through a powerful search scheme. Second, the structure of SOFNN can be self-organized using the relative importance index of each rule. The fuzzy rules used in SOFNN with SOA (SOA-SOFNN) are generated or pruned automatically to reduce the computational complexity and potentially improve the generalization power. Finally, a theoretical analysis on the learning convergence of the proposed SOA-SOFNN is given to show the computational efficiency. To demonstrate the merits of our proposed approach for data modeling, several benchmark datasets, and a real world application associated with nonlinear systems modeling problems are examined with comparisons against other existing methods. The results indicate that our proposed SOA-SOFNN performs favorably in terms of both learning speed and prediction accuracy for online data modeling.

Cross-Participant EEG-Based Assessment of Cognitive Workload Using Multi-Path Convolutional Recurrent Neural Networks

Applying deep learning methods to electroencephalograph (EEG) data for cognitive state assessment has yielded improvements over previous modeling methods. However, research focused on cross-participant cognitive workload modeling using these techniques is underrepresented. We study the problem of cross-participant state estimation in a non-stimulus-locked task environment, where a trained model is used to make workload estimates on a new participant who is not represented in the training set. Using experimental data from the Multi-Attribute Task Battery (MATB) environment, a variety of deep neural network models are evaluated in the trade-space of computational efficiency, model accuracy, variance and temporal specificity yielding three important contributions: (1) The performance of ensembles of individually-trained models is statistically indistinguishable from group-trained methods at most sequence lengths. These ensembles can be trained for a fraction of the computational cost compared to group-trained methods and enable simpler model updates. (2) While increasing temporal sequence length improves mean accuracy, it is not sufficient to overcome distributional dissimilarities between individuals’ EEG data, as it results in statistically significant increases in cross-participant variance. (3) Compared to all other networks evaluated, a novel convolutional-recurrent model using multi-path subnetworks and bi-directional, residual recurrent layers resulted in statistically significant increases in predictive accuracy and decreases in cross-participant variance.

A mental fatigue index based on regression using mulitband EEG features with application in simulated driving

Development of accurate fatigue level prediction models is of great importance for driving safety. In parallel, a limited number of sensors is a prerequisite for development of applicable wearable devices. Several EEG-based studies so far have performed classification in two or few levels, while others have proposed indices based on power ratios. Here, we utilized a regression Random Forest model in order to provide more accurate continuous fatigue level prediction. In detail, multiband power features were extracted from EEG data recorded from one hour simulated driving task. Next, cross-subject regression was performed to obtain common fatigue-related discriminative features. We achieved satisfactory prediction accuracy and simultaneously we minimized required electrodes, proposing to use a set of 3 electrodes.

Correlation of reaction time and EEG log bandpower from dry frontal electrodes in a passive fatigue driving simulation experiment

Fatigue is one of the causes of falling asleep at the wheel, which can result in fatal accidents. Thus, it is necessary to have practical fatigue detection solutions for drivers. In literature, electroencephalography (EEG) along with the surrogate measure of reaction time (RT) has been used to develop fatigue detection algorithms. However, these solutions are often based upon wet multi-channel EEG electrodes which are not feasible or practical for drivers. Using dry electrodes and headband like designs would be better. Hence, this study aims to investigate the correlation of EEG log bandpower against RT via a Muse headband which has dry frontal EEG electrodes. 31 subjects underwent an hour-long driving simulation experiment with car deviation events. Based on the video and EEG data, 5 `Sleepy' and 5 `Alert' subjects are identified and analyzed. A differential signal between Fp1 and Fp2 is computed so as to remove the effects of eye blinks, and is analyzed for correlation with RT. Significant positive correlation is found for log delta (1-4 Hz) bandpower, and significant negative correlations for log theta (4-8 Hz) and alpha (8-12 Hz) bandpowers, but the positive correlation of log beta (12-30 Hz) bandpower with RT is not significant. This is a good first step towards building a practical fatigue detection solution for drivers in the future.

Fuzzy Decision-Making Fuser (FDMF) for Integrating Human-Machine Autonomous (HMA) Systems with Adaptive Evidence Sources

A brain-computer interface (BCI) creates a direct communication pathway between the human brain and an external device or system. In contrast to patient-oriented BCIs, which are intended to restore inoperative or malfunctioning aspects of the nervous system, a growing number of BCI studies focus on designing auxiliary systems that are intended for everyday use. The goal of building these BCIs is to provide capabilities that augment existing intact physical and mental capabilities. However, a key challenge to BCI research is human variability; factors such as fatigue, inattention, and stress vary both across different individuals and for the same individual over time. If these issues are addressed, autonomous systems may provide additional benefits that enhance system performance and prevent problems introduced by individual human variability. This study proposes a human-machine autonomous (HMA) system that simultaneously aggregates human and machine knowledge to recognize targets in a rapid serial visual presentation (RSVP) task. The HMA focuses on integrating an RSVP BCI with computer vision techniques in an image-labeling domain. A fuzzy decision-making fuser (FDMF) is then applied in the HMA system to provide a natural adaptive framework for evidence-based inference by incorporating an integrated summary of the available evidence (i.e., human and machine decisions) and associated uncertainty. Consequently, the HMA system dynamically aggregates decisions involving uncertainties from both human and autonomous agents. The collaborative decisions made by an HMA system can achieve and maintain superior performance more efficiently than either the human or autonomous agents can achieve independently. The experimental results shown in this study suggest that the proposed HMA system with the FDMF can effectively fuse decisions from human brain activities and the computer vision techniques to improve overall performance on the RSVP recognition task. This conclusion demonstrates the potential benefits of integrating autonomous systems with BCI systems.