Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review

Lemniscate of Bernoulli's map quantifiers: innovative measures for EEG emotion recognition

Thanks to the advent of affective computing, designing an automatic human emotion recognition system for clinical and non-clinical applications has attracted the attention of many researchers. Currently, multi-channel electroencephalogram (EEG)-based emotion recognition is a fundamental but challenging issue. This experiment envisioned developing a new scheme for automated EEG affect recognition. An innovative nonlinear feature engineering approach was presented based on Lemniscate of Bernoulli's Map (LBM), which belongs to the family of chaotic maps, in line with the EEG's nonlinear nature. As far as the authors know, LBM has not been utilized for biological signal analysis. Next, the map was characterized using several graphical indices. The feature vector was imposed on the feature selection algorithm while evaluating the role of the feature vector dimension on emotion recognition rates. Finally, the efficiency of the features on emotion recognition was appraised using two conventional classifiers and validated using the Database for Emotion Analysis using Physiological signals (DEAP) and SJTU Emotion EEG Dataset-IV (SEED-IV) benchmark databases. The experimental results showed a maximum accuracy of 92.16% for DEAP and 90.7% for SEED-IV. Achieving higher recognition rates compared to the state-of-art EEG emotion recognition systems suggest the proposed method based on LBM could have potential both in characterizing bio-signal dynamics and detecting affect-deficit disorders.

Usability Comparison Among Healthy Participants of an Anthropomorphic Digital Human and a Text-Based Chatbot as a Responder to Questions on Mental Health: Randomized Controlled Trial

BACKGROUND

The use of chatbots in mental health support has increased exponentially in recent years, with studies showing that they may be effective in treating mental health problems. More recently, the use of visual avatars called digital humans has been introduced. Digital humans have the capability to use facial expressions as another dimension in human-computer interactions. It is important to study the difference in emotional response and usability preferences between text-based chatbots and digital humans for interacting with mental health services.

OBJECTIVE

This study aims to explore to what extent a digital human interface and a text-only chatbot interface differed in usability when tested by healthy participants, using BETSY (Behavior, Emotion, Therapy System, and You) which uses 2 distinct interfaces: a digital human with anthropomorphic features and a text-only user interface. We also set out to explore how chatbot-generated conversations on mental health (specific to each interface) affected self-reported feelings and biometrics.

METHODS

We explored to what extent a digital human with anthropomorphic features differed from a traditional text-only chatbot regarding perception of usability through the System Usability Scale, emotional reactions through electroencephalography, and feelings of closeness. Healthy participants (n=45) were randomized to 2 groups that used a digital human with anthropomorphic features (n=25) or a text-only chatbot with no such features (n=20). The groups were compared by linear regression analysis and t tests.

RESULTS

No differences were observed between the text-only and digital human groups regarding demographic features. The mean System Usability Scale score was 75.34 (SD 10.01; range 57-90) for the text-only chatbot versus 64.80 (SD 14.14; range 40-90) for the digital human interface. Both groups scored their respective chatbot interfaces as average or above average in usability. Women were more likely to report feeling annoyed by BETSY.

CONCLUSIONS

The text-only chatbot was perceived as significantly more user-friendly than the digital human, although there were no significant differences in electroencephalography measurements. Male participants exhibited lower levels of annoyance with both interfaces, contrary to previously reported findings.

EEG-based emotion recognition using a temporal-difference minimizing neural network

Electroencephalogram (EEG) emotion recognition plays an important role in human-computer interaction. An increasing number of algorithms for emotion recognition have been proposed recently. However, it is still challenging to make efficient use of emotional activity knowledge. In this paper, based on prior knowledge that emotion varies slowly across time, we propose a temporal-difference minimizing neural network (TDMNN) for EEG emotion recognition. We use maximum mean discrepancy (MMD) technology to evaluate the difference in EEG features across time and minimize the difference by a multibranch convolutional recurrent network. State-of-the-art performances are achieved using the proposed method on the SEED, SEED-IV, DEAP and DREAMER datasets, demonstrating the effectiveness of including prior knowledge in EEG emotion recognition.

Brain-computer interface digital prescription for neurological disorders

Neurological and psychiatric diseases can lead to motor, language, emotional disorder, and cognitive, hearing or visual impairment By decoding the intention of the brain in real time, the Brain-computer interface (BCI) can first assist in the diagnosis of diseases, and can also compensate for its damaged function by directly interacting with the environment; In addition, provide output signals in various forms, such as actual motion, tactile or visual feedback, to assist in rehabilitation training; Further intervention in brain disorders is achieved by close-looped neural modulation. In this article, we envision the future BCI digital prescription system for patients with different functional disorders and discuss the key contents in the prescription the brain signals, coding and decoding protocols and interaction paradigms, and assistive technology. Then, we discuss the details that need to be specially included in the digital prescription for different intervention technologies. The third part summarizes previous examples of intervention, focusing on how to select appropriate interaction paradigms for patients with different functional impairments. For the last part, we discussed the indicators and influencing factors in evaluating the therapeutic effect of BCI as intervention.

Automatic Classification of Emotions Based on Cardiac Signals: A Systematic Literature Review

Emotions play a pivotal role in human cognition, exerting influence across diverse domains of individuals' lives. The widespread adoption of artificial intelligence and machine learning has spurred interest in systems capable of automatically recognizing and classifying emotions and affective states. However, the accurate identification of human emotions remains a formidable challenge, as they are influenced by various factors and accompanied by physiological changes. Numerous solutions have emerged to enable emotion recognition, leveraging the characterization of biological signals, including the utilization of cardiac signals acquired from low-cost and wearable sensors. The objective of this work was to comprehensively investigate the current trends in the field by conducting a Systematic Literature Review (SLR) that focuses specifically on the detection, recognition, and classification of emotions based on cardiac signals, to gain insights into the prevailing techniques employed for signal acquisition, the extracted features, the elicitation process, and the classification methods employed in these studies. A SLR was conducted using four research databases, and articles were assessed concerning the proposed research questions. Twenty seven articles met the selection criteria and were assessed for the feasibility of using cardiac signals, acquired from low-cost and wearable devices, for emotion recognition. Several emotional elicitation methods were found in the literature, including the algorithms applied for automatic classification, as well as the key challenges associated with emotion recognition relying solely on cardiac signals. This study extends the current body of knowledge and enables future research by providing insights into suitable techniques for designing automatic emotion recognition applications. It emphasizes the importance of utilizing low-cost, wearable, and unobtrusive devices to acquire cardiac signals for accurate and accessible emotion recognition.

Enhancing the accuracy of electroencephalogram-based emotion recognition through Long Short-Term Memory recurrent deep neural networks

Introduction

Emotions play a critical role in human communication, exerting a significant influence on brain function and behavior. One effective method of observing and analyzing these emotions is through electroencephalography (EEG) signals. Although numerous studies have been dedicated to emotion recognition (ER) using EEG signals, achieving improved accuracy in recognition remains a challenging task. To address this challenge, this paper presents a deep-learning approach for ER using EEG signals.

Background

ER is a dynamic field of research with diverse practical applications in healthcare, human-computer interaction, and affective computing. In ER studies, EEG signals are frequently employed as they offer a non-invasive and cost-effective means of measuring brain activity. Nevertheless, accurately identifying emotions from EEG signals poses a significant challenge due to the intricate and non-linear nature of these signals.

Methods

The present study proposes a novel approach for ER that encompasses multiple stages, including feature extraction, feature selection (FS) employing clustering, and classification using Dual-LSTM. To conduct the experiments, the DEAP dataset was employed, wherein a clustering technique was applied to Hurst's view and statistical features during the FS phase. Ultimately, Dual-LSTM was employed for accurate ER.

Results

The proposed method achieved a remarkable accuracy of 97.5% in accurately classifying emotions across four categories: arousal, valence, liking/disliking, dominance, and familiarity. This high level of accuracy serves as strong evidence for the effectiveness of the deep-learning approach to emotion recognition (ER) utilizing EEG signals.

Conclusion

The deep-learning approach proposed in this paper has shown promising results in emotion recognition using EEG signals. This method can be useful in various applications, such as developing more effective therapies for individuals with mood disorders or improving human-computer interaction by allowing machines to respond more intelligently to users' emotional states. However, further research is needed to validate the proposed method on larger datasets and to investigate its applicability to real-world scenarios.

EEG-Based Emotion Recognition with Consideration of Individual Difference

Electroencephalograms (EEGs) are often used for emotion recognition through a trained EEG-to-emotion models. The training samples are EEG signals recorded while participants receive external induction labeled as various emotions. Individual differences such as emotion degree and time response exist under the same external emotional inductions. These differences can lead to a decrease in the accuracy of emotion classification models in practical applications. The brain-based emotion recognition model proposed in this paper is able to sufficiently consider these individual differences. The proposed model comprises an emotion classification module and an individual difference module (IDM). The emotion classification module captures the spatial and temporal features of the EEG data, while the IDM introduces personalized adjustments to specific emotional features by accounting for participant-specific variations as a form of interference. This approach aims to enhance the classification performance of EEG-based emotion recognition for diverse participants. The results of our comparative experiments indicate that the proposed method obtains a maximum accuracy of 96.43% for binary classification on DEAP data. Furthermore, it performs better in scenarios with significant individual differences, where it reaches a maximum accuracy of 98.92%.

Machine learning-based classifying of risk-takers and risk-aversive individuals using resting-state EEG data: A pilot feasibility study

BACKGROUND

Decision-making is vital in interpersonal interactions and a country's economic and political conditions. People, especially managers, have to make decisions in different risky situations. There has been a growing interest in identifying managers' personality traits (i.e., risk-taking or risk-averse) in recent years. Although there are findings of signal decision-making and brain activity, the implementation of an intelligent brain-based technique to predict risk-averse and risk-taking managers is still in doubt.

METHODS

This study proposes an electroencephalogram (EEG)-based intelligent system to distinguish risk-taking managers from risk-averse ones by recording the EEG signals from 30 managers. In particular, wavelet transform, a time-frequency domain analysis method, was used on resting-state EEG data to extract statistical features. Then, a two-step statistical wrapper algorithm was used to select the appropriate features. The support vector machine classifier, a supervised learning method, was used to classify two groups of managers using chosen features.

RESULTS

Intersubject predictive performance could classify two groups of managers with 74.42% accuracy, 76.16% sensitivity, 72.32% specificity, and 75% F1-measure, indicating that machine learning (ML) models can distinguish between risk-taking and risk-averse managers using the features extracted from the alpha frequency band in 10 s analysis window size.

CONCLUSIONS

The findings of this study demonstrate the potential of using intelligent (ML-based) systems in distinguish between risk-taking and risk-averse managers using biological signals.

Recent advancements in multimodal human-robot interaction

Robotics have advanced significantly over the years, and human-robot interaction (HRI) is now playing an important role in delivering the best user experience, cutting down on laborious tasks, and raising public acceptance of robots. New HRI approaches are necessary to promote the evolution of robots, with a more natural and flexible interaction manner clearly the most crucial. As a newly emerging approach to HRI, multimodal HRI is a method for individuals to communicate with a robot using various modalities, including voice, image, text, eye movement, and touch, as well as bio-signals like EEG and ECG. It is a broad field closely related to cognitive science, ergonomics, multimedia technology, and virtual reality, with numerous applications springing up each year. However, little research has been done to summarize the current development and future trend of HRI. To this end, this paper systematically reviews the state of the art of multimodal HRI on its applications by summing up the latest research articles relevant to this field. Moreover, the research development in terms of the input signal and the output signal is also covered in this manuscript.

Electroencephalogram based brain-computer interface: Applications, challenges, and opportunities

Brain-Computer Interfaces (BCI) is an exciting and emerging research area for researchers and scientists. It is a suitable combination of software and hardware to operate any device mentally. This review emphasizes the significant stages in the BCI domain, current problems, and state-of-the-art findings. This article also covers how current results can contribute to new knowledge about BCI, an overview of BCI from its early developments to recent advancements, BCI applications, challenges, and future directions. The authors pointed to unresolved issues and expressed how BCI is valuable for analyzing the human brain. Humans' dependence on machines has led humankind into a new future where BCI can play an essential role in improving this modern world.

Emotion Recognition Using a Novel Granger Causality Quantifier and Combined Electrodes of EEG

Electroencephalogram (EEG) connectivity patterns can reflect neural correlates of emotion. However, the necessity of evaluating bulky data for multi-channel measurements increases the computational cost of the EEG network. To date, several approaches have been presented to pick the optimal cerebral channels, mainly depending on available data. Consequently, the risk of low data stability and reliability has increased by reducing the number of channels. Alternatively, this study suggests an electrode combination approach in which the brain is divided into six areas. After extracting EEG frequency bands, an innovative Granger causality-based measure was introduced to quantify brain connectivity patterns. The feature was subsequently subjected to a classification module to recognize valence-arousal dimensional emotions. A Database for Emotion Analysis Using Physiological Signals (DEAP) was used as a benchmark database to evaluate the scheme. The experimental results revealed a maximum accuracy of 89.55%. Additionally, EEG-based connectivity in the beta-frequency band was able to effectively classify dimensional emotions. In sum, combined EEG electrodes can efficiently replicate 32-channel EEG information.

Machine learning in biosignals processing for mental health: A narrative review

Machine Learning (ML) offers unique and powerful tools for mental health practitioners to improve evidence-based psychological interventions and diagnoses. Indeed, by detecting and analyzing different biosignals, it is possible to differentiate between typical and atypical functioning and to achieve a high level of personalization across all phases of mental health care. This narrative review is aimed at presenting a comprehensive overview of how ML algorithms can be used to infer the psychological states from biosignals. After that, key examples of how they can be used in mental health clinical activity and research are illustrated. A description of the biosignals typically used to infer cognitive and emotional correlates (e.g., EEG and ECG), will be provided, alongside their application in Diagnostic Precision Medicine, Affective Computing, and brain-computer Interfaces. The contents will then focus on challenges and research questions related to ML applied to mental health and biosignals analysis, pointing out the advantages and possible drawbacks connected to the widespread application of AI in the medical/mental health fields. The integration of mental health research and ML data science will facilitate the transition to personalized and effective medicine, and, to do so, it is important that researchers from psychological/ medical disciplines/health care professionals and data scientists all share a common background and vision of the current research.

Long-Term Exercise Assistance: Group and One-on-One Interactions between a Social Robot and Seniors

For older adults, regular exercises can provide both physical and mental benefits, increase their independence, and reduce the risks of diseases associated with aging. However, only a small portion of older adults regularly engage in physical activity. Therefore, it is important to promote exercise among older adults to help maintain overall health. In this paper, we present the first exploratory long-term human–robot interaction (HRI) study conducted at a local long-term care facility to investigate the benefits of one-on-one and group exercise interactions with an autonomous socially assistive robot and older adults. To provide targeted facilitation, our robot utilizes a unique emotion model that can adapt its assistive behaviors to users’ affect and track their progress towards exercise goals through repeated sessions using the Goal Attainment Scale (GAS), while also monitoring heart rate to prevent overexertion. Results of the study show that users had positive valence and high engagement towards the robot and were able to maintain their exercise performance throughout the study. Questionnaire results showed high robot acceptance for both types of interactions. However, users in the one-on-one sessions perceived the robot as more sociable and intelligent, and had more positive perception of the robot’s appearance and movements.

Identifying Thematics in a Brain-Computer Interface Research

This umbrella review is motivated to understand the shift in research themes on brain-computer interfacing (BCI) and it determined that a shift away from themes that focus on medical advancement and system development to applications that included education, marketing, gaming, safety, and security has occurred. The background of this review examined aspects of BCI categorisation, neuroimaging methods, brain control signal classification, applications, and ethics. The specific area of BCI software and hardware development was not examined. A search using One Search was undertaken and 92 BCI reviews were selected for inclusion. Publication demographics indicate the average number of authors on review papers considered was 4.2 ± 1.8. The results also indicate a rapid increase in the number of BCI reviews from 2003, with only three reviews before that period, two in 1972, and one in 1996. While BCI authors were predominantly Euro-American in early reviews, this shifted to a more global authorship, which China dominated by 2020-2022. The review revealed six disciplines associated with BCI systems: life sciences and biomedicine (n = 42), neurosciences and neurology (n = 35), and rehabilitation (n = 20); (2) the second domain centred on the theme of functionality: computer science (n = 20), engineering (n = 28) and technology (n = 38). There was a thematic shift from understanding brain function and modes of interfacing BCI systems to more applied research novel areas of research-identified surround artificial intelligence, including machine learning, pre-processing, and deep learning. As BCI systems become more invasive in the lives of "normal" individuals, it is expected that there will be a refocus and thematic shift towards increased research into ethical issues and the need for legal oversight in BCI application.

Identifying Complex Emotions in Alexithymia Affected Adolescents Using Machine Learning Techniques

Many scientific researchers' study focuses on enhancing automated systems to identify emotions and thus relies on brain signals. This study focuses on how brain wave signals can be used to classify many emotional states of humans. Electroencephalography (EEG)-based affective computing predominantly focuses on emotion classification based on facial expression, speech recognition, and text-based recognition through multimodality stimuli. The proposed work aims to implement a methodology to identify and codify discrete complex emotions such as pleasure and grief in a rare psychological disorder known as alexithymia. This type of disorder is highly elicited in unstable, fragile countries such as South Sudan, Lebanon, and Mauritius. These countries are continuously affected by civil wars and disaster and politically unstable, leading to a very poor economy and education system. This study focuses on an adolescent age group dataset by recording physiological data when emotion is exhibited in a multimodal virtual environment. We decocted time frequency analysis and amplitude time series correlates including frontal alpha symmetry using a complex Morlet wavelet. For data visualization, we used the UMAP technique to obtain a clear district view of emotions. We performed 5-fold cross validation along with 1 s window subjective classification on the dataset. We opted for traditional machine learning techniques to identify complex emotion labeling.

Multi-Kernel Temporal and Spatial Convolution for EEG-Based Emotion Classification

Deep learning using an end-to-end convolutional neural network (ConvNet) has been applied to several electroencephalography (EEG)-based brain-computer interface tasks to extract feature maps and classify the target output. However, the EEG analysis remains challenging since it requires consideration of various architectural design components that influence the representational ability of extracted features. This study proposes an EEG-based emotion classification model called the multi-kernel temporal and spatial convolution network (MultiT-S ConvNet). The multi-scale kernel is used in the model to learn various time resolutions, and separable convolutions are applied to find related spatial patterns. In addition, we enhanced both the temporal and spatial filters with a lightweight gating mechanism. To validate the performance and classification accuracy of MultiT-S ConvNet, we conduct subject-dependent and subject-independent experiments on EEG-based emotion datasets: DEAP and SEED. Compared with existing methods, MultiT-S ConvNet outperforms with higher accuracy results and a few trainable parameters. Moreover, the proposed multi-scale module in temporal filtering enables extracting a wide range of EEG representations, covering short- to long-wavelength components. This module could be further implemented in any model of EEG-based convolution networks, and its ability potentially improves the model's learning capacity.

Emotion recognition based on multi-modal physiological signals and transfer learning

In emotion recognition based on physiological signals, collecting enough labeled data of a single subject for training is time-consuming and expensive. The physiological signals’ individual differences and the inherent noise will significantly affect emotion recognition accuracy. To overcome the difference in subject physiological signals, we propose a joint probability domain adaptation with the bi-projection matrix algorithm (JPDA-BPM). The bi-projection matrix method fully considers the source and target domain’s different feature distributions. It can better project the source and target domains into the feature space, thereby increasing the algorithm’s performance. We propose a substructure-based joint probability domain adaptation algorithm (SSJPDA) to overcome physiological signals’ noise effect. This method can avoid the shortcomings that the domain level matching is too rough and the sample level matching is susceptible to noise. In order to verify the effectiveness of the proposed transfer learning algorithm in emotion recognition based on physiological signals, we verified it on the database for emotion analysis using physiological signals (DEAP dataset). The experimental results show that the average recognition accuracy of the proposed SSJPDA-BPM algorithm in the multimodal fusion physiological data from the DEAP dataset is 63.6 and 64.4% in valence and arousal, respectively. Compared with joint probability domain adaptation (JPDA), the performance of valence and arousal recognition accuracy increased by 17.6 and 13.4%, respectively.

An Approach to Emotion Recognition Using Brain Rhythm Sequencing and Asymmetric Features

Emotion can be influenced during self-isolation, and to avoid severe mood swings, emotional regulation is meaningful. To achieve this, efficiently recognizing emotion is a vital step, which can be realized by electroencephalography signals. Previously, inspired by the knowledge of sequencing in bioinformatics, a method termed brain rhythm sequencing that analyzes electroencephalography as the sequence consisting of the dominant rhythm has been proposed for seizure detection. In this work, with the help of similarity measure methods, the asymmetric features are extracted from the sequences generated by different channel data. After evaluating all asymmetric features for emotion recognition, the optimal feature that yields remarkable accuracy is identified. Therefore, the classification task can be accomplished through a small amount of channel data. From a music emotion recognition experiment and a public DEAP dataset, the classification accuracies of various test sets are approximately 80–85% when employing an optimal feature extracted from one pair of symmetrical channels. Such performances are impressive when using fewer resources is a concern. Further investigation revealed that emotion recognition shows strongly individual characteristics, so an appropriate solution is to include the subject-dependent properties. Compared to the existing works, this method benefits from the design of a portable emotion-aware device used during self-isolation, as fewer scalp sensors are needed. Hence, it would provide a novel way to realize emotional applications in the future.

EEG-Based Empathic Safe Cobot

An empathic collaborative robot (cobot) was realized through the transmission of fear from a human agent to a robot agent. Such empathy was induced through an electroencephalographic (EEG) sensor worn by the human agent, thus realizing an empathic safe brain-computer interface (BCI). The empathic safe cobot reacts to the fear and in turn transmits it to the human agent, forming a social circle of empathy and safety. A first randomized, controlled experiment involved two groups of 50 healthy subjects (100 total subjects) to measure the EEG signal in the presence or absence of a frightening event. The second randomized, controlled experiment on two groups of 50 different healthy subjects (100 total subjects) exposed the subjects to comfortable and uncomfortable movements of a collaborative robot (cobot) while the subjects’ EEG signal was acquired. The result was that a spike in the subject’s EEG signal was observed in the presence of uncomfortable movement. The questionnaires were distributed to the subjects, and confirmed the results of the EEG signal measurement. In a controlled laboratory setting, all experiments were found to be statistically significant. In the first experiment, the peak EEG signal measured just after the activating event was greater than the resting EEG signal (p < 10−3). In the second experiment, the peak EEG signal measured just after the uncomfortable movement of the cobot was greater than the EEG signal measured under conditions of comfortable movement of the cobot (p < 10−3). In conclusion, within the isolated and constrained experimental environment, the results were satisfactory.

Innovative Poincare's plot asymmetry descriptors for EEG emotion recognition

Given the importance of emotion recognition in both medical and non-medical applications, designing an automatic system has captured the attention of several scholars. Currently, EEG-based emotion recognition has a special position, which has not fulfilled the desired accuracy rates yet. This experiment intended to provide novel EEG asymmetry measures to improve emotion recognition rates. Four emotional states have been classified using the k-nearest neighbor (kNN), support vector machine, and Naïve Bayes. Feature selection has been performed, and the role of employing a different number of top-ranked features on emotion recognition rates has been assessed. To validate the efficiency of the proposed scheme, two public databases, including the SJTU Emotion EEG Dataset-IV (SEED-IV) and a Database for Emotion Analysis using Physiological signals (DEAP) were evaluated. The experimental results indicated that kNN outperformed the other classifiers with a maximum accuracy of 95.49 and 98.63% using SEED-IV and DEAP datasets, respectively. In conclusion, the results of the proposed novel EEG-asymmetry measures make the framework a superior one compared to the state-of-art EEG emotion recognition approaches.

Predict Students’ Attention in Online Learning Using EEG Data

In education, it is critical to monitor students’ attention and measure the extents to which students participate and the differences in their levels and abilities. The overall goal of this study was to increase the quality of distance education. In particular, in order to craft an approach that will effectively augment online learning using objective measures of brain activity, we propose a brain–computer interface (BCI) system that aims to use electroencephalography (EEG) signals for the detection of student’s attention during online classes. This system will aid teachers to objectively assess student attention and engagement. To this end, experiments were conducted on a public dataset; we extracted power spectral density (PSD) features using used a fast Fourier transform. Different attention indexes were calculated. Then, we built three different classification algorithms: k-nearest neighbors (KNN), support vector machine (SVM), and random forest (RF). Our proposed random forest classifier achieved a higher accuracy (96%) than KNN and SVM. Moreover, our results compared to state-of-the-art attention-detection systems with respect to the same dataset. Our findings revealed that the proposed RF approach can be used to effectively distinguish the attention state of a user.

Sparse representations of high dimensional neural data

Conventional Vector Autoregressive (VAR) modelling methods applied to high dimensional neural time series data result in noisy solutions that are dense or have a large number of spurious coefficients. This reduces the speed and accuracy of auxiliary computations downstream and inflates the time required to compute functional connectivity networks by a factor that is at least inversely proportional to the true network density. As these noisy solutions have distorted coefficients, thresholding them as per some criterion, statistical or otherwise, does not alleviate the problem. Thus obtaining a sparse representation of such data is important since it provides an efficient representation of the data and facilitates its further analysis. We propose a fast Sparse Vector Autoregressive Greedy Search (SVARGS) method that works well for high dimensional data, even when the number of time points is relatively low, by incorporating only statistically significant coefficients. In numerical experiments, our methods show high accuracy in recovering the true sparse model. The relative absence of spurious coefficients permits accurate, stable and fast evaluation of derived quantities such as power spectrum, coherence and Granger causality. Consequently, sparse functional connectivity networks can be computed, in a reasonable time, from data comprising tens of thousands of channels/voxels. This enables a much higher resolution analysis of functional connectivity patterns and community structures in such large networks than is possible using existing time series methods. We apply our method to EEG data where computed network measures and community structures are used to distinguish emotional states as well as to ADHD fMRI data where it is used to distinguish children with ADHD from typically developing children.

Past, Present, and Future of EEG-Based BCI Applications

An electroencephalography (EEG)-based brain-computer interface (BCI) is a system that provides a pathway between the brain and external devices by interpreting EEG. EEG-based BCI applications have initially been developed for medical purposes, with the aim of facilitating the return of patients to normal life. In addition to the initial aim, EEG-based BCI applications have also gained increasing significance in the non-medical domain, improving the life of healthy people, for instance, by making it more efficient, collaborative and helping develop themselves. The objective of this review is to give a systematic overview of the literature on EEG-based BCI applications from the period of 2009 until 2019. The systematic literature review has been prepared based on three databases PubMed, Web of Science and Scopus. This review was conducted following the PRISMA model. In this review, 202 publications were selected based on specific eligibility criteria. The distribution of the research between the medical and non-medical domain has been analyzed and further categorized into fields of research within the reviewed domains. In this review, the equipment used for gathering EEG data and signal processing methods have also been reviewed. Additionally, current challenges in the field and possibilities for the future have been analyzed.

Techniques and countermeasures for preventing insider threats

With the wide use of technologies nowadays, various security issues have emerged. Public and private sectors are both spending a large portion of their budget to protect the confidentiality, integrity, and availability of their data from possible attacks. Among these attacks are insider attacks which are more serious than external attacks, as insiders are authorized users who have legitimate access to sensitive assets of an organization. As a result, several studies exist in the literature aimed to develop techniques and tools to detect and prevent various types of insider threats. This article reviews different techniques and countermeasures that are proposed to prevent insider attacks. A unified classification model is proposed to classify the insider threat prevention approaches into two categories (biometric-based and asset-based metric). The biometric-based category is also classified into (physiological, behavioral and physical), while the asset metric-based category is also classified into (host, network and combined). This classification systematizes the reviewed approaches that are validated with empirical results utilizing the grounded theory method for rigorous literature review. Additionally, the article compares and discusses significant theoretical and empirical factors that play a key role in the effectiveness of insider threat prevention approaches (e.g., datasets, feature domains, classification algorithms, evaluation metrics, real-world simulation, stability and scalability, etc.). Major challenges are also highlighted which need to be considered when deploying real-world insider threat prevention systems. Some research gaps and recommendations are also presented for future research directions.

Emotion Self-Regulation in Neurotic Students: A Pilot Mindfulness-Based Intervention to Assess Its Effectiveness through Brain Signals and Behavioral Data

Neuroticism has recently received increased attention in the psychology field due to the finding of high implications of neuroticism on an individual’s life and broader public health. This study aims to investigate the effect of a brief 6-week breathing-based mindfulness intervention (BMI) on undergraduate neurotic students’ emotion regulation. We acquired data of their psychological states, physiological changes, and electroencephalogram (EEG), before and after BMI, in resting states and tasks. Through behavioral analysis, we found the students’ anxiety and stress levels significantly reduced after BMI, with p-values of 0.013 and 0.027, respectively. Furthermore, a significant difference between students in emotion regulation strategy, that is, suppression, was also shown. The EEG analysis demonstrated significant differences between students before and after MI in resting states and tasks. Fp1 and O2 channels were identified as the most significant channels in evaluating the effect of BMI. The potential of these channels for classifying (single-channel-based) before and after BMI conditions during eyes-opened and eyes-closed baseline trials were displayed by a good performance in terms of accuracy (~77%), sensitivity (76–80%), specificity (73–77%), and area-under-the-curve (AUC) (0.66–0.8) obtained by k-nearest neighbor (KNN) and support vector machine (SVM) algorithms. Mindfulness can thus improve the self-regulation of the emotional state of neurotic students based on the psychometric and electrophysiological analyses conducted in this study.

SCC-MPGCN: Self-Attention Coherence Clustering Based on Multi-Pooling Graph Convolutional Network for EEG Emotion Recognition

The emotion recognition with electroencephalography (EEG) has been widely studied using the deep learning methods, but the topology of EEG channels is rarely exploited completely. In this paper, we propose a self-attention coherence clustering based on multi-pooling graph convolutional network (SCC-MPGCN) model for EEG emotion recognition. locking value (PLV)The adjacency matrix is constructed based on phase- to describe the intrinsic relationship between different EEG electrodes as graph signals. The Laplacian matrix of a graph is obtained from the adjacency matrix and then is fed into the graph convolutional layers to learn the generalized features. Moreover, we propose a novel graph coarsening method called self-attention coherence clustering (SCC), using the coherence to cluster the nodes. The benefits are that the global information can be achieved from the raw data and the dimensionality of input can be reduced. Meanwhile, a multi-pooling graph convolutional network (MPGCN) block is introduced to learn the generalized emotional states features and tackle the problem of imbalanced dimensionality. The fully-connected layer and a softmax layer are adopted to drive the final prediction. We carry out the extensive experiments on DEAP dataset and the experimental results show that the proposed method has better classification results than the state-of-the-art methods with the 10-fold cross-validation. And the model achieves the emotion recognition performance with a mean accuracy of 96.37%, 97.02%, 96.72% on valence, arousal, and dominance dimension, respectively.

Review on EEG-Based Authentication Technology

With the rapid development of brain-computer interface technology, as a new biometric feature, EEG signal has been widely concerned in recent years. The safety of brain-computer interface and the long-term insecurity of biometric authentication have a new solution. This review analyzes the biometrics of EEG signals, and the latest research is involved in the authentication process. This review mainly introduced the method of EEG-based authentication and systematically introduced EEG-based biometric cryptosystems for authentication for the first time. In cryptography, the key is the core basis of authentication in the cryptographic system, and cryptographic technology can effectively improve the security of biometric authentication and protect biometrics. The revocability of EEG-based biometric cryptosystems is an advantage that traditional biometric authentication does not have. Finally, the existing problems and future development directions of identity authentication technology based on EEG signals are proposed, providing a reference for the related studies.

Applying machine learning EEG signal classification to emotion‑related brain anticipatory activity

Machine learning approaches have been fruitfully applied to several neurophysiological signal classification problems. Considering the relevance of emotion in human cognition and behaviour, an important application of machine learning has been found in the field of emotion identification based on neurophysiological activity. Nonetheless, there is high variability in results in the literature depending on the neuronal activity measurement, the signal features and the classifier type. The present work aims to provide new methodological insight into machine learning applied to emotion identification based on electrophysiological brain activity. For this reason, we analysed previously recorded EEG activity measured while emotional stimuli, high and low arousal (auditory and visual) were provided to a group of healthy participants. Our target signal to classify was the pre-stimulus onset brain activity. Classification performance of three different classifiers (LDA, SVM and kNN) was compared using both spectral and temporal features. Furthermore, we also contrasted the performance of static and dynamic (time evolving) approaches. The best static feature-classifier combination was the SVM with spectral features (51.8%), followed by LDA with spectral features (51.4%) and kNN with temporal features (51%). The best dynamic feature classifier combination was the SVM with temporal features (63.8%), followed by kNN with temporal features (63.70%) and LDA with temporal features (63.68%). The results show a clear increase in classification accuracy with temporal dynamic features.

EEG Mental Stress Assessment Using Hybrid Multi-Domain Feature Sets of Functional Connectivity Network and Time-Frequency Features

Exposure to mental stress for long period leads to serious accidents and health problems. To avoid negative consequences on health and safety, it is very important to detect mental stress at its early stages, i.e., when it is still limited to acute or episodic stress. In this study, we developed an experimental protocol to induce two different levels of stress by utilizing a mental arithmetic task with time pressure and negative feedback as the stressors. We assessed the levels of stress on 22 healthy subjects using frontal electroencephalogram (EEG) signals, salivary alpha-amylase level (AAL), and multiple machine learning (ML) classifiers. The EEG signals were analyzed using a fusion of functional connectivity networks estimated by the Phase Locking Value (PLV) and temporal and spectral domain features. A total of 210 different features were extracted from all domains. Only the optimum multi-domain features were used for classification. We then quantified stress levels using statistical analysis and seven ML classifiers. Our result showed that the AAL level was significantly increased (p < 0.01) under stress condition in all subjects. Likewise, the functional connectivity network demonstrated a significant decrease under stress, p < 0.05. Moreover, we achieved the highest stress classification accuracy of 93.2% using the Support Vector Machine (SVM) classifier. Other classifiers produced relatively similar results.

Positive and Negative Emotion Classification Based on Multi-channel

The EEG features of different emotions were extracted based on multi-channel and forehead channels in this study. The EEG signals of 26 subjects were collected by the emotional video evoked method. The results show that the energy ratio and differential entropy of the frequency band can be used to classify positive and negative emotions effectively, and the best effect can be achieved by using an SVM classifier. When only the forehead and forehead signals are used, the highest classification accuracy can reach 66%. When the data of all channels are used, the highest accuracy of the model can reach 82%. After channel selection, the best model of this study can be obtained. The accuracy is more than 86%.

Predicting Human Intention-Behavior Through EEG Signal Analysis Using Multi-Scale CNN

At present, the application of Electroencephalogram (EEG) signal classification to human intention-behavior prediction has become a hot topic in the brain computer interface (BCI) research field. In recent studies, the introduction of convolutional neural networks (CNN) has contributed to substantial improvements in the EEG signal classification performance. However, there is still a key challenge with the existing CNN-based EEG signal classification methods, the accuracy of them is not very satisfying. This is because most of the existing methods only utilize the feature maps in the last layer of CNN for EEG signal classification, which might miss some local and detailed information for accurate classification. To address this challenge, this paper proposes a multi-scale CNN model-based EEG signal classification method. In this method, first, the EEG signals are preprocessed and converted to time-frequency images using the short-time Fourier Transform (STFT) technique. Then, a multi-scale CNN model is designed for EEG signal classification, which takes the converted time-frequency image as the input. Especially, in the designed multi-scale CNN model, both the local and global information is taken into consideration. The performance of the proposed method is verified on the benchmark data set 2b used in the BCI contest IV. The experimental results show that the average accuracy of the proposed method is 73.9 percent, which improves the classification accuracy of 10.4, 5.5, 16.2 percent compared with the traditional methods including artificial neural network, support vector machine, and stacked auto-encoder.

Automatic subject-specific spatiotemporal feature selection for subject-independent affective BCI

The dimensionality of the spatially distributed channels and the temporal resolution of electroencephalogram (EEG) based brain-computer interfaces (BCI) undermine emotion recognition models. Thus, prior to modeling such data, as the final stage of the learning pipeline, adequate preprocessing, transforming, and extracting temporal (i.e., time-series signals) and spatial (i.e., electrode channels) features are essential phases to recognize underlying human emotions. Conventionally, inter-subject variations are dealt with by avoiding the sources of variation (e.g., outliers) or turning the problem into a subject-deponent. We address this issue by preserving and learning from individual particularities in response to affective stimuli. This paper investigates and proposes a subject-independent emotion recognition framework that mitigates the subject-to-subject variability in such systems. Using an unsupervised feature selection algorithm, we reduce the feature space that is extracted from time-series signals. For the spatial features, we propose a subject-specific unsupervised learning algorithm that learns from inter-channel co-activation online. We tested this framework on real EEG benchmarks, namely DEAP, MAHNOB-HCI, and DREAMER. We train and test the selection outcomes using nested cross-validation and a support vector machine (SVM). We compared our results with the state-of-the-art subject-independent algorithms. Our results show an enhanced performance by accurately classifying human affection (i.e., based on valence and arousal) by 16%–27% compared to other studies. This work not only outperforms other subject-independent studies reported in the literature but also proposes an online analysis solution to affection recognition.

Brain-Computer Interface: Advancement and Challenges

Brain-Computer Interface (BCI) is an advanced and multidisciplinary active research domain based on neuroscience, signal processing, biomedical sensors, hardware, etc. Since the last decades, several groundbreaking research has been conducted in this domain. Still, no comprehensive review that covers the BCI domain completely has been conducted yet. Hence, a comprehensive overview of the BCI domain is presented in this study. This study covers several applications of BCI and upholds the significance of this domain. Then, each element of BCI systems, including techniques, datasets, feature extraction methods, evaluation measurement matrices, existing BCI algorithms, and classifiers, are explained concisely. In addition, a brief overview of the technologies or hardware, mostly sensors used in BCI, is appended. Finally, the paper investigates several unsolved challenges of the BCI and explains them with possible solutions.

A new semi-supervised algorithm combined with MCICA optimizing SVM for motion imagination EEG classification

This paper proposed a new semi-supervised algorithm combined with Mutual-cross Imperial Competition Algorithm (MCICA) optimizing Support Vector Machine (SVM) for motion imagination EEG classification, which not only reduces the tedious and time-consuming training process and enhances the adaptability of Brain Computer Interface (BCI), but also utilizes the MCICA to optimize the parameters of SVM in the semi-supervised process. This algorithm combines mutual information and cross validation to construct objective function in the semi-supervised training process, and uses the constructed objective function to establish the semi-supervised model of MCICA for optimizing the parameters of SVM, and finally applies the selected optimal parameters to the data set Iva of 2005 BCI competition to verify its effectiveness. The results showed that the proposed algorithm is effective in optimizing parameters and has good robustness and generalization in solving small sample classification problems.

EEG data augmentation for emotion recognition with a multiple generator conditional Wasserstein GAN

EEG-based emotion recognition has attracted substantial attention from researchers due to its extensive application prospects, and substantial progress has been made in feature extraction and classification modelling from EEG data. However, insufficient high-quality training data are available for building EEG-based emotion recognition models via machine learning or deep learning methods. The artificial generation of high-quality data is an effective approach for overcoming this problem. In this paper, a multi-generator conditional Wasserstein GAN method is proposed for the generation of high-quality artificial that covers a more comprehensive distribution of real data through the use of various generators. Experimental results demonstrate that the artificial data that are generated by the proposed model can effectively improve the performance of emotion classification models that are based on EEG.

Motor Training Using Mental Workload (MWL) With an Assistive Soft Exoskeleton System: A Functional Near-Infrared Spectroscopy (fNIRS) Study for Brain-Machine Interface (BMI)

Mental workload is a neuroergonomic human factor, which is widely used in planning a system's safety and areas like brain-machine interface (BMI), neurofeedback, and assistive technologies. Robotic prosthetics methodologies are employed for assisting hemiplegic patients in performing routine activities. Assistive technologies' design and operation are required to have an easy interface with the brain with fewer protocols, in an attempt to optimize mobility and autonomy. The possible answer to these design questions may lie in neuroergonomics coupled with BMI systems. In this study, two human factors are addressed: designing a lightweight wearable robotic exoskeleton hand that is used to assist the potential stroke patients with an integrated portable brain interface using mental workload (MWL) signals acquired with portable functional near-infrared spectroscopy (fNIRS) system. The system may generate command signals for operating a wearable robotic exoskeleton hand using two-state MWL signals. The fNIRS system is used to record optical signals in the form of change in concentration of oxy and deoxygenated hemoglobin (HbO and HbR) from the pre-frontal cortex (PFC) region of the brain. Fifteen participants participated in this study and were given hand-grasping tasks. Two-state MWL signals acquired from the PFC region of the participant's brain are segregated using machine learning classifier-support vector machines (SVM) to utilize in operating a robotic exoskeleton hand. The maximum classification accuracy is 91.31%, using a combination of mean-slope features with an average information transfer rate (ITR) of 1.43. These results show the feasibility of a two-state MWL (fNIRS-based) robotic exoskeleton hand (BMI system) for hemiplegic patients assisting in the physical grasping tasks.

Exploring EEG Characteristics to Identify Emotional Reactions under Videogame Scenarios

In this article we present the study of electroencephalography (EEG) traits for emotion recognition process using a videogame as a stimuli tool, and considering two different kind of information related to emotions: arousal–valence self-assesses answers from participants, and game events that represented positive and negative emotional experiences under the videogame context. We performed a statistical analysis using Spearman’s correlation between the EEG traits and the emotional information. We found that EEG traits had strong correlation with arousal and valence scores; also, common EEG traits with strong correlations, belonged to the theta band of the central channels. Then, we implemented a regression algorithm with feature selection to predict arousal and valence scores using EEG traits. We achieved better result for arousal regression, than for valence regression. EEG traits selected for arousal and valence regression belonged to time domain (standard deviation, complexity, mobility, kurtosis, skewness), and frequency domain (power spectral density—PDS, and differential entropy—DE from theta, alpha, beta, gamma, and all EEG frequency spectrum). Addressing game events, we found that EEG traits related with the theta, alpha and beta band had strong correlations. In addition, distinctive event-related potentials where identified in the presence of both types of game events. Finally, we implemented a classification algorithm to discriminate between positive and negative events using EEG traits to identify emotional information. We obtained good classification performance using only two traits related with frequency domain on the theta band and on the full EEG spectrum.

A Comparative Study of Window Size and Channel Arrangement on EEG-Emotion Recognition Using Deep CNN

Emotion recognition based on electroencephalograms has become an active research area. Yet, identifying emotions using only brainwaves is still very challenging, especially the subject-independent task. Numerous studies have tried to propose methods to recognize emotions, including machine learning techniques like convolutional neural network (CNN). Since CNN has shown its potential in generalization to unseen subjects, manipulating CNN hyperparameters like the window size and electrode order might be beneficial. To our knowledge, this is the first work that extensively observed the parameter selection effect on the CNN. The temporal information in distinct window sizes was found to significantly affect the recognition performance, and CNN was found to be more responsive to changing window sizes than the support vector machine. Classifying the arousal achieved the best performance with a window size of ten seconds, obtaining 56.85% accuracy and a Matthews correlation coefficient (MCC) of 0.1369. Valence recognition had the best performance with a window length of eight seconds at 73.34% accuracy and an MCC value of 0.4669. Spatial information from varying the electrode orders had a small effect on the classification. Overall, valence results had a much more superior performance than arousal results, which were, perhaps, influenced by features related to brain activity asymmetry between the left and right hemispheres.

Emotion Recognition From Multi-Channel EEG via Deep Forest

Recently, deep neural networks (DNNs) have been applied to emotion recognition tasks based on electroencephalography (EEG), and have achieved better performance than traditional algorithms. However, DNNs still have the disadvantages of too many hyperparameters and lots of training data. To overcome these shortcomings, in this article, we propose a method for multi-channel EEG-based emotion recognition using deep forest. First, we consider the effect of baseline signal to preprocess the raw artifact-eliminated EEG signal with baseline removal. Secondly, we construct 2 D frame sequences by taking the spatial position relationship across channels into account. Finally, 2 D frame sequences are input into the classification model constructed by deep forest that can mine the spatial and temporal information of EEG signals to classify EEG emotions. The proposed method can eliminate the need for feature extraction in traditional methods and the classification model is insensitive to hyperparameter settings, which greatly reduce the complexity of emotion recognition. To verify the feasibility of the proposed model, experiments were conducted on two public DEAP and DREAMER databases. On the DEAP database, the average accuracies reach to 97.69% and 97.53% for valence and arousal, respectively; on the DREAMER database, the average accuracies reach to 89.03%, 90.41%, and 89.89% for valence, arousal and dominance, respectively. These results show that the proposed method exhibits higher accuracy than the state-of-art methods.

Recognition of Consumer Preference by Analysis and Classification EEG Signals

Neuromarketing has gained attention to bridge the gap between conventional marketing studies and electroencephalography (EEG)-based brain-computer interface (BCI) research. It determines what customers actually want through preference prediction. The performance of EEG-based preference detection systems depends on a suitable selection of feature extraction techniques and machine learning algorithms. In this study, We examined preference detection of neuromarketing dataset using different feature combinations of EEG indices and different algorithms for feature extraction and classification. For EEG feature extraction, we employed discrete wavelet transform (DWT) and power spectral density (PSD), which were utilized to measure the EEG-based preference indices that enhance the accuracy of preference detection. Moreover, we compared deep learning with other traditional classifiers, such as k-nearest neighbor (KNN), support vector machine (SVM), and random forest (RF). We also studied the effect of preference indicators on the performance of classification algorithms. Through rigorous offline analysis, we investigated the computational intelligence for preference detection and classification. The performance of the proposed deep neural network (DNN) outperforms KNN and SVM in accuracy, precision, and recall; however, RF achieved results similar to those of the DNN for the same dataset.

Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

A fascinating challenge in the field of human-robot interaction is the possibility to endow robots with emotional intelligence in order to make the interaction more intuitive, genuine, and natural. To achieve this, a critical point is the capability of the robot to infer and interpret human emotions. Emotion recognition has been widely explored in the broader fields of human-machine interaction and affective computing. Here, we report recent advances in emotion recognition, with particular regard to the human-robot interaction context. Our aim is to review the state of the art of currently adopted emotional models, interaction modalities, and classification strategies and offer our point of view on future developments and critical issues. We focus on facial expressions, body poses and kinematics, voice, brain activity, and peripheral physiological responses, also providing a list of available datasets containing data from these modalities.

EEG Fingerprints under Naturalistic Viewing Using a Portable Device

The electroencephalogram (EEG) has been proven to be a promising technique for personal identification and verification. Recently, the aperiodic component of the power spectrum was shown to outperform other commonly used EEG features. Beyond that, EEG characteristics may capture relevant features related to emotional states. In this work, we aim to understand if the aperiodic component of the power spectrum, as shown for resting-state experimental paradigms, is able to capture EEG-based subject-specific features in a naturalistic stimuli scenario. In order to answer this question, we performed an analysis using two freely available datasets containing EEG recordings from participants during viewing of film clips that aim to trigger different emotional states. Our study confirms that the aperiodic components of the power spectrum, as evaluated in terms of offset and exponent parameters, are able to detect subject-specific features extracted from the scalp EEG. In particular, our results show that the performance of the system was significantly higher for the film clip scenario if compared with resting-state, thus suggesting that under naturalistic stimuli it is even easier to identify a subject. As a consequence, we suggest a paradigm shift, from task-based or resting-state to naturalistic stimuli, when assessing the performance of EEG-based biometric systems.

Predicting Stimulus Modality and Working Memory Load During Visual- and Audiovisual-Acquired Equivalence Learning

Scholars have extensively studied the electroencephalography (EEG) correlates of associative working memory (WM) load. However, the effect of stimulus modality on EEG patterns within this process is less understood. To fill this research gap, the present study re-analyzed EEG datasets recorded during visual and audiovisual equivalence learning tasks from earlier studies. The number of associations required to be maintained (WM load) in WM was increased using the staircase method during the acquisition phase of the tasks. The support vector machine algorithm was employed to predict WM load and stimulus modality using the power, phase connectivity, and cross-frequency coupling (CFC) values obtained during time segments with different WM loads in the visual and audiovisual tasks. A high accuracy (>90%) in predicting stimulus modality based on power spectral density and from the theta-beta CFC was observed. However, accuracy in predicting WM load was higher (≥75% accuracy) than that in predicting stimulus modality (which was at chance level) using theta and alpha phase connectivity. Under low WM load conditions, this connectivity was highest between the frontal and parieto-occipital channels. The results validated our findings from earlier studies that dissociated stimulus modality based on power-spectra and CFC during equivalence learning. Furthermore, the results emphasized the importance of alpha and theta frontoparietal connectivity in WM load.