Extracting neurophysiological signals reflecting users’ emotional and affective responses to BCI use: A systematic literature review

A fine-tuning deep residual convolutional neural network for emotion recognition based on frequency-channel matrices representation of one-dimensional electroencephalography

Emotion recognition (ER) plays a crucial role in enabling machines to perceive human emotional and psychological states, thus enhancing human-machine interaction. Recently, there has been a growing interest in ER based on electroencephalogram (EEG) signals. However, due to the noisy, nonlinear, and nonstationary properties of electroencephalography signals, developing an automatic and high-accuracy ER system is still a challenging task. In this study, a pretrained deep residual convolutional neural network model, including 17 convolutional layers and one fully connected layer with transfer learning technique in combination frequency-channel matrices (FCM) of two-dimensional data based on Welch power spectral density estimate from the one-dimensional EEG data has been proposed for improving the ER by automatically learning the underlying intrinsic features of multi-channel EEG data. The experiment result shows a mean accuracy of 93.61 ± 0.84%, a mean precision of 94.70 ± 0.60%, a mean sensitivity of 95.13 ± 1.02%, a mean specificity of 91.04 ± 1.02%, and a mean F1-score of 94.91 ± 0.68%, respectively using 5-fold cross-validation on the DEAP dataset. Meanwhile, to better explore and understand how the proposed model works, we noted that the ranking of clustering effect of FCM for the same category by employing the t-distributed stochastic neighbor embedding strategy is: softmax layer activation is the best, the middle convolutional layer activation is the second, and the early max pooling layer activation is the worst. These findings confirm the promising potential of combining deep learning approaches with transfer learning techniques and FCM for effective ER tasks.

A transfer learning-based feedback training motivates the performance of SMR-BCI

Objective. Feedback training is a practical approach to brain-computer interface (BCI) end-users learning to modulate their sensorimotor rhythms (SMRs). BCI self-regulation learning has been shown to be influenced by subjective psychological factors, such as motivation. However, few studies have taken into account the users' self-motivation as additional guidance for the cognitive process involved in BCI learning. In this study we tested a transfer learning (TL) feedback method designed to increase self-motivation by providing information about past performance.Approach. Electroencephalography (EEG) signals from the previous runs were affine transformed and displayed as points on the screen, along with the newly recorded EEG signals in the current run, giving the subjects a context for self-motivation. Subjects were asked to separate the feedback points for the current run under the display of the separability of prior training. We conducted a between-subject feedback training experiment, in which 24 healthy SMR-BCI naive subjects were trained to imagine left- and right-hand movements. The participants were provided with either TL feedback or typical cursor-bar (CB) feedback (control condition), for three sessions on separate days.Main results. The behavioral results showed an increased challenge and stable mastery confidence, suggesting that subjects' motivation grew as the feedback training went on. The EEG results showed favorable overall training effects with TL feedback in terms of the class distinctiveness and EEG discriminancy. Performance was 28.5% higher in the third session than in the first. About 41.7% of the subjects were 'learners' including not only low-performance subjects, but also good-performance subjects who might be affected by the ceiling effect. Subjects were able to control BCI with TL feedback with a higher performance of 60.5% during the last session compared to CB feedback.Significance. The present study demonstrated that the proposed TL feedback method boosted psychological engagement through the self-motivated context, and further allowed subjects to modulate SMR effectively. The proposed TL feedback method also provided an alternative to typical CB feedback.

The Effects of Word Priming on Emotion Classification from Neurological Signals

Affective states play an important role in human behavior and decision-making. In recent years, several affective brain-computer interface (aBCI) studies have focused on developing an emotion classifier based on elicited emotions within the user. However, it is difficult to achieve consistency in elicited emotions across populations, which can lead to dataset imbalances. The experimental design presented in this paper seeks to avoid consistency issues by asking the participant to classify the emotion portrayed in images of facial expressions, rather than their own emotions. Priming is also a common technique used in psychology studies that is known to influence emotional perception. To improve participant accuracy, we investigated matching and mis-matched word priming for the facial expression images. Electro-encephalogram (EEG) data were used to generate images fed into a classifier based on the Big Transfer model, BiT-M R101x1. The primed images resulted in higher classification accuracy overall. Further, by building different classifier models for both mis-matched primed images and matching primed images, we were able to achieve classification accuracies above 90%. We also provided the classifier with the true labels of the photographs instead of the labels generated by the participants and achieved similar results. The experimental paradigm of measuring brain activity during the emotional classification of another individual provides consistently high, balanced classification accuracies.

Concerns in the Blurred Divisions between Medical and Consumer Neurotechnology

Neurotechnology has traditionally been central to the diagnosis and treatment of neurological disorders. While these devices have initially been utilized in clinical and research settings, recent advancements in neurotechnology have yielded devices that are more portable, user-friendly, and less expensive. These improvements allow laypeople to monitor their brain waves and interface their brains with external devices. Such improvements have led to the rise of wearable neurotechnology that is marketed to the consumer. While many of the consumer devices are marketed for innocuous applications, such as use in video games, there is potential for them to be repurposed for medical use. How do we manage neurotechnologies that skirt the line between medical and consumer applications and what can be done to ensure consumer safety? Here, we characterize neurotechnology based on medical and consumer applications and summarize currently marketed uses of consumer-grade wearable headsets. We lay out concerns that may arise due to the similar claims associated with both medical and consumer devices, the possibility of consumer devices being repurposed for medical uses, and the potential for medical uses of neurotechnology to influence commercial markets related to employment and self-enhancement.

Risk of Bleeding during Implant Surgery in Patients Taking Antithrombotics: A Systematic Review

The objective of this systematic review is to assess the risk of postoperative bleeding in oral surgery for implant placement in individuals taking antithrombotics (i.e., anticoagulants and/or antiplatelet agents). A literature search was performed in PubMed (MEDLINE), Web of Science, Scopus, and EMBASE databases for articles published until August 2020, with no date restriction, and manually completed. We included prospective clinical studies that provided information regarding the presence of an experimental group (i.e., implant placement), a control group (patients not under treatment with antithrombotics), and a well-established protocol for evaluating bleeding. Meta-analysis determined the risk of bleeding during the placement of implants in antithrombotic-treated patients. Of the 756 potentially eligible articles, 5 were included in the analysis with 4 ranked as high and 1 as medium quality. Antithrombotic treatment comprised the following drug classes: (1) anticoagulants: vitamin K antagonists, (2) nonvitamin K antagonist oral anticoagulants, (3) low-molecular-weight heparin, and (4) antiplatelet agents (not specified). The results suggest that the risk of bleeding is not substantially higher in antithrombotic-treated patients (odds ratio = 2.19; 95% confidence interval: 0.88-5.44, p = 0.09) compared with nontreated patients. This systematic review suggests that the absolute risk is low and there is no need to discontinue or alter the dose of the antithrombotic treatment for implant placement surgery.

Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

A fascinating challenge in the field of human-robot interaction is the possibility to endow robots with emotional intelligence in order to make the interaction more intuitive, genuine, and natural. To achieve this, a critical point is the capability of the robot to infer and interpret human emotions. Emotion recognition has been widely explored in the broader fields of human-machine interaction and affective computing. Here, we report recent advances in emotion recognition, with particular regard to the human-robot interaction context. Our aim is to review the state of the art of currently adopted emotional models, interaction modalities, and classification strategies and offer our point of view on future developments and critical issues. We focus on facial expressions, body poses and kinematics, voice, brain activity, and peripheral physiological responses, also providing a list of available datasets containing data from these modalities.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Brain computer interface based applications for training and rehabilitation of students with neurodevelopmental disorders. A literature review

The aim of this article is to explore a paradigm shift on Brain Computer Interface (BCI) research, as well as on intervention best practices for training and rehabilitation of students with neurodevelopmental disorders. Recent studies indicate that BCI devices have positive impact on students' attention skills and working memory as well as on other skills, such as visuospatial, social, imaginative and emotional abilities. BCI applications aim to emulate humans' brain and address the appropriate understanding for each student's neurodevelopmental disorders. Studies conducted to provide knowledge about BCI-based intervention applications regarding memory, attention, visuospatial, learning, collaboration, and communication, social, creative and emotional skills are highlighted. Only non-invasive BCI type of applications are being investigated based upon representative, non-exhaustive and state-of-the-art studies within the field. This article examines the progress of BCI research so far, while different BCI paradigms are investigated. BCI-based applications could successfully regulate students' cognitive abilities when used for their training and rehabilitation. Future directions to investigate BCI-based applications for training and rehabilitation of students with neurodevelopmental disorders concerning the different populations involved are discussed.

Emotion recognition with convolutional neural network and EEG-based EFDMs

Electroencephalogram (EEG), as a direct response to brain activity, can be used to detect mental states and physical conditions. Among various EEG-based emotion recognition studies, due to the non-linear, non-stationary and the individual difference of EEG signals, traditional recognition methods still have the disadvantages of complicated feature extraction and low recognition rates. Thus, this paper first proposes a novel concept of electrode-frequency distribution maps (EFDMs) with short-time Fourier transform (STFT). Residual block based deep convolutional neural network (CNN) is proposed for automatic feature extraction and emotion classification with EFDMs. Aim at the shortcomings of the small amount of EEG samples and the challenge of differences in individual emotions, which makes it difficult to construct a universal model, this paper proposes a cross-datasets emotion recognition method of deep model transfer learning. Experiments carried out on two publicly available datasets. The proposed method achieved an average classification score of 90.59% based on a short length of EEG data on SEED, which is 4.51% higher than the baseline method. Then, the pre-trained model was applied to DEAP through deep model transfer learning with a few samples, resulted an average accuracy of 82.84%. Finally, this paper adopts the gradient weighted class activation mapping (Grad-CAM) to get a glimpse of what features the CNN has learned during training from EFDMs and concludes that the high frequency bands are more favorable for emotion recognition.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment

The purpose of this work is to set up a model that can estimate the mental fatigue of users based on the fusion of relevant features extracted from Positive 300 (P300) and steady state visual evoked potentials (SSVEP) measured by electroencephalogram. To this end, an experimental protocol describes the induction of P300, SSVEP and mental workload (which leads to mental fatigue by varying time-on-task) in different scenarios where environmental artifacts are controlled (obstacles number, obstacles velocities, ambient luminosity). Ten subjects took part in the experiment (with two suffering from cerebral palsy). Their mission is to navigate along a corridor from a starting point A to a goal point B where specific flickering stimuli are introduced to perform the P300 task. On the other hand, SSVEP task is elicited thanks to 10 Hz flickering lights. Correlated features are considered as inputs to fusion block which estimates mental workload. In order to deal with uncertainties and heterogeneity of P300 and SSVEP features, Dempster-Shafer (D-S) evidential reasoning is introduced. As the goal is to assess the reliability for the estimation of mental fatigue levels, D-S is compared to multi layer perception and linear discriminant analysis. The results show that D-S globally outperforms the other classifiers (although its performance significantly decreases between healthy and palsied groups). Finally we discuss the feasibility of such a fusion proposal in real life situation.

Functional near-infrared spectroscopy-based affective neurofeedback: feedback effect, illiteracy phenomena, and whole-connectivity profiles

Background: Affective neurofeedback constitutes a suitable approach to control abnormal neural activities associated with psychiatric disorders and might consequently relief symptom severity. However, different aspects of neurofeedback remain unclear, such as its neural basis, the performance variation, the feedback effect, among others. Aim: First, we aimed to propose a functional near-infrared spectroscopy (fNIRS)-based affective neurofeedback based on the self-regulation of frontal and occipital networks. Second, we evaluated three different feedback approaches on performance: real, fixed, and random feedback. Third, we investigated different demographic, psychological, and physiological predictors of performance. Approach: Thirty-three healthy participants performed a task whereby an amorphous figure changed its shape according to the elicited affect (positive or neutral). During the task, the participants randomly received three different feedback approaches: real feedback, with no change of the classifier output; fixed feedback, keeping the feedback figure unmodified; and random feedback, where the classifier output was multiplied by an arbitrary value, causing a feedback different than expected by the subject. Then, we applied a multivariate comparison of the whole-connectivity profiles according to the affective states and feedback approaches, as well as during a pretask resting-state block, to predict performance. Results: Participants were able to control this feedback system with 70.00 % ± 24.43 % ( p < 0.01 ) of performance during the real feedback trials. No significant differences were found when comparing the average performances of the feedback approaches. However, the whole functional connectivity profiles presented significant Mahalanobis distances ( p ≪ 0.001 ) when comparing both affective states and all feedback approaches. Finally, task performance was positively correlated to the pretask resting-state whole functional connectivity ( r = 0.512 , p = 0.009 ). Conclusions: Our results suggest that fNIRS might be a feasible tool to develop a neurofeedback system based on the self-regulation of affective networks. This finding enables future investigations using an fNIRS-based affective neurofeedback in psychiatric populations. Furthermore, functional connectivity profiles proved to be a good predictor of performance and suggested an increased effort to maintain task control in the presence of feedback distractors.

Why people use and don't use technologies: Introduction to the special issue on assistive technologies for cognition/cognitive support technologies

This special issue focuses on assistive technologies for cognition/cognitive support technologies as well as the ways in which individuals are assessed and trained in their use. We provide eleven diverse articles that give information on products, why they are used and not used, and best professional practices in service provision. Our goal is to highlight a broad topic that has received limited research investigation and offer an insight into how different countries and programs are promoting access to and use of assistive technologies for cognition/cognitive support technologies.

Biological Computation Indexes of Brain Oscillations in Unattended Facial Expression Processing Based on Event-Related Synchronization/Desynchronization

Estimation of human emotions from Electroencephalogram (EEG) signals plays a vital role in affective Brain Computer Interface (BCI). The present study investigated the different event-related synchronization (ERS) and event-related desynchronization (ERD) of typical brain oscillations in processing Facial Expressions under nonattentional condition. The results show that the lower-frequency bands are mainly used to update Facial Expressions and distinguish the deviant stimuli from the standard ones, whereas the higher-frequency bands are relevant to automatically processing different Facial Expressions. Accordingly, we set up the relations between each brain oscillation and processing unattended Facial Expressions by the measures of ERD and ERS. This research first reveals the contributions of each frequency band for comprehension of Facial Expressions in preattentive stage. It also evidences that participants have emotional experience under nonattentional condition. Therefore, the user's emotional state under nonattentional condition can be recognized in real time by the ERD/ERS computation indexes of different frequency bands of brain oscillations, which can be used in affective BCI to provide the user with more natural and friendly ways.