Generative Adversarial Networks-Based Data Augmentation for Brain-Computer Interface

On the role of generative artificial intelligence in the development of brain-computer interfaces

Since their inception more than 50 years ago, Brain-Computer Interfaces (BCIs) have held promise to compensate for functions lost by people with disabilities through allowing direct communication between the brain and external devices. While research throughout the past decades has demonstrated the feasibility of BCI to act as a successful assistive technology, the widespread use of BCI outside the lab is still beyond reach. This can be attributed to a number of challenges that need to be addressed for BCI to be of practical use including limited data availability, limited temporal and spatial resolutions of brain signals recorded non-invasively and inter-subject variability. In addition, for a very long time, BCI development has been mainly confined to specific simple brain patterns, while developing other BCI applications relying on complex brain patterns has been proven infeasible. Generative Artificial Intelligence (GAI) has recently emerged as an artificial intelligence domain in which trained models can be used to generate new data with properties resembling that of available data. Given the enhancements observed in other domains that possess similar challenges to BCI development, GAI has been recently employed in a multitude of BCI development applications to generate synthetic brain activity; thereby, augmenting the recorded brain activity. Here, a brief review of the recent adoption of GAI techniques to overcome the aforementioned BCI challenges is provided demonstrating the enhancements achieved using GAI techniques in augmenting limited EEG data, enhancing the spatiotemporal resolution of recorded EEG data, enhancing cross-subject performance of BCI systems and implementing end-to-end BCI applications. GAI could represent the means by which BCI would be transformed into a prevalent assistive technology, thereby improving the quality of life of people with disabilities, and helping in adopting BCI as an emerging human-computer interaction technology for general use.

Recruiting neural field theory for data augmentation in a motor imagery brain-computer interface

We introduce a novel approach to training data augmentation in brain-computer interfaces (BCIs) using neural field theory (NFT) applied to EEG data from motor imagery tasks. BCIs often suffer from limited accuracy due to a limited amount of training data. To address this, we leveraged a corticothalamic NFT model to generate artificial EEG time series as supplemental training data. We employed the BCI competition IV '2a' dataset to evaluate this augmentation technique. For each individual, we fitted the model to common spatial patterns of each motor imagery class, jittered the fitted parameters, and generated time series for data augmentation. Our method led to significant accuracy improvements of over 2% in classifying the "total power" feature, but not in the case of the "Higuchi fractal dimension" feature. This suggests that the fit NFT model may more favorably represent one feature than the other. These findings pave the way for further exploration of NFT-based data augmentation, highlighting the benefits of biophysically accurate artificial data.

A Data Augmentation Method for Motor Imagery EEG Signals Based on DCGAN-GP Network

Motor imagery electroencephalography (EEG) signals have garnered attention in brain-computer interface (BCI) research due to their potential in promoting motor rehabilitation and control. However, the limited availability of labeled data poses challenges for training robust classifiers. In this study, we propose a novel data augmentation method utilizing an improved Deep Convolutional Generative Adversarial Network with Gradient Penalty (DCGAN-GP) to address this issue. We transformed raw EEG signals into two-dimensional time-frequency maps and employed a DCGAN-GP network to generate synthetic time-frequency representations resembling real data. Validation experiments were conducted on the BCI IV 2b dataset, comparing the performance of classifiers trained with augmented and unaugmented data. Results demonstrated that classifiers trained with synthetic data exhibit enhanced robustness across multiple subjects and achieve higher classification accuracy. Our findings highlight the effectiveness of utilizing a DCGAN-GP-generated synthetic EEG data to improve classifier performance in distinguishing different motor imagery tasks. Thus, the proposed data augmentation method based on a DCGAN-GP offers a promising avenue for enhancing BCI system performance, overcoming data scarcity challenges, and bolstering classifier robustness, thereby providing substantial support for the broader adoption of BCI technology in real-world applications.

Subject-independent meta-learning framework towards optimal training of EEG-based classifiers

Advances in deep learning have shown great promise towards the application of performing high-accuracy Electroencephalography (EEG) signal classification in a variety of tasks. However, many EEG-based datasets are often plagued by the issue of high inter-subject signal variability. Robust deep learning models are notoriously difficult to train under such scenarios, often leading to subpar or widely varying performance across subjects under the leave-one-subject-out paradigm. Recently, the model agnostic meta-learning framework was introduced as a way to increase the model's ability to generalize towards new tasks. While the original framework focused on task-based meta-learning, this research aims to show that the meta-learning methodology can be modified towards subject-based signal classification while maintaining the same task objectives and achieve state-of-the-art performance. Namely, we propose the novel implementation of a few/zero-shot subject-independent meta-learning framework towards multi-class inner speech and binary class motor imagery classification. Compared to current subject-adaptive methods which utilize large number of labels from the target, the proposed framework shows its effectiveness in training zero-calibration and few-shot models for subject-independent EEG classification. The proposed few/zero-shot subject-independent meta-learning mechanism performs well on both small and large datasets and achieves robust, generalized performance across subjects. The results obtained shows a significant improvement over the current state-of-the-art, with the binary class motor imagery achieving 88.70% and the accuracy of multi-class inner speech achieving an average of 31.15%. Codes will be made available to public upon publication.

Data augmentation for invasive brain-computer interfaces based on stereo-electroencephalography (SEEG)

Objective.Deep learning is increasingly used for brain-computer interfaces (BCIs). However, the quantity of available data is sparse, especially for invasive BCIs. Data augmentation (DA) methods, such as generative models, can help to address this sparseness. However, all the existing studies on brain signals were based on convolutional neural networks and ignored the temporal dependence. This paper attempted to enhance generative models by capturing the temporal relationship from a time-series perspective.Approach. A conditional generative network (conditional transformer-based generative adversarial network (cTGAN)) based on the transformer model was proposed. The proposed method was tested using a stereo-electroencephalography (SEEG) dataset which was recorded from eight epileptic patients performing five different movements. Three other commonly used DA methods were also implemented: noise injection (NI), variational autoencoder (VAE), and conditional Wasserstein generative adversarial network with gradient penalty (cWGANGP). Using the proposed method, the artificial SEEG data was generated, and several metrics were used to compare the data quality, including visual inspection, cosine similarity (CS), Jensen-Shannon distance (JSD), and the effect on the performance of a deep learning-based classifier.Main results. Both the proposed cTGAN and the cWGANGP methods were able to generate realistic data, while NI and VAE outputted inferior samples when visualized as raw sequences and in a lower dimensional space. The cTGAN generated the best samples in terms of CS and JSD and outperformed cWGANGP significantly in enhancing the performance of a deep learning-based classifier (each of them yielding a significant improvement of 6% and 3.4%, respectively).Significance. This is the first time that DA methods have been applied to invasive BCIs based on SEEG. In addition, this study demonstrated the advantages of the model that preserves the temporal dependence from a time-series perspective.

Noise-Factorized Disentangled Representation Learning for Generalizable Motor Imagery EEG Classification

Motor Imagery (MI) Electroencephalography (EEG) is one of the most common Brain-Computer Interface (BCI) paradigms that has been widely used in neural rehabilitation and gaming. Although considerable research efforts have been dedicated to developing MI EEG classification algorithms, they are mostly limited in handling scenarios where the training and testing data are not from the same subject or session. Such poor generalization capability significantly limits the realization of BCI in real-world applications. In this paper, we proposed a novel framework to disentangle the representation of raw EEG data into three components, subject/session-specific, MI-task-specific, and random noises, so that the subject/session-specific feature extends the generalization capability of the system. This is realized by a joint discriminative and generative framework, supported by a series of fundamental training losses and training strategies. We evaluated our framework on three public MI EEG datasets, and detailed experimental results show that our method can achieve superior performance by a large margin compared to current state-of-the-art benchmark algorithms.

EEG dataset for energy data visualizations

User behavior plays a substantial role in shaping household energy use. Nevertheless, the methodologies employed by researchers to examine user behavior exhibit certain limitations in terms of their reach. The present article introduces an openly accessible collection of electroencephalography (EEG) recordings, comprising EEG data collected from individuals who were subjected to energy data visualizations. The dataset comprises EEG recordings obtained from 28 individuals who were in good health. The EEG recordings were collected using a 32-channel EMOTIV EEG device, and the international 10-20 electrode system was employed for precise electrode placement. The energy data visualizations were generated and showcased utilizing the PsychoPy software. To ascertain the participants' affective state, they were requested to rate the valence and arousal of each stimulus through the utilization of a self-assessment manikin (SAM). Additionally, three inquiries were posed for every stimulation. The dataset includes both original data visualizations and ratings. Additionally, the raw EEG data has been divided into segments consisting of data visualizations and neutral images, with the use of event markers, in order to assist analysis. The EEG recordings were recorded and stored utilizing the EMOTIVPro application, whereas the subjective reactions were captured and preserved using the PsychoPy application. Furthermore, the generation of synthetic EEG data is accomplished by employing the Generative Adversarial Network (GAN) architecture on the acquired EEG dataset. The synthetic EEG data created is integrated with empirical EEG data, and afterwards subjected to qualitative and quantitative analysis in order to improve performance. The dataset presented herein showcases a pioneering utilization of EEG investigation and offers a valuable foundation for scholars in the domains of computer science, energy conservation, artificial intelligence, brain-computer interfaces, and human-computer interaction.

Deep Unsupervised Representation Learning for Feature-Informed EEG Domain Extraction

In electroencephalography (EEG) classification paradigms, data from a target subject is often difficult to obtain, leading to difficulties in training a robust deep learning network. Transfer learning and their variations are effective tools in improving such models suffering from lack of data. However, many of the proposed variations and deep models often rely on a single assumed distribution to represent the latent features which may not scale well due to inter- and intra-subject variations in signals. This leads to significant instability in individual subject decoding performances. The presence of non-trivial domain differences between different sets of training or transfer learning data causes poorer model generalization towards the target subject. However, the detection of these domain differences is often difficult to perform due to the ill-defined nature of the EEG domain features. This study proposes a novel inference model, the Joint Embedding Variational Autoencoder, that offers conditionally tighter approximation of the estimated spatiotemporal feature distribution through the use of jointly optimised variational autoencoders to achieve optimizable data dependent inputs as an additional variable for improved overall model optimisation and scaling without sacrificing model tightness. To learn the variational bound, we show that maximising the marginal log-likelihood of only the second embedding section is required to achieve conditionally tighter lower bounds. Furthermore, we show that this model provides state-of-the-art EEG data reconstruction and deep feature extraction. The extracted domains of the EEG signals across each subject displays the rationale as to why there exists disparity between subjects' adaptation efficacy.

Data Augmentation in Defect Detection of Sanitary Ceramics in Small and Non-i.i.d Datasets

In this study, a data-augmentation method is proposed to narrow the significant difference between the distribution of training and test sets when small sample sizes are concerned. Two major obstacles exist in the process of defect detection on sanitary ceramics. The first results from the high cost of sample collection, namely, the difficulty in obtaining a large number of training images required by deep-learning algorithms, which limits the application of existing algorithms in sanitary-ceramic defect detection. Second, due to the limitation of production processes, the collected defect images are often marked, thereby resulting in great differences in distribution compared with the images of test sets, which further affects the performance of detect-detection algorithms. The lack of training data and the differences in distribution between training and test sets lead to the fact that existing deep learning-based algorithms cannot be used directly in the defect detection of sanitary ceramics. The method proposed in this study, which is based on a generative adversarial network and the Gaussian mixture model, can effectively increase the number of training samples and reduce distribution differences between training and test sets, and the features of the generated images can be controlled to a certain extent. By applying this method, the accuracy is improved from approximately 75% to nearly 90% in almost all experiments on different classification networks.

Generation of synthetic EEG data for training algorithms supporting the diagnosis of major depressive disorder

Introduction

Major depressive disorder (MDD) is the most common mental disorder worldwide, leading to impairment in quality and independence of life. Electroencephalography (EEG) biomarkers processed with machine learning (ML) algorithms have been explored for objective diagnoses with promising results. However, the generalizability of those models, a prerequisite for clinical application, is restricted by small datasets. One approach to train ML models with good generalizability is complementing the original with synthetic data produced by generative algorithms. Another advantage of synthetic data is the possibility of publishing the data for other researchers without risking patient data privacy. Synthetic EEG time-series have not yet been generated for two clinical populations like MDD patients and healthy controls.

Methods

We first reviewed 27 studies presenting EEG data augmentation with generative algorithms for classification tasks, like diagnosis, for the possibilities and shortcomings of recent methods. The subsequent empirical study generated EEG time-series based on two public datasets with 30/28 and 24/29 subjects (MDD/controls). To obtain baseline diagnostic accuracies, convolutional neural networks (CNN) were trained with time-series from each dataset. The data were synthesized with generative adversarial networks (GAN) consisting of CNNs. We evaluated the synthetic data qualitatively and quantitatively and finally used it for re-training the diagnostic model.

Results

The reviewed studies improved their classification accuracies by between 1 and 40% with the synthetic data. Our own diagnostic accuracy improved up to 10% for one dataset but not significantly for the other. We found a rich repertoire of generative models in the reviewed literature, solving various technical issues. A major shortcoming in the field is the lack of meaningful evaluation metrics for synthetic data. The few studies analyzing the data in the frequency domain, including our own, show that only some features can be produced truthfully.

Discussion

The systematic review combined with our own investigation provides an overview of the available methods for generating EEG data for a classification task, their possibilities, and shortcomings. The approach is promising and the technical basis is set. For a broad application of these techniques in neuroscience research or clinical application, the methods need fine-tuning facilitated by domain expertise in (clinical) EEG research.

BNLoop-GAN: a multi-loop generative adversarial model on brain network learning to classify Alzheimer's disease

Recent advancements in AI, big data analytics, and magnetic resonance imaging (MRI) have revolutionized the study of brain diseases such as Alzheimer's Disease (AD). However, most AI models used for neuroimaging classification tasks have limitations in their learning strategies, that is batch training without the incremental learning capability. To address such limitations, the systematic Brain Informatics methodology is reconsidered to realize evidence combination and fusion computing with multi-modal neuroimaging data through continuous learning. Specifically, we introduce the BNLoop-GAN (Loop-based Generative Adversarial Network for Brain Network) model, utilizing multiple techniques such as conditional generation, patch-based discrimination, and Wasserstein gradient penalty to learn the implicit distribution of brain networks. Moreover, a multiple-loop-learning algorithm is developed to combine evidence with better sample contribution ranking during training processes. The effectiveness of our approach is demonstrated through a case study on the classification of individuals with AD and healthy control groups using various experimental design strategies and multi-modal brain networks. The BNLoop-GAN model with multi-modal brain networks and multiple-loop-learning can improve classification performance.

MultiResUNet3+: A Full-Scale Connected Multi-Residual UNet Model to Denoise Electrooculogram and Electromyogram Artifacts from Corrupted Electroencephalogram Signals

Electroencephalogram (EEG) signals immensely suffer from several physiological artifacts, including electrooculogram (EOG), electromyogram (EMG), and electrocardiogram (ECG) artifacts, which must be removed to ensure EEG's usability. This paper proposes a novel one-dimensional convolutional neural network (1D-CNN), i.e., MultiResUNet3+, to denoise physiological artifacts from corrupted EEG. A publicly available dataset containing clean EEG, EOG, and EMG segments is used to generate semi-synthetic noisy EEG to train, validate and test the proposed MultiResUNet3+, along with four other 1D-CNN models (FPN, UNet, MCGUNet, LinkNet). Adopting a five-fold cross-validation technique, all five models' performance is measured by estimating temporal and spectral percentage reduction in artifacts, temporal and spectral relative root mean squared error, and average power ratio of each of the five EEG bands to whole spectra. The proposed MultiResUNet3+ achieved the highest temporal and spectral percentage reduction of 94.82% and 92.84%, respectively, in EOG artifacts removal from EOG-contaminated EEG. Moreover, compared to the other four 1D-segmentation models, the proposed MultiResUNet3+ eliminated 83.21% of the spectral artifacts from the EMG-corrupted EEG, which is also the highest. In most situations, our proposed model performed better than the other four 1D-CNN models, evident by the computed performance evaluation metrics.

Generative adversarial networks in EEG analysis: an overview

Electroencephalogram (EEG) signals have been utilized in a variety of medical as well as engineering applications. However, one of the challenges associated with recording EEG data is the difficulty of recording large amounts of data. Consequently, data augmentation is a potential solution to overcome this challenge in which the objective is to increase the amount of data. Inspired by the success of Generative Adversarial Networks (GANs) in image processing applications, generating artificial EEG data from the limited recorded data using GANs has seen recent success. This article provides an overview of various techniques and approaches of GANs for augmenting EEG signals. We focus on the utility of GANs in different applications including Brain-Computer Interface (BCI) paradigms such as motor imagery and P300-based systems, in addition to emotion recognition, epileptic seizures detection and prediction, and various other applications. We address in this article how GANs have been used in each study, the impact of using GANs on the model performance, the limitations of each algorithm, and future possibilities for developing new algorithms. We emphasize the utility of GANs in augmenting the limited EEG data typically available in the studied applications.

Towards Generating Realistic Wrist Pulse Signals Using Enhanced One Dimensional Wasserstein GAN

For the past several years, there has been an increasing focus on deep learning methods applied into computational pulse diagnosis. However, one factor restraining its development lies in the small wrist pulse dataset, due to privacy risks or lengthy experiments cost. In this study, for the first time, we address the challenging by presenting a novel one-dimension generative adversarial networks (GAN) for generating wrist pulse signals, which manages to learn a mapping strategy from a random noise space to the original wrist pulse data distribution automatically. Concretely, Wasserstein GAN with gradient penalty (WGAN-GP) is employed to alleviate the mode collapse problem of vanilla GANs, which could be able to further enhance the performance of the generated pulse data. We compared our proposed model performance with several typical GAN models, including vanilla GAN, deep convolutional GAN (DCGAN) and Wasserstein GAN (WGAN). To verify the feasibility of the proposed algorithm, we trained our model with a dataset of real recorded wrist pulse signals. In conducted experiments, qualitative visual inspection and several quantitative metrics, such as maximum mean deviation (MMD), sliced Wasserstein distance (SWD) and percent root mean square difference (PRD), are examined to measure performance comprehensively. Overall, WGAN-GP achieves the best performance and quantitative results show that the above three metrics can be as low as 0.2325, 0.0112 and 5.8748, respectively. The positive results support that generating wrist pulse data from a small ground truth is possible. Consequently, our proposed WGAN-GP model offers a potential innovative solution to address data scarcity challenge for researchers working with computational pulse diagnosis, which are expected to improve the performance of pulse diagnosis algorithms in the future.

Status of deep learning for EEG-based brain-computer interface applications

In the previous decade, breakthroughs in the central nervous system bioinformatics and computational innovation have prompted significant developments in brain-computer interface (BCI), elevating it to the forefront of applied science and research. BCI revitalization enables neurorehabilitation strategies for physically disabled patients (e.g., disabled patients and hemiplegia) and patients with brain injury (e.g., patients with stroke). Different methods have been developed for electroencephalogram (EEG)-based BCI applications. Due to the lack of a large set of EEG data, methods using matrix factorization and machine learning were the most popular. However, things have changed recently because a number of large, high-quality EEG datasets are now being made public and used in deep learning-based BCI applications. On the other hand, deep learning is demonstrating great prospects for solving complex relevant tasks such as motor imagery classification, epileptic seizure detection, and driver attention recognition using EEG data. Researchers are doing a lot of work on deep learning-based approaches in the BCI field right now. Moreover, there is a great demand for a study that emphasizes only deep learning models for EEG-based BCI applications. Therefore, we introduce this study to the recent proposed deep learning-based approaches in BCI using EEG data (from 2017 to 2022). The main differences, such as merits, drawbacks, and applications are introduced. Furthermore, we point out current challenges and the directions for future studies. We argue that this review study will help the EEG research community in their future research.

Multi-source online transfer algorithm based on source domain selection for EEG classification

The non-stationary nature of electroencephalography (EEG) signals and individual variability makes it challenging to obtain EEG signals from users by utilizing brain-computer interface techniques. Most of the existing transfer learning methods are based on batch learning in offline mode, which cannot adapt well to the changes generated by EEG signals in the online situation. To address this problem, a multi-source online migrating EEG classification algorithm based on source domain selection is proposed in this paper. By utilizing a small number of labeled samples from the target domain, the source domain selection method selects the source domain data similar to the target data from multiple source domains. After training a classifier for each source domain, the proposed method adjusts the weight coefficients of each classifier according to the prediction results to avoid the negative transfer problem. This algorithm was applied to two publicly available motor imagery EEG datasets, namely, BCI Competition Ⅳ Dataset Ⅱa and BNCI Horizon 2020 Dataset 2, and it achieved average accuracies of 79.29 and 70.86%, respectively, which are superior to those of several multi-source online transfer algorithms, confirming the effectiveness of the proposed algorithm.

Staging study of single-channel sleep EEG signals based on data augmentation

Introduction

Accurate sleep staging is an essential basis for sleep quality assessment and plays an important role in sleep quality research. However, the occupancy of different sleep stages is unbalanced throughout the sleep process, which makes the EEG datasets of different sleep stages have a class imbalance, which will eventually affect the automatic assessment of sleep stages.

Method

In this paper, we propose a Residual Dense Block and Deep Convolutional Generative Adversarial Network (RDB-DCGAN) data augmentation model based on the DCGAN and RDB, which takes two-dimensional continuous wavelet time-frequency maps as input, expands the minority class of sleep EEG data and later performs sleep staging by Convolutional Neural Network (CNN).

Results and discussion

The results of the CNN classification comparison test with the publicly available dataset Sleep-EDF show that the overall sleep staging accuracy of each stage after data augmentation is improved by 6%, especially the N1 stage, which has low classification accuracy due to less original data, also has a significant improvement of 19%. It is fully verified that data augmentation by improving the DCGAN model can effectively improve the classification problem of the class imbalance sleep dataset.

A Novel Image-Based Diagnosis Method Using Improved DCGAN for Rotating Machinery

Rotating machinery plays an important role in industrial systems, and faults in the machinery may damage the system health. A novel image-based diagnosis method using improved deep convolutional generative adversarial networks (DCGAN) is proposed for the feature recognition and fault classification of rotating machinery. First, vibration signal data from the rotating machinery is transformed into time-frequency feature 2-D image data by a continuous wavelet transform and used for fault classification with the neural network method. The adaptive deep convolution neural network (ADCNN) is then combined with the generative adversarial networks (GANs) to improve the performance of the feature self-learning ability from input data. Compared with different fault diagnosis methods, the proposed method has better performance for image feature classification in rotating machinery.

Closed-loop motor imagery EEG simulation for brain-computer interfaces

In a brain-computer interface (BCI) system, the testing of decoding algorithms, tasks, and their parameters is critical for optimizing performance. However, conducting human experiments can be costly and time-consuming, especially when investigating broad sets of parameters. Attempts to utilize previously collected data in offline analysis lack a co-adaptive feedback loop between the system and the user present online, limiting the applicability of the conclusions obtained to real-world uses of BCI. As such, a number of studies have attempted to address this cost-wise middle ground between offline and live experimentation with real-time neural activity simulators. We present one such system which generates motor imagery electroencephalography (EEG) via forward modeling and novel motor intention encoding models for conducting sensorimotor rhythm (SMR)-based continuous cursor control experiments in a closed-loop setting. We use the proposed simulator with 10 healthy human subjects to test the effect of three decoder and task parameters across 10 different values. Our simulated approach produces similar statistical conclusions to those produced during parallel, paired, online experimentation, but in 55% of the time. Notably, both online and simulated experimentation expressed a positive effect of cursor velocity limit on performance regardless of subject average performance, supporting the idea of relaxing constraints on cursor gain in online continuous cursor control. We demonstrate the merits of our closed-loop motor imagery EEG simulation, and provide an open-source framework to the community for closed-loop SMR-based BCI studies in the future. All code including the simulator have been made available on GitHub.

Semi-supervised generative and discriminative adversarial learning for motor imagery-based brain-computer interface

Convolutional neural networks (CNNs), which can recognize structural/configuration patterns in data with different architectures, have been studied for feature extraction. However, challenges remain regarding leveraging advanced deep learning methods in BCIs. We focus on problems of small-sized training samples and interpretability of the learned parameters and leverages a semi-supervised generative and discriminative learning framework that effectively utilizes synthesized samples with real samples to discover class-discriminative features. Our framework learns the distributional characteristics of EEG signals in an embedding space using a generative model. By using artificially generated and real EEG signals, our framework finds class-discriminative spatio-temporal feature representations that help to correctly discriminate input EEG signals. It is noteworthy that the framework facilitates the exploitation of real, unlabeled samples to better uncover the underlying patterns inherent in a user’s EEG signals. To validate our framework, we conducted experiments comparing our method with conventional linear models by utilizing variants of three existing CNN architectures as generator networks and measuring the performance on three public datasets. Our framework exhibited statistically significant improvements over the competing methods. We investigated the learned network via activation pattern maps and visualized generated artificial samples to empirically justify the stability and neurophysiological plausibility of our model.

Seizure Prediction in EEG Signals Using STFT and Domain Adaptation

Epileptic seizure prediction is one of the most used therapeutic adjuvant strategies for drug-resistant epilepsy. Conventional approaches commonly collect training and testing samples from the same patient due to inter-individual variability. However, the challenging problem of domain shift between various subjects remains unsolved, resulting in a low conversion rate to the clinic. In this work, a domain adaptation (DA)-based model is proposed to circumvent this issue. The short-time Fourier transform (STFT) is employed to extract the time-frequency features from raw EEG data, and an autoencoder is developed to map these features into high-dimensional space. By minimizing the inter-domain distance in the embedding space, this model learns the domain-invariant information, such that the generalization ability is improved by distribution alignment. Besides, to increase the feasibility of its application, this work mimics the data distribution under the clinical sampling situation and tests the model under this condition, which is the first study that adopts the assessment strategy. Experimental results on both intracranial and scalp EEG databases demonstrate that this method can minimize the domain gap effectively compared with previous approaches.

Denoising EEG Signals for Real-World BCI Applications Using GANs

As a measure of the brain's electrical activity, electroencephalography (EEG) is the primary signal of interest for brain-computer-interfaces (BCI). BCIs offer a communication pathway between a brain and an external device, translating thought into action with suitable processing. EEG data is the most common signal source for such technologies. However, artefacts induced in BCIs in the real-world context can severely degrade their performance relative to their in-laboratory performance. In most cases, the recorded signals are so heavily corrupted by noise that they are unusable and restrict BCI's broader applicability. To realise the use of portable BCIs capable of high-quality performance in a real-world setting, we use Generative Adversarial Networks (GANs) that can adopt both supervised and unsupervised learning approaches. Although our approach is supervised, the same model can be used for unsupervised tasks such as data augmentation/imputation in the low resource setting. Exploiting recent advancements in Generative Adversarial Networks (GAN), we construct a pipeline capable of denoising artefacts from EEG time series data. In the case of denoising data, it maps noisy EEG signals to clean EEG signals, given the nature of the respective artefact. We demonstrate the capability of our network on a toy dataset and a benchmark EEG dataset developed explicitly for deep learning denoising techniques. Our datasets consist of an artificially added mains noise (50/60 Hz) artefact dataset and an open-source EEG benchmark dataset with two artificially added artefacts. Artificially inducing myogenic and ocular artefacts for the benchmark dataset allows us to present qualitative and quantitative evidence of the GANs denoising capabilities and rank it among the current gold standard deep learning EEG denoising techniques. We show the power spectral density (PSD), signal-to-noise ratio (SNR), and other classical time series similarity measures for quantitative metrics and compare our model to those previously used in the literature. To our knowledge, this framework is the first example of a GAN capable of EEG artefact removal and generalisable to more than one artefact type. Our model has provided a competitive performance in advancing the state-of-the-art deep learning EEG denoising techniques. Furthermore, given the integration of AI into wearable technology, our method would allow for portable EEG devices with less noisy and more stable brain signals.

Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN

Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Data Augmentation for Deep Neural Networks Model in EEG Classification Task: A Review

Classification of electroencephalogram (EEG) is a key approach to measure the rhythmic oscillations of neural activity, which is one of the core technologies of brain-computer interface systems (BCIs). However, extraction of the features from non-linear and non-stationary EEG signals is still a challenging task in current algorithms. With the development of artificial intelligence, various advanced algorithms have been proposed for signal classification in recent years. Among them, deep neural networks (DNNs) have become the most attractive type of method due to their end-to-end structure and powerful ability of automatic feature extraction. However, it is difficult to collect large-scale datasets in practical applications of BCIs, which may lead to overfitting or weak generalizability of the classifier. To address these issues, a promising technique has been proposed to improve the performance of the decoding model based on data augmentation (DA). In this article, we investigate recent studies and development of various DA strategies for EEG classification based on DNNs. The review consists of three parts: what kind of paradigms of EEG-based on BCIs are used, what types of DA methods are adopted to improve the DNN models, and what kind of accuracy can be obtained. Our survey summarizes the current practices and performance outcomes that aim to promote or guide the deployment of DA to EEG classification in future research and development.

A Survey on EEG Signal Processing Techniques and Machine Learning: Applications to the Neurofeedback of Autobiographical Memory Deficits in Schizophrenia

In this paper, a general overview regarding neural recording, classical signal processing techniques and machine learning classification algorithms applied to monitor brain activity is presented. Currently, several approaches classified as electrical, magnetic, neuroimaging recordings and brain stimulations are available to obtain neural activity of the human brain. Among them, non-invasive methods like electroencephalography (EEG) are commonly employed, as they can provide a high degree of temporal resolution (on the order of milliseconds) and acceptable space resolution. In addition, it is simple, quick, and does not create any physical harm or stress to patients. Concerning signal processing, once the neural signals are acquired, different procedures can be applied for feature extraction. In particular, brain signals are normally processed in time, frequency, and/or space domains. The features extracted are then used for signal classification depending on its characteristics such us the mean, variance or band power. The role of machine learning in this regard has become of key importance during the last years due to its high capacity to analyze complex amounts of data. The algorithms employed are generally classified in supervised, unsupervised and reinforcement techniques. A deep review of the most used machine learning algorithms and the advantages/drawbacks of most used methods is presented. Finally, a study of these procedures utilized in a very specific and novel research field of electroencephalography, i.e., autobiographical memory deficits in schizophrenia, is outlined.

Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion

Most of the existing laser welding process monitoring technologies focus on the detection of post-engineering defects, but in the mass production of electronic equipment, such as laser welding metal plates, the real-time identification of defect detection has more important practical significance. The data set of laser welding process is often difficult to build and there is not enough experimental data, which hinder the applications of the data-driven laser welding defect detection method. In this paper, an intelligent welding defect diagnosis method based on auxiliary classifier generative adversarial networks (ACGAN) has been proposed. Firstly, a ten-class dataset consisting of 6467 samples, was constructed, which originate from the optical and thermal sensory parameters in the welding process. A new structured ACGAN network model is proposed to generate fake data similar to the true defect feature distributions. In addition, in order to make the difference between different defects categories more obvious after data expansion, a data filtering and data purification scheme was proposed based on ensemble learning and an SVM (support vector machine), which is used to filter the bad generated data. In the experiments, the classification accuracy can reach 96.83% and 85.13%, for the CNN (convolutional neural network) algorithm model and ACGAN model, respectively. However, the accuracy can further improve to 97.86% and 98.37% for the fusion models of ACGAN-CNN and ACGAN-SVM-CNN models, respectively. The results show that ACGAN can not only be used as an algorithm model for classification, but also be used to achieve superior real-time classification and recognition through data enhancement and multi-model fusion.

EEGdenoiseNet: a benchmark dataset for deep learning solutions of EEG denoising

Objective.Deep learning (DL) networks are increasingly attracting attention across various fields, including electroencephalography (EEG) signal processing. These models provide comparable performance to that of traditional techniques. At present, however, there is a lack of well-structured and standardized datasets with specific benchmark limit the development of DL solutions for EEG denoising.Approach.Here, we present EEGdenoiseNet, a benchmark EEG dataset that is suited for training and testing DL-based denoising models, as well as for performance comparisons across models. EEGdenoiseNet contains 4514 clean EEG segments, 3400 ocular artifact segments and 5598 muscular artifact segments, allowing users to synthesize contaminated EEG segments with the ground-truth clean EEG.Main results.We used EEGdenoiseNet to evaluate denoising performance of four classical networks (a fully-connected network, a simple and a complex convolution network, and a recurrent neural network). Our results suggested that DL methods have great potential for EEG denoising even under high noise contamination.Significance.Through EEGdenoiseNet, we hope to accelerate the development of the emerging field of DL-based EEG denoising. The dataset and code are available athttps://github.com/ncclabsustech/EEGdenoiseNet.

A Survey on Deep Learning-Based Short/Zero-Calibration Approaches for EEG-Based Brain-Computer Interfaces

Brain-computer interfaces (BCIs) utilizing machine learning techniques are an emerging technology that enables a communication pathway between a user and an external system, such as a computer. Owing to its practicality, electroencephalography (EEG) is one of the most widely used measurements for BCI. However, EEG has complex patterns and EEG-based BCIs mostly involve a cost/time-consuming calibration phase; thus, acquiring sufficient EEG data is rarely possible. Recently, deep learning (DL) has had a theoretical/practical impact on BCI research because of its use in learning representations of complex patterns inherent in EEG. Moreover, algorithmic advances in DL facilitate short/zero-calibration in BCI, thereby suppressing the data acquisition phase. Those advancements include data augmentation (DA), increasing the number of training samples without acquiring additional data, and transfer learning (TL), taking advantage of representative knowledge obtained from one dataset to address the so-called data insufficiency problem in other datasets. In this study, we review DL-based short/zero-calibration methods for BCI. Further, we elaborate methodological/algorithmic trends, highlight intriguing approaches in the literature, and discuss directions for further research. In particular, we search for generative model-based and geometric manipulation-based DA methods. Additionally, we categorize TL techniques in DL-based BCIs into explicit and implicit methods. Our systematization reveals advances in the DA and TL methods. Among the studies reviewed herein, ~45% of DA studies used generative model-based techniques, whereas ~45% of TL studies used explicit knowledge transferring strategy. Moreover, based on our literature review, we recommend an appropriate DA strategy for DL-based BCIs and discuss trends of TLs used in DL-based BCIs.

Learning Common Time-Frequency-Spatial Patterns for Motor Imagery Classification

The common spatial patterns (CSP) algorithm is the most popular spatial filtering method applied to extract electroencephalogram (EEG) features for motor imagery (MI) based brain-computer interface (BCI) systems. The effectiveness of the CSP algorithm depends on optimal selection of the frequency band and time window from the EEG. Many algorithms have been designed to optimize frequency band selection for CSP, while few algorithms seek to optimize the time window. This study proposes a novel framework, termed common time-frequency-spatial patterns (CTFSP), to extract sparse CSP features from multi-band filtered EEG data in multiple time windows. Specifically, the whole MI period is first segmented into multiple subseries using a sliding time window approach. Then, sparse CSP features are extracted from multiple frequency bands in each time window. Finally, multiple support vector machine (SVM) classifiers with the Radial Basis Function (RBF) kernel are trained to identify the MI tasks and the voting result of these classifiers determines the final output of the BCI. This study applies the proposed CTFSP algorithm to three public EEG datasets (BCI competition III dataset IVa, BCI competition III dataset IIIa, and BCI competition IV dataset 1) to validate its effectiveness, compared against several other state-of-the-art methods. The experimental results demonstrate that the proposed algorithm is a promising candidate for improving the performance of MI-BCI systems.

Enhancing magnetic resonance imaging-driven Alzheimer's disease classification performance using generative adversarial learning

BACKGROUND

Generative adversarial networks (GAN) can produce images of improved quality but their ability to augment image-based classification is not fully explored. We evaluated if a modified GAN can learn from magnetic resonance imaging (MRI) scans of multiple magnetic field strengths to enhance Alzheimer's disease (AD) classification performance.

METHODS

T1-weighted brain MRI scans from 151 participants of the Alzheimer's Disease Neuroimaging Initiative (ADNI), who underwent both 1.5-Tesla (1.5-T) and 3-Tesla imaging at the same time were selected to construct a GAN model. This model was trained along with a three-dimensional fully convolutional network (FCN) using the generated images (3T*) as inputs to predict AD status. Quality of the generated images was evaluated using signal to noise ratio (SNR), Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) and Natural Image Quality Evaluator (NIQE). Cases from the Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing (AIBL, n = 107) and the National Alzheimer's Coordinating Center (NACC, n = 565) were used for model validation.

RESULTS

The 3T*-based FCN classifier performed better than the FCN model trained using the 1.5-T scans. Specifically, the mean area under curve increased from 0.907 to 0.932, from 0.934 to 0.940, and from 0.870 to 0.907 on the ADNI test, AIBL, and NACC datasets, respectively. Additionally, we found that the mean quality of the generated (3T*) images was consistently higher than the 1.5-T images, as measured using SNR, BRISQUE, and NIQE on the validation datasets.

CONCLUSION

This study demonstrates a proof of principle that GAN frameworks can be constructed to augment AD classification performance and improve image quality.

Data Augmentation: Using Channel-Level Recombination to Improve Classification Performance for Motor Imagery EEG

The quality and quantity of training data are crucial to the performance of a deep-learning-based brain-computer interface (BCI) system. However, it is not practical to record EEG data over several long calibration sessions. A promising time- and cost-efficient solution is artificial data generation or data augmentation (DA). Here, we proposed a DA method for the motor imagery (MI) EEG signal called brain-area-recombination (BAR). For the BAR, each sample was first separated into two ones (named half-sample) by left/right brain channels, and the artificial samples were generated by recombining the half-samples. We then designed two schemas (intra- and adaptive-subject schema) corresponding to the single- and multi-subject scenarios. Extensive experiments using the classifier of EEGnet were conducted on two public datasets under various training set sizes. In both schemas, the BAR method can make the EEGnet have a better performance of classification (p < 0.01). To make a comparative investigation, we selected two common DA methods (noise-added and flipping), and the BAR method beat them (p < 0.05). Further, using the proposed BAR for augmentation, EEGnet achieved up to 8.3% improvement than a typical decoding algorithm CSP-SVM (p < 0.01), note that both the models were trained on the augmented dataset. This study shows that BAR usage can significantly improve the classification ability of deep learning to MI-EEG signals. To a certain extent, it may promote the development of deep learning technology in the field of BCI.