A transformer-based deep neural network model for SSVEP classification

Reducing calibration efforts of SSVEP-BCIs by shallow fine-tuning-based transfer learning

The utilization of transfer learning (TL), particularly through pre-training and fine-tuning, in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) has substantially reduced the calibration efforts. However, commonly employed fine-tuning approaches, including end-to-end fine-tuning and last-layer fine-tuning, require data from target subjects that encompass all categories (stimuli), resulting in a time-consuming data collection process, especially in systems with numerous categories. To address this challenge, this study introduces a straightforward yet effective ShallOw Fine-Tuning (SOFT) method to substantially reduce the number of calibration categories needed for model fine-tuning, thereby further mitigating the calibration efforts for target subjects. Specifically, SOFT involves freezing the parameters of the deeper layers while updating those of the shallow layers during fine-tuning. Freezing the parameters of deeper layers preserves the model's ability to recognize semantic and high-level features across all categories, as established during pre-training. Moreover, data from different categories exhibit similar individual-specific low-level features in SSVEP-BCIs. Consequently, updating the parameters of shallow layers-responsible for processing low-level features-with data solely from partial categories enables the fine-tuned model to efficiently capture the individual-related features shared by all categories. The effectiveness of SOFT is validated using two public datasets. Comparative analysis with commonly used end-to-end and last-layer fine-tuning methods reveals that SOFT achieves higher classification accuracy while requiring fewer calibration categories. The proposed SOFT method further decreases the calibration efforts for target subjects by reducing the calibration category requirements, thereby improving the feasibility of SSVEP-BCIs for real-world applications.

Cross-session SSVEP brainprint recognition using attentive multi-sub-band depth identity embedding learning network

Brainprint recognition technology, regarded as a promising biometric technology, encounters challenges stemming from the time-varied, low signal-to-noise ratio of brain signals, such as electroencephalogram (EEG). Steady-state visual evoked potentials (SSVEP) exhibit high signal-to-noise ratio and frequency locking, making them a promising paradigm for brainprint recognition. Consequently, the extraction of time-invariant identity information from SSVEP EEG signals is essential. In this paper, we propose an Attentive Multi-sub-band Depth Identity Embedding Learning Network for stable cross-session SSVEP brainprint recognition. To address the issue of low recognition accuracy across sessions, we introduce the Sub-band Attentive Frequency mechanism, which integrates the frequency-domain relevant characteristics of the SSVEP paradigm and focuses on exploring depth-frequency identity embedding information. Also, we employ Attentive Statistic Pooling to enhance the stability of frequency domain feature distributions across sessions. Extensive experimentation and validation were conducted on two multi-session SSVEP benchmark datasets. The experimental results show that our approach outperforms other state-of-art models on 2-second samples across sessions and has the potential to serve as a benchmark in multi-subject biometric recognition systems.

Enhancing robustness of spatial filters in motor imagery based brain-computer interface via temporal learning

BACKGROUND

In motor imagery-based brain-computer interface (MI-BCI) EEG decoding, spatial filtering play a crucial role in feature extraction. Recent studies have emphasized the importance of temporal filtering for extracting discriminative features in MI tasks. While many efforts have been made to optimize feature extraction externally, stabilizing features from spatial filtering remains underexplored.

NEW METHOD

To address this problem, we propose an approach to improve the robustness of temporal features by minimizing instability in the temporal domain. Specifically, we utilize Jensen-Shannon divergence to quantify temporal instability and integrate decision variables to construct an objective function that minimizes this instability. Our method enhances the stability of variance and mean values in the extracted features, improving the identification of discriminative features and reducing the effects of instability.

RESULTS

The proposed method was applied to spatial filtering models, and tested on two publicly datasets as well as a self-collected dataset. Results demonstrate that the proposed method significantly boosts classification accuracy, confirming its effectiveness in enhancing temporal feature stability.

COMPARISON WITH EXISTING METHODS

We compared our method with spatial filtering methods, and the-state-of-the-art models. The proposed approach achieves the highest accuracy, with 92.43 % on BCI competition III IVa dataset, 84.45 % on BCI competition IV 2a dataset, and 73.18 % on self-collected dataset.

CONCLUSIONS

Enhancing the instability of temporal features contributes to improved MI-BCI performance. This not only improves classification performance but also provides a stable foundation for future advancements. The proposed method shows great potential for EEG decoding.

Data augmentation using masked principal component representation for deep learning-based SSVEP-BCIs

Objective.Data augmentation has been demonstrated to improve the classification accuracy of deep learning models in steady-state visual evoked potential-based brain-computer interfaces (BCIs), particularly when dealing with limited electroencephalography (EEG) data. However, current data augmentation methods often rely on signal-level manipulations, which may lead to significant distortion of EEG signals. To overcome this limitation, this study proposes a component-level data augmentation method called masked principal component representation (MPCR).Approach.MPCR utilizes a principal component-based reconstruction approach, integrating a random masking strategy applied to principal component representations. Specifically, certain principal components are randomly selected and set to zero, thereby introducing random perturbations in the reconstructed samples. Furthermore, reconstructing samples via linear combinations of the remaining components effectively preserves the primary inherent structure of EEG signals. By expanding the input space covered by training samples, MPCR helps the trained model learn more robust features. To validate the efficacy of MPCR, experiments are performed on two widely utilized public datasets.Main results.Experimental results indicate that MPCR substantially enhances classification accuracy across diverse deep learning models. Additionally, in comparison to various state-of-the-art data augmentation approaches, MPCR demonstrates both greater performance and high compatibility.Significance.This study proposes a simple yet effective component-level data augmentation method, contributing valuable insights for advancing data augmentation methods in EEG-based BCIs.

Spiking-PhysFormer: Camera-based remote photoplethysmography with parallel spike-driven transformer

Artificial neural networks (ANNs) can help camera-based remote photoplethysmography (rPPG) in measuring cardiac activity and physiological signals from facial videos, such as pulse wave, heart rate and respiration rate with better accuracy. However, most existing ANN-based methods require substantial computing resources, which poses challenges for effective deployment on mobile devices. Spiking neural networks (SNNs), on the other hand, hold immense potential for energy-efficient deep learning owing to their binary and event-driven architecture. To the best of our knowledge, we are the first to introduce SNNs into the realm of rPPG, proposing a hybrid neural network (HNN) model, the Spiking-PhysFormer, aimed at reducing power consumption. Specifically, the proposed Spiking-PhyFormer consists of an ANN-based patch embedding block, SNN-based transformer blocks, and an ANN-based predictor head. First, to simplify the transformer block while preserving its capacity to aggregate local and global spatio-temporal features, we design a parallel spike transformer block to replace sequential sub-blocks. Additionally, we propose a simplified spiking self-attention mechanism that omits the value parameter without compromising the model's performance. Experiments conducted on four datasets-PURE, UBFC-rPPG, UBFC-Phys, and MMPD demonstrate that the proposed model achieves a 10.1% reduction in power consumption compared to PhysFormer. Additionally, the power consumption of the transformer block is reduced by a factor of 12.2, while maintaining decent performance as PhysFormer and other ANN-based models.

RMKD: Relaxed matching knowledge distillation for short-length SSVEP-based brain-computer interfaces

Accurate decoding of electroencephalogram (EEG) signals in the shortest possible time is essential for the realization of a high-performance brain-computer interface (BCI) system based on the steady-state visual evoked potential (SSVEP). However, the degradation of decoding performance of short-length EEG signals is often unavoidable due to the reduced information, which hinders the development of BCI systems in real-world applications. In this paper, we propose a relaxed matching knowledge distillation (RMKD) method to transfer both feature-level and logit-level knowledge in a relaxed manner to improve the decoding performance of short-length EEG signals. Specifically, the long-length EEG signals and short-length EEG signals are decoded into the frequency representation by the teacher and student models, respectively. At the feature-level, the frequency-masked generation distillation is designed to improve the representation ability of student features by forcing the randomly masked student features to generate full teacher features. At the logit-level, the non-target class knowledge distillation and the inter-class relation distillation are combined to mitigate loss conflicts by imitating the distribution of non-target classes and preserve the inter-class relation in the prediction vectors of the teacher and student models. We conduct comprehensive experiments on two public SSVEP datasets in the subject-independent scenario with six different signal lengths. The extensive experimental results demonstrate that the proposed RMKD method has significantly improved the decoding performance of short-length EEG signals in SSVEP-based BCI systems.

Decoding SSVEP Via Calibration-Free TFA-Net: A Novel Network Using Time-Frequency Features

Brain-computer interfaces (BCIs) based on steady-state visual evoked potential (SSVEP) signals offer high information transfer rates and non-invasive brain-to-device connectivity, making them highly practical. In recent years, deep learning techniques, particularly convolutional neural network (CNN) architectures, have gained prominence in EEG (e.g., SSVEP) decoding because of their nonlinear modeling capabilities and autonomy from manual feature extraction. However, most studies using CNNs employ temporal signals as the input and cannot directly mine the implicit frequency information, which may cause crucial frequency details to be lost and challenges in decoding. By contrast, the prevailing supervised recognition algorithms rely on a lengthy calibration phase to enhance algorithm performance, which could impede the popularization of SSVEP based BCIs. To address these problems, this study proposes the Time-Frequency Attention Network (TFA-Net), a novel CNN model tailored for SSVEP signal decoding without the calibration phase. Additionally, we introduce the Frequency Attention and Channel Recombination modules to enhance ability of TFA-Net to infer finer frequency-wise attention and extract features efficiently from SSVEP in the time-frequency domain. Classification results on a public dataset demonstrated that the proposed TFA-Net outperforms all the compared models, achieving an accuracy of 79.00% $\pm$ 0.27% and information transfer rate of 138.82 $\pm$ 0.78 bits/min with a 1-s data length. TFA-Net represents a novel approach to SSVEP identification as well as time-frequency signal analysis, offering a calibration-free solution that enhances the generalizability and practicality of SSVEP based BCIs.

EEG-Deformer: A Dense Convolutional Transformer for Brain-Computer Interfaces

Effectively learning the temporal dynamics in electroencephalogram (EEG) signals is challenging yet essential for decoding brain activities using brain-computer interfaces (BCIs). Although Transformers are popular for their long-term sequential learning ability in the BCI field, most methods combining Transformers with convolutional neural networks (CNNs) fail to capture the coarse-to-fine temporal dynamics of EEG signals. To overcome this limitation, we introduce EEG-Deformer, which incorporates two main novel components into a CNN-Transformer: (1) a Hierarchical Coarse-to-Fine Transformer (HCT) block that integrates a Fine-grained Temporal Learning (FTL) branch into Transformers, effectively discerning coarse-to-fine temporal patterns; and (2) a Dense Information Purification (DIP) module, which utilizes multi-level, purified temporal information to enhance decoding accuracy. Comprehensive experiments on three representative cognitive tasksâcognitive attention, driving fatigue, and mental workload detectionâconsistently confirm the generalizability of our proposed EEG-Deformer, demonstrating that it either outperforms or performs comparably to existing state-of-the-art methods. Visualization results show that EEG-Deformer learns from neurophysiologically meaningful brain regions for the corresponding cognitive tasks.

Enhancing detection of SSVEPs using discriminant compacted network

Objective. Steady-state visual evoked potential-based brain-computer interfaces (SSVEP-BCIs) have gained significant attention due to their simplicity, high signal to noise ratio and high information transfer rates (ITRs). Currently, accurate detection is a critical issue for enhancing the performance of SSVEP-BCI systems.Approach.This study proposed a novel decoding method called Discriminant Compacted Network (Dis-ComNet), which exploited the advantages of both spatial filtering and deep learning (DL). Specifically, this study enhanced SSVEP features using global template alignment and discriminant spatial pattern, and then designed a compacted temporal-spatio module (CTSM) to extract finer features. The proposed method was evaluated on a self-collected high-frequency dataset, a public Benchmark dataset and a public wearable dataset.Main Results.The results showed that Dis-ComNet significantly outperformed state-of-the-art spatial filtering methods, DL methods, and other fusion methods. Remarkably, Dis-ComNet improved the classification accuracy by 3.9%, 3.5%, 3.2%, 13.3%, 17.4%, 37.5%, and 2.5% when comparing with eTRCA, eTRCA-R, TDCA, DNN, EEGnet, Ensemble-DNN, and TRCA-Net respectively in the high-frequency dataset. The achieved results were 4.7%, 4.6%, 23.6%, 52.5%, 31.7%, and 7.0% higher than those of eTRCA, eTRCA-R, DNN, EEGnet, Ensemble-DNN, and TRCA-Net, respectively, and were comparable to those of TDCA in Benchmark dataset. The accuracy of Dis-ComNet in the wearable dataset was 9.5%, 7.1%, 36.1%, 26.3%, 15.7% and 4.7% higher than eTRCA, eTRCA-R, DNN, EEGnet, Ensemble-DNN, and TRCA-Net respectively, and comparable to TDCA. Besides, our model achieved the ITRs up to 126.0 bits/min, 236.4 bits/min and 103.6 bits/min in the high-frequency, Benchmark and the wearable datasets respectively.Significance.This study develops an effective model for the detection of SSVEPs, facilitating the development of high accuracy SSVEP-BCI systems.

MDD-SSTNet: detecting major depressive disorder by exploring spectral-spatial-temporal information on resting-state electroencephalography data based on deep neural network

Major depressive disorder (MDD) is a psychiatric disorder characterized by persistent lethargy that can lead to suicide in severe cases. Hence, timely and accurate diagnosis and treatment are crucial. Previous neuroscience studies have demonstrated that major depressive disorder subjects exhibit topological brain network changes and different temporal electroencephalography (EEG) characteristics compared to healthy controls. Based on these phenomena, we proposed a novel model, termed as MDD-SSTNet, for detecting major depressive disorder by exploring spectral-spatial-temporal information from resting-state EEG with deep convolutional neural network. Firstly, MDD-SSTNet used the Sinc filter to obtain specific frequency band features from pre-processed EEG data. Secondly, two parallel branches were used to extract temporal and spatial features through convolution and other operations. Finally, the model was trained with a combined loss function of center loss and Binary Cross-Entropy Loss. Using leave-one-subject-out cross-validation on the HUSM dataset and MODMA dataset, the MDD-SSTNet model outperformed six baseline models, achieving average classification accuracies of 93.85% and 65.08%, respectively. These results indicate that MDD-SSTNet could effectively mine spatial-temporal difference information between major depressive disorder subjects and healthy control subjects, and it holds promise to provide an efficient approach for MDD detection with EEG data.

Enhancing Domain Diversity of Transfer Learning-Based SSVEP-BCIs by the Reconstruction of Channel Correlation

OBJECTIVE

The application of transfer learning, specifically pre-training and fine-tuning, in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) has been demonstrated to effectively improve the classification performance of deep learning methods with limited calibration data. However, effectively learning task-related knowledge from source domains during the pre-training phase remains challenging. To address this issue, this study proposes an effective data augmentation method called Reconstruction of Channel Correlation (RCC) to optimize the utilization of the source domain data.

METHODS

Concretely, RCC reconstructs training samples using probabilistically mixed eigenvector matrices derived from covariance matrices across source domains. This process manipulates the channel correlation of training samples, implicitly creating novel synthesized domains. By increasing the diversity of source domains, RCC aims to enhance the domain generalizability of the pre-trained model. The effectiveness of RCC is validated through subject-independent and subject-adaptive classification experiments.

RESULTS

The results of subject-independent classification demonstrate that RCC significantly improves the classification performance of the pre-trained model on unseen target subjects. Moreover, when compared to the fine-tuning process using the RCC-absent pre-trained model, the fine-tuning process using the RCC-enhanced pre-trained model yields significantly improved performance in the subject-adaptive classification.

CONCLUSION

RCC proves to enhance the performance of transfer learning by optimizing the utilization of the source domain data.

SIGNIFICANCE

The RCC-enhanced transfer learning has the potential to facilitate the practical implementation of SSVEP-BCIs in real-world scenarios.

Non-Invasive Brain-Computer Interfaces: State of the Art and Trends

Brain-computer interface (BCI) is a rapidly evolving technology that has the potential to widely influence research, clinical and recreational use. Non-invasive BCI approaches are particularly common as they can impact a large number of participants safely and at a relatively low cost. Where traditional non-invasive BCIs were used for simple computer cursor tasks, it is now increasingly common for these systems to control robotic devices for complex tasks that may be useful in daily life. In this review, we provide an overview of the general BCI framework as well as the various methods that can be used to record neural activity, extract signals of interest, and decode brain states. In this context, we summarize the current state-of-the-art of non-invasive BCI research, focusing on trends in both the application of BCIs for controlling external devices and algorithm development to optimize their use. We also discuss various open-source BCI toolboxes and software, and describe their impact on the field at large.

A portable EEG signal acquisition system and a limited-electrode channel classification network for SSVEP

Brain-computer interfaces (BCIs) have garnered significant research attention, yet their complexity has hindered widespread adoption in daily life. Most current electroencephalography (EEG) systems rely on wet electrodes and numerous electrodes to enhance signal quality, making them impractical for everyday use. Portable and wearable devices offer a promising solution, but the limited number of electrodes in specific regions can lead to missing channels and reduced BCI performance. To overcome these challenges and enable better integration of BCI systems with external devices, this study developed an EEG signal acquisition platform (Gaitech BCI) based on the Robot Operating System (ROS) using a 10-channel dry electrode EEG device. Additionally, a multi-scale channel attention selection network based on the Squeeze-and-Excitation (SE) module (SEMSCS) is proposed to improve the classification performance of portable BCI devices with limited channels. Steady-state visual evoked potential (SSVEP) data were collected using the developed BCI system to evaluate both the system and network performance. Offline data from ten subjects were analyzed using within-subject and cross-subject experiments, along with ablation studies. The results demonstrated that the SEMSCS model achieved better classification performance than the comparative reference model, even with a limited number of channels. Additionally, the implementation of online experiments offers a rational solution for controlling external devices via BCI.

Enhancing the performance of SSVEP-based BCIs by combining task-related component analysis and deep neural network

Steady-State Visually Evoked Potential (SSVEP) signals can be decoded by either a traditional machine learning algorithm or a deep learning network. Combining the two methods is expected to enhance the performance of an SSVEP-based brain-computer interface (BCI) by exploiting their advantages. However, an efficient strategy for integrating the two methods has not yet been established. To address this issue, we propose a classification framework named eTRCA + sbCNN that combines an ensemble task-related component analysis (eTRCA) algorithm and a sub-band convolutional neural network (sbCNN) for recognizing the frequency of SSVEP signals. The two models are first trained separately, then their classification score vectors are added together, and finally the frequency corresponding to the maximal summed score is decided as the frequency of SSVEP signals. The proposed framework can effectively exploit the complementarity between the two kinds of feature signals and significantly improve the classification performance of SSVEP-based BCIs. The performance of the proposed method is validated on two SSVEP BCI datasets and compared with that of eTRCA, sbCNN and other state-of-the-art models. Experimental results indicate that the proposed method significantly outperform the compared algorithms, and thus helps to promote the practical application of SSVEP- BCI systems.

A temporal-spectral fusion transformer with subject-specific adapter for enhancing RSVP-BCI decoding

The Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interface (BCI) is an efficient technology for target retrieval using electroencephalography (EEG) signals. The performance improvement of traditional decoding methods relies on a substantial amount of training data from new test subjects, which increases preparation time for BCI systems. Several studies introduce data from existing subjects to reduce the dependence of performance improvement on data from new subjects, but their optimization strategy based on adversarial learning with extensive data increases training time during the preparation procedure. Moreover, most previous methods only focus on the single-view information of EEG signals, but ignore the information from other views which may further improve performance. To enhance decoding performance while reducing preparation time, we propose a Temporal-Spectral fusion transformer with Subject-specific Adapter (TSformer-SA). Specifically, a cross-view interaction module is proposed to facilitate information transfer and extract common representations across two-view features extracted from EEG temporal signals and spectrogram images. Then, an attention-based fusion module fuses the features of two views to obtain comprehensive discriminative features for classification. Furthermore, a multi-view consistency loss is proposed to maximize the feature similarity between two views of the same EEG signal. Finally, we propose a subject-specific adapter to rapidly transfer the knowledge of the model trained on data from existing subjects to decode data from new subjects. Experimental results show that TSformer-SA significantly outperforms comparison methods and achieves outstanding performance with limited training data from new subjects. This facilitates efficient decoding and rapid deployment of BCI systems in practical use.

OS-SSVEP: One-shot SSVEP classification

It is extremely challenging to classify steady-state visual evoked potentials (SSVEPs) in scenarios characterized by a huge scarcity of calibration data where only one calibration trial is available for each stimulus target. To address this challenge, we introduce a novel approach named OS-SSVEP, which combines a dual domain cross-subject fusion network (CSDuDoFN) with the task-related and task-discriminant component analysis (TRCA and TDCA) based on data augmentation. The CSDuDoFN framework is designed to comprehensively transfer information from source subjects, while TRCA and TDCA are employed to exploit the information from the single available calibration trial of the target subject. Specifically, CSDuDoFN uses multi-reference least-squares transformation (MLST) to map data from both the source subjects and the target subject into the domain of sine-cosine templates, thereby reducing cross-subject domain gap and benefiting transfer learning. In addition, CSDuDoFN is fed with both transformed and original data, with an adequate fusion of their features occurring at different network layers. To capitalize on the calibration trial of the target subject, OS-SSVEP utilizes source aliasing matrix estimation (SAME)-based data augmentation to incorporate into the training process of the ensemble TRCA (eTRCA) and TDCA models. Ultimately, the outputs of CSDuDoFN, eTRCA, and TDCA are combined for the SSVEP classification. The effectiveness of our proposed approach is comprehensively evaluated on three publicly available SSVEP datasets, achieving the best performance on two datasets and competitive performance on the third. Further, it is worth noting that our method follows a different technical route from the current state-of-the-art (SOTA) method and the two are complementary. The performance is significantly improved when our method is combined with the SOTA method. This study underscores the potential to integrate the SSVEP-based brain-computer interface (BCI) into daily life. The corresponding source code is accessible at https://github.com/Sungden/One-shot-SSVEP-classification.

Convolutional Transformer-Based Cross Subject Model for SSVEP-Based BCI Classification

Steady-state visual evoked potential (SSVEP) is a commonly used brain-computer interface (BCI) paradigm. The performance of cross-subject SSVEP classification has a strong impact on SSVEP-BCI. This study designed a cross subject generalization SSVEP classification model based on an improved transformer structure that uses domain generalization (DG). The global receptive field of multi-head self-attention is used to learn the global generalized SSVEP temporal information across subjects. This is combined with a parallel local convolution module, designed to avoid oversmoothing the oscillation characteristics of temporal SSVEP data and better fit the feature. Moreover, to improve the cross-subject calibration-free SSVEP classification performance, an DG method named StableNet is combined with the proposed convolutional transformer structure to form the DG-Conformer method, which can eliminate spurious correlations between SSVEP discriminative information and background noise to improve cross-subject generalization. Experiments on two public datasets, Benchmark and BETA, demonstrated the outstanding performance of the proposed DG-Conformer compared with other calibration-free methods, FBCCA, tt-CCA, Compact-CNN, FB-tCNN, and SSVEPNet. Additionally, DG-Conformer outperforms the classic calibration-required algorithms eCCA, eTRCA and eSSCOR when calibration is used. An incomplete partial stimulus calibration scheme was also explored on the Benchmark dataset, and it was demonstrated to be a potential solution for further high-performance personalized SSVEP-BCI with quick calibration.

Filter banks guided correlational convolutional neural network for SSVEPs based BCI classification

Objective.In the field of steady-state visual evoked potential brain computer interfaces (SSVEP-BCIs) research, convolutional neural networks (CNNs) have gradually been proved to be an effective method. Whereas, majority works apply the frequency domain characteristics in long time window to train the network, thus lead to insufficient performance of those networks in short time window. Furthermore, only the frequency domain information for classification lacks of other task-related information.Approach.To address these issues, we propose a time-frequency domain generalized filter-bank convolutional neural network (FBCNN-G) to improve the SSVEP-BCIs classification performance. The network integrates multiple frequency information of electroencephalogram (EEG) with template and predefined prior of sine-cosine signals to perform feature extraction, which contains correlation analyses in both template and signal aspects. Then the classification is performed at the end of the network. In addition, the method proposes the use of filter banks divided into specific frequency bands as pre-filters in the network to fully consider the fundamental and harmonic frequency characteristics of the signal.Main results.The proposed FBCNN-G model is compared with other methods on the public dataset Benchmark. The results manifest that this model has higher accuracy of character recognition accuracy and information transfer rates in several time windows. Particularly, in the 0.2 s time window, the mean accuracy of the proposed method reaches62.02%±5.12%, indicating its superior performance.Significance.The proposed FBCNN-G model is critical for the exploitation of SSVEP-BCIs character recognition models.

Short-length SSVEP data extension by a novel generative adversarial networks based framework

Steady-state visual evoked potentials (SSVEPs) based brain-computer interface (BCI) has received considerable attention due to its high information transfer rate (ITR) and available quantity of targets. However, the performance of frequency identification methods heavily hinges on the amount of user calibration data and data length, which hinders the deployment in real-world applications. Recently, generative adversarial networks (GANs)-based data generation methods have been widely adopted to create synthetic electroencephalography data, holds promise to address these issues. In this paper, we proposed a GAN-based end-to-end signal transformation network for Time-window length Extension, termed as TEGAN. TEGAN transforms short-length SSVEP signals into long-length artificial SSVEP signals. Additionally, we introduced a two-stage training strategy and the LeCam-divergence regularization term to regularize the training process of GAN during the network implementation. The proposed TEGAN was evaluated on two public SSVEP datasets (a 4-class and 12-class dataset). With the assistance of TEGAN, the performance of traditional frequency recognition methods and deep learning-based methods have been significantly improved under limited calibration data. And the classification performance gap of various frequency recognition methods has been narrowed. This study substantiates the feasibility of the proposed method to extend the data length for short-time SSVEP signals for developing a high-performance BCI system. The proposed GAN-based methods have the great potential of shortening the calibration time and cutting down the budget for various real-world BCI-based applications.

Deep Ensemble Learning-Based Sensor for Flotation Froth Image Recognition

Froth flotation is a widespread and important method for mineral separation, significantly influencing the purity and quality of extracted minerals. Traditionally, workers need to control chemical dosages by observing the visual characteristics of flotation froth, but this requires considerable experience and operational skills. This paper designs a deep ensemble learning-based sensor for flotation froth image recognition to monitor actual flotation froth working conditions, so as to assist operators in facilitating chemical dosage adjustments and achieve the industrial goals of promoting concentrate grade and mineral recovery. In our approach, training and validation data on flotation froth images are partitioned in K-fold cross validation, and deep neural network (DNN) based learners are generated through pre-trained DNN models in image-enhanced training data, in order to improve their generalization and robustness. Then, a membership function utilizing the performance information of the DNN-based learners during the validation is proposed to improve the recognition accuracy of the DNN-based learners. Subsequently, a technique for order preference by similarity to an ideal solution (TOPSIS) based on the F1 score is proposed to select the most probable working condition of flotation froth images through a decision matrix composed of the DNN-based learners' predictions via a membership function, which is adopted to optimize the combination process of deep ensemble learning. The effectiveness and superiority of the designed sensor are verified in a real industrial gold-antimony froth flotation application.

MASA-TCN: Multi-Anchor Space-Aware Temporal Convolutional Neural Networks for Continuous and Discrete EEG Emotion Recognition

Emotion recognition from electroencephalogram (EEG) signals is a critical domain in biomedical research with applications ranging from mental disorder regulation to human-computer interaction. In this paper, we address two fundamental aspects of EEG emotion recognition: continuous regression of emotional states and discrete classification of emotions. While classification methods have garnered significant attention, regression methods remain relatively under-explored. To bridge this gap, we introduce MASA-TCN, a novel unified model that leverages the spatial learning capabilities of Temporal Convolutional Networks (TCNs) for EEG emotion regression and classification tasks. The key innovation lies in the introduction of a space-aware temporal layer, which empowers TCN to capture spatial relationships among EEG electrodes, enhancing its ability to discern nuanced emotional states. Additionally, we design a multi-anchor block with attentive fusion, enabling the model to adaptively learn dynamic temporal dependencies within the EEG signals. Experiments on two publicly available datasets show that MASA-TCN achieves higher results than the state-of-the-art methods for both EEG emotion regression and classification tasks.

A Convolutional Neural Network for SSVEP Identification by Using a Few-Channel EEG

The application of wearable electroencephalogram (EEG) devices is growing in brain-computer interfaces (BCI) owing to their good wearability and portability. Compared with conventional devices, wearable devices typically support fewer EEG channels. Devices with few-channel EEGs have been proven to be available for steady-state visual evoked potential (SSVEP)-based BCI. However, fewer-channel EEGs can cause the BCI performance to decrease. To address this issue, an attention-based complex spectrum-convolutional neural network (atten-CCNN) is proposed in this study, which combines a CNN with a squeeze-and-excitation block and uses the spectrum of the EEG signal as the input. The proposed model was assessed on a wearable 40-class dataset and a public 12-class dataset under subject-independent and subject-dependent conditions. The results show that whether using a three-channel EEG or single-channel EEG for SSVEP identification, atten-CCNN outperformed the baseline models, indicating that the new model can effectively enhance the performance of SSVEP-BCI with few-channel EEGs. Therefore, this SSVEP identification algorithm based on a few-channel EEG is particularly suitable for use with wearable EEG devices.

Multi-tailed vision transformer for efficient inference

Recently, Vision Transformer (ViT) has achieved promising performance in image recognition and gradually serves as a powerful backbone in various vision tasks. To satisfy the sequential input of Transformer, the tail of ViT first splits each image into a sequence of visual tokens with a fixed length. Then, the following self-attention layers construct the global relationship between tokens to produce useful representation for the downstream tasks. Empirically, representing the image with more tokens leads to better performance, yet the quadratic computational complexity of self-attention layer to the number of tokens could seriously influence the efficiency of ViT's inference. For computational reduction, a few pruning methods progressively prune uninformative tokens in the Transformer encoder, while leaving the number of tokens before the Transformer untouched. In fact, fewer tokens as the input for the Transformer encoder can directly reduce the following computational cost. In this spirit, we propose a Multi-Tailed Vision Transformer (MT-ViT) in the paper. MT-ViT adopts multiple tails to produce visual sequences of different lengths for the following Transformer encoder. A tail predictor is introduced to decide which tail is the most efficient for the image to produce accurate prediction. Both modules are optimized in an end-to-end fashion, with the Gumbel-Softmax trick. Experiments on ImageNet-1K demonstrate that MT-ViT can achieve a significant reduction on FLOPs with no degradation of the accuracy and outperform compared methods in both accuracy and FLOPs.

Non-Invasive Biosensing for Healthcare Using Artificial Intelligence: A Semi-Systematic Review

The rapid development of biosensing technologies together with the advent of deep learning has marked an era in healthcare and biomedical research where widespread devices like smartphones, smartwatches, and health-specific technologies have the potential to facilitate remote and accessible diagnosis, monitoring, and adaptive therapy in a naturalistic environment. This systematic review focuses on the impact of combining multiple biosensing techniques with deep learning algorithms and the application of these models to healthcare. We explore the key areas that researchers and engineers must consider when developing a deep learning model for biosensing: the data modality, the model architecture, and the real-world use case for the model. We also discuss key ongoing challenges and potential future directions for research in this field. We aim to provide useful insights for researchers who seek to use intelligent biosensing to advance precision healthcare.

Dynamic decomposition graph convolutional neural network for SSVEP-based brain-computer interface

The SSVEP-based paradigm serves as a prevalent approach in the realm of brain-computer interface (BCI). However, the processing of multi-channel electroencephalogram (EEG) data introduces challenges due to its non-Euclidean characteristic, necessitating methodologies that account for inter-channel topological relations. In this paper, we introduce the Dynamic Decomposition Graph Convolutional Neural Network (DDGCNN) designed for the classification of SSVEP EEG signals. Our approach incorporates layerwise dynamic graphs to address the oversmoothing issue in Graph Convolutional Networks (GCNs), employing a dense connection mechanism to mitigate the gradient vanishing problem. Furthermore, we enhance the traditional linear transformation inherent in GCNs with graph dynamic fusion, thereby elevating feature extraction and adaptive aggregation capabilities. Our experimental results demonstrate the effectiveness of proposed approach in learning and extracting features from EEG topological structure. The results shown that DDGCNN outperforms other state-of-the-art (SOTA) algorithms reported on two datasets (Dataset 1: 54 subjects, 4 targets, 2 sessions; Dataset 2: 35 subjects, 40 targets). Additionally, we showcase the implementation of DDGCNN in the context of synchronized BCI robotic fish control. This work represents a significant advancement in the field of EEG signal processing for SSVEP-based BCIs. Our proposed method processes SSVEP time domain signals directly as an end-to-end system, making it easy to deploy. The code is available at https://github.com/zshubin/DDGCNN.

Enhancing SSVEP-BCI Performance Under Fatigue State Using Dynamic Stopping Strategy

Steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) have emerged as a prominent technology due to their high information transfer rate, rapid calibration time, and robust signal-to-noise ratio. However, a critical challenge for practical applications is performance degradation caused by user fatigue during prolonged use. This work proposes novel methods to address this challenge by dynamically adjusting data acquisition length and updating detection models based on a fatigue-aware stopping strategy. Two 16-target SSVEP-BCIs were employed, one using low-frequency and the other using high-frequency stimulation. A self-recorded fatigue dataset from 24 subjects was utilized for extensive evaluation. A simulated online experiment demonstrated that the proposed methods outperform the conventional fixed stopping strategy in terms of classification accuracy, information transfer rate, and selection time, irrespective of stimulation frequency. These findings suggest that the proposed approach can significantly improve SSVEP-BCI performance under fatigue conditions, leading to superior performance during extended use.

A Novel Data Augmentation Approach Using Mask Encoding for Deep Learning-Based Asynchronous SSVEP-BCI

Deep learning (DL)-based methods have been successfully employed as asynchronous classification algorithms in the steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) system. However, these methods often suffer from the limited amount of electroencephalography (EEG) data, leading to overfitting. This study proposes an effective data augmentation approach called EEG mask encoding (EEG-ME) to mitigate overfitting. EEG-ME forces models to learn more robust features by masking partial EEG data, leading to enhanced generalization capabilities of models. Three different network architectures, including an architecture integrating convolutional neural networks (CNN) with Transformer (CNN-Former), time domain-based CNN (tCNN), and a lightweight architecture (EEGNet) are utilized to validate the effectiveness of EEG-ME on publicly available benchmark and BETA datasets. The results demonstrate that EEG-ME significantly enhances the average classification accuracy of various DL-based methods with different data lengths of time windows on two public datasets. Specifically, CNN-Former, tCNN, and EEGNet achieve respective improvements of 3.18%, 1.42%, and 3.06% on the benchmark dataset as well as 11.09%, 3.12%, and 2.81% on the BETA dataset, with the 1-second time window as an example. The enhanced performance of SSVEP classification with EEG-ME promotes the implementation of the asynchronous SSVEP-BCI system, leading to improved robustness and flexibility in human-machine interaction.

Improving Generalized Zero-Shot Learning SSVEP Classification Performance From Data-Efficient Perspective

Generalized zero-shot learning (GZSL) has significantly reduced the training requirements for steady-state visual evoked potential (SSVEP) based brain-computer interfaces (BCIs). Traditional methods require complete class data sets for training, but GZSL allows for only partial class data sets, dividing them into 'seen' (those with training data) and 'unseen' classes (those without training data). However, inefficient utilization of SSVEP data limits the accuracy and information transfer rate (ITR) of existing GZSL methods. To this end, we proposed a framework for more effective utilization of SSVEP data at three systematically combined levels: data acquisition, feature extraction, and decision-making. First, prevalent SSVEP-based BCIs overlook the inter-subject variance in visual latency and employ fixed sampling starting time (SST). We introduced a dynamic sampling starting time (DSST) strategy at the data acquisition level. This strategy uses the classification results on the validation set to find the optimal sampling starting time (OSST) for each subject. In addition, we developed a Transformer structure to capture the global information of input data for compensating the small receptive field of existing networks. The global receptive fields of the Transformer can adequately process the information from longer input sequences. For the decision-making level, we designed a classifier selection strategy that can automatically select the optimal classifier for the seen and unseen classes, respectively. We also proposed a training procedure to make the above solutions in conjunction with each other. Our method was validated on three public datasets and outperformed the state-of-the-art (SOTA) methods. Crucially, we also outperformed the representative methods that require training data for all classes.

FB-CCNN: A Filter Bank Complex Spectrum Convolutional Neural Network with Artificial Gradient Descent Optimization

The brain-computer interface (BCI) provides direct communication between human brains and machines, including robots, drones and wheelchairs, without the involvement of peripheral systems. BCI based on electroencephalography (EEG) has been applied in many fields, including aiding people with physical disabilities, rehabilitation, education and entertainment. Among the different EEG-based BCI paradigms, steady-state visual evoked potential (SSVEP)-based BCIs are known for their lower training requirements, high classification accuracy and high information transfer rate (ITR). In this article, a filter bank complex spectrum convolutional neural network (FB-CCNN) was proposed, and it achieved leading classification accuracies of 94.85 ± 6.18% and 80.58 ± 14.43%, respectively, on two open SSVEP datasets. An optimization algorithm named artificial gradient descent (AGD) was also proposed to generate and optimize the hyperparameters of the FB-CCNN. AGD also revealed correlations between different hyperparameters and their corresponding performances. It was experimentally demonstrated that FB-CCNN performed better when the hyperparameters were fixed values rather than channel number-based. In conclusion, a deep learning model named FB-CCNN and a hyperparameter-optimizing algorithm named AGD were proposed and demonstrated to be effective in classifying SSVEP through experiments. The hyperparameter design process and analysis were carried out using AGD, and advice on choosing hyperparameters for deep learning models in classifying SSVEP was provided.