Transformer-based hand gesture recognition from instantaneous to fused neural decomposition of high-density EMG signals

Online and Cross-User Finger Movement Pattern Recognition by Decoding Neural Drive Information from Surface Electromyogram

Cross-user variability is a well-known challenge that leads to severe performance degradation and impacts the robustness of practical myoelectric control systems. To address this issue, a novel method for myoelectric recognition of finger movement patterns is proposed by incorporating a neural decoding approach with unsupervised domain adaption (UDA) learning. In our method, the neural decoding approach is implemented by extracting microscopic features characterizing individual motor unit (MU) activities obtained from a two-stage online surface electromyogram (SEMG) decomposition. A specific deep learning model is designed and initially trained using labeled data from a set of existing users. The model can update adaptively when recognizing the movement patterns of a new user. The final movement pattern was determined by a fuzzy weighted decision strategy. SEMG signals were collected from the finger extensor muscles of 15 subjects to detect seven dexterous finger-movement patterns. The proposed method achieved a movement pattern recognition accuracy of ([Formula: see text])% over seven movements under cross-user testing scenarios, much higher than that of the conventional methods using global SEMG features. Our study presents a novel robust myoelectric pattern recognition approach at a fine-grained MU level, with wide applications in neural interface and prosthesis control.

STCNet: Spatio-Temporal Cross Network with subject-aware contrastive learning for hand gesture recognition in surface EMG

This paper introduces the Spatio-Temporal Cross Network (STCNet), a novel deep learning architecture tailored for robust hand gesture recognition in surface electromyography (sEMG) across multiple subjects. We address the challenges associated with the inter-subject variability and environmental factors such as electrode shift and muscle fatigue, which traditionally undermine the robustness of gesture recognition systems. STCNet integrates a convolutional-recurrent architecture with a spatio-temporal block that extracts features over segmented time intervals, enhancing both spatial and temporal analysis. Additionally, a rolling convolution technique designed to reflect the circular band structure of the sEMG measurement device is incorporated, thus capturing the inherent spatial relationships more effectively. We further propose a subject-aware contrastive learning framework that utilizes both subject and gesture label information to align the representation of vector space. Our comprehensive experimental evaluations demonstrate the superiority of STCNet under aggregated conditions, achieving state-of-the-art performance on benchmark datasets and effectively managing the variability among different subjects. The implemented code can be found at https://github.com/KNU-BrainAI/STCNet.

A U-Net based partial convolutional time-domain separation model to identify motor units from surface electromyographic signals in real time

This study proposed a U-Net based partial convolutional time-domain model for a real-time high-density surface electromyography (HD-sEMG) decomposition. The model combines U-Net and a separation block containing partial convolution, aiming to efficiently identify motor units (MUs) without preprocessing. The proposed U-Net based network was trained by the HD-sEMG signals with innervation pulse trains (IPTs) labels, and the results are compared between different step sizes, noises, and model structures under the sliding time window with 120 sampling points. The U-Net based model got an accuracy greater than 94 % under simulated signals and 85 % under experimental signals, and identified more MUs than the structures based on convolutional neural network (CNN) and temporal convolutional network (TCN). The average latency of the U-Net based model is only 64 ms (a window duration time plus the prediction time) under the step size 20 data in both types of signals, and can be generalized to new data at different signal-to-noise (SNR). The efficiency of the proposed model is significantly higher than traditional methods such as gCKC. Meanwhile, the accuracy of the proposed model was not significantly different from the gCKC. In addition, the performance of the network under different step sizes of the sliding time window was verified. The experimental results indicate that the U-Net based model provides an efficient framework for blind source separation (BSS) of EMG signals, expanding the application of EMG signals in neural interaction.

Screen Guided Training Does Not Capture Goal-Oriented Behaviours: Learning Myoelectric Control Mappings From Scratch Using Context Informed Incremental Learning

Human-machine interfaces based on myoelectric signals typically use screen-guided training (SGT) for model calibration, but this approach fails to capture realistic user behaviors. This study evaluates a user-in-the-loop context-informed incremental learning (CIIL) framework, comparing SGT, SGT followed by CIIL adaptation (SGT-A), and a novel zero-shot adaptation (ZS-A) CIIL approach that begins adapting with no prior training. Sixteen participants completed a Fitts' Law targeting task using these control schemes, with performance measured via online throughput and offline classification accuracy. Despite lower offline accuracy, the ZS-A model achieved the highest online throughput (1.47 ± 0.46 bits/s), significantly outperforming the SGT baseline (1.15 ± 0.37 bits/s) and reached competitive performance within 200 seconds. To further enhance control performance, a novel adaptive sigmoid-based proportional control mapping was introduced, dynamically adjusting control signals to allow precise control near neutral positions and rapid movements at higher activation levels, better aligning with natural user behaviors. These findings demonstrate that CIIL can surpass traditional SGT methods in online performance and emphasize the value of real-time user-in-the-loop data for developing adaptable and intuitive myoelectric interfaces, with implications for prosthetics, rehabilitation, and telerobotics.

Continuous Estimation of Hand Kinematics from Electromyographic Signals based on Power-and Time-Efficient Transformer Deep Learning Network

Surface Electromyographic (sEMG) signals contain motor-related information and therefore can be used for human-machine interaction (HMI). Deep learning plays an important role in extracting motor-related information from sEMG signals. However, most studies prioritize model accuracy without sufficient consideration of model efficiency, including the model size, power consumption, and the computational speed of the model. This leads to impractical power consumption, heat dissipation levels and processing time in wearable computation scenarios. Here, we propose an efficient Transformer method that employs the EMSA (Efficient Multiple Self-Attention) and pruning mechanism to improve efficiency and accuracy concurrently, when estimating finger joint angles from sEMG signals. The proposed method does not only achieve state-of-the-art accuracy but can also be deployed on wearable devices to satisfy real-time applications. We applied the proposed model on the Ninapro DB2-dataset to estimate finger joint angles during grasping tasks. RNN series models, Convolution series models, and Transformer series models were used as reference models for comparison. In addition to common model accuracy, the deployment performance of the models was tested on microprocessors, such as Intel CPU i5, Apple M1, and Raspberry Pi 4B. When tested on 38 subjects of the Ninapro DB2, the proposed model resulted in a correlation coefficient of 0.82 ± 0.04, root mean squared error (RMSE) of 10.77 ± 1.48, and normalized RMSE of 0.11 ± 0.01, which were all similar to the results achieved by the state-of-the-art (SOTA) reference methods. Further, the computational time of the proposed methods was 65.99 ms on the Raspberry Pi 4B, which outperformed all the RNN series models and the Transformer series models. The model size and the power (the minimum size and power are 0.39 MB and 2.28 w) consumption of the proposed model also outperformed that of all reference Transformer methods. These experimental results indicate that our model can maintain the accuracy of the SOTA methods while significantly improving efficiency, thus being a promising approach for real-life applications in wearable devices.

Analysis of electrode locations on limb condition effect for myoelectric pattern recognition

BACKGROUND

Gesture recognition using surface electromyography (sEMG) has garnered significant attention due to its potential for intuitive and natural control in wearable human-machine interfaces. However, ensuring robustness remains essential and is currently the primary challenge for practical applications.

METHODS

This study investigates the impact of limb conditions and analyzes the influence of electrode placement. Both static and dynamic limb conditions were examined using electrodes positioned on the wrist, elbow, and the midpoint between them. Initially, we compared classification performance across various training conditions at these three electrode locations. Subsequently, a feature space analysis was conducted to quantify the effects of limb conditions. Finally, strategies for group training and feature selection were explored to mitigate these effects.

RESULTS

The results indicate that with the state-of-the-art method, classification performance at the wrist was comparable to that at the middle position, both of which outperformed the elbow, consistent with the findings from the feature space analysis. In inter-condition classification, training under dynamic limb conditions yielded better results than training under static conditions, especially at the positions covered by dynamic training. Additionally, fast and slow movement speeds produced similar performance outcomes. To mitigate the effects of limb conditions, adding more training conditions reduced classification errors; however, this reduction plateaued after four conditions, resulting in classification errors of 22.72%, 22.65%, and 26.58% for the wrist, middle, and elbow, respectively. Feature selection further improved classification performance, reducing errors to 19.98%, 19.75%, and 27.14% at the respective electrode locations, using three optimal features derived from single-condition training.

CONCLUSIONS

The study demonstrated that the impact of limb conditions was mitigated when electrodes were placed near the wrist. Dynamic limb condition training, combined with feature optimization, proved to be an effective strategy for reducing this effect. This work contributes to enhancing the robustness of myoelectric-controlled interfaces, thereby advancing the development of wearable intelligent devices.

A Real-Time Hand Gesture Recognition System for Low-Latency HMI via Transient HD-SEMG and In-Sensor Computing

In real-time human-machine interaction (HMI) applications, hand gesture recognition (HGR) requires high accuracy with low latency. Surface electromyography (sEMG), a physiological electrical signal reflecting muscle activation, is extensively used in HMI. Recently, transient sEMG, generated during the gesture transitions, has been employed in HGR to achieve lower observational latency compared to steady-state sEMG. However, the use of long feature windows (up to 200 ms) still make it less desirable in low-latency HMI. In addition, most studies have relied on remote computing, where remote data processing and large data transfer result in high computation and network latency. In this paper, we proposed a method leveraging transient high density sEMG (HD-sEMG) and in-sensor computing to achieve low-latency HGR. An sEMG contrastive convolution network (sCCN) was proposed for HGR. The mean absolute value and its average integration were used to train the sCCN in a contrastive learning manner. In addition, all signal acquisition, data processing, and pattern recognition processes were deployed within designed sensor for in-sensor computing. Compared to the state-of-the-art study using multi-channel 200-ms transient sEMG, our proposed method achieved a comparable HGR accuracy of 0.963, and a 58% lower observational latency of only 84 ms. In-sensor computing realizes a 4 times lower computation latency of 3 ms, and significantly reduces the network latency to 2 ms. The proposed method offers a promising approach to achieving low-latency HGR without compromising accuracy. This facilitates real-time HMI in biomedical applications such as prostheses, exoskeletons, virtual reality, and video games.

Topology of surface electromyogram signals: hand gesture decoding on Riemannian manifolds

Objective. Decoding gestures from the upper limb using noninvasive surface electromyogram (sEMG) signals is of keen interest for the rehabilitation of amputees, artificial supernumerary limb augmentation, gestural control of computers, and virtual/augmented realities. We show that sEMG signals recorded across an array of sensor electrodes in multiple spatial locations around the forearm evince a rich geometric pattern of global motor unit (MU) activity that can be leveraged to distinguish different hand gestures.Approach. We demonstrate a simple technique to analyze spatial patterns of muscle MU activity within a temporal window and show that distinct gestures can be classified in both supervised and unsupervised manners. Specifically, we construct symmetric positive definite covariance matrices to represent the spatial distribution of MU activity in a time window of interest, calculated as pairwise covariance of electrical signals measured across different electrodes.Main results. This allows us to understand and manipulate multivariate sEMG timeseries on a more natural subspace-the Riemannian manifold. Furthermore, it directly addresses signal variability across individuals and sessions, which remains a major challenge in the field. sEMG signals measured at a single electrode lack contextual information such as how various anatomical and physiological factors influence the signals and how their combined effect alters the evident interaction among neighboring muscles.Significance. As we show here, analyzing spatial patterns using covariance matrices on Riemannian manifolds allows us to robustly model complex interactions across spatially distributed MUs and provides a flexible and transparent framework to quantify differences in sEMG signals across individuals. The proposed method is novel in the study of sEMG signals and its performance exceeds the current benchmarks while being computationally efficient.

Integrated block-wise neural network with auto-learning search framework for finger gesture recognition using sEMG signals

Accurate finger gesture recognition with surface electromyography (sEMG) is essential and long-challenge in the muscle-computer interface, and many high-performance deep learning models have been developed to predict gestures. For these models, problem-specific tuning of network architecture is essential for improving the performance, yet it requires substantial knowledge of network architecture design and commitment of time and effort. This process thus imposes a major obstacle to the widespread and flexible application of modern deep learning. To address this issue, we present an auto-learning search framework (ALSF) to generate the integrated block-wised neural network (IBWNN) for sEMG-based gesture recognition. IBWNN contains several feature extraction blocks and dimensional reduction layers, and each feature extraction block integrates two sub-blocks (i.e., multi-branch convolutional block and triplet attention block). Meanwhile, ALSF generates optimal models for gesture recognition through the reinforcement learning method. The results show that the generated models yield state-of-the-art results compared to the modern popular networks on the open dataset Ninapro DB5. Moreover, compared to other networks, the generated models have fewer parameters and can be deployed in practical applications with less resource consumption.

Synthesis of sEMG Signals for Hand Gestures Using a 1DDCGAN

The emergence of modern prosthetics controlled by bio-signals has been facilitated by AI and microchip technology innovations. AI algorithms are trained using sEMG produced by muscles during contractions. The data acquisition procedure may result in discomfort and fatigue, particularly for amputees. Furthermore, prosthetic companies restrict sEMG signal exchange, limiting data-driven research and reproducibility. GANs present a viable solution to the aforementioned concerns. GANs can generate high-quality sEMG, which can be utilised for data augmentation, decrease the training time required by prosthetic users, enhance classification accuracy and ensure research reproducibility. This research proposes the utilisation of a one-dimensional deep convolutional GAN (1DDCGAN) to generate the sEMG of hand gestures. This approach involves the incorporation of dynamic time wrapping, fast Fourier transform and wavelets as discriminator inputs. Two datasets were utilised to validate the methodology, where five windows and increments were utilised to extract features to evaluate the synthesised sEMG quality. In addition to the traditional classification and augmentation metrics, two novel metrics-the Mantel test and the classifier two-sample test-were used for evaluation. The 1DDCGAN preserved the inter-feature correlations and generated high-quality signals, which resembled the original data. Additionally, the classification accuracy improved by an average of 1.21-5%.