Surface EMG-Based Instantaneous Hand Gesture Recognition Using Convolutional Neural Network with the Transfer Learning Method

Multimodal optimal matching and augmentation method for small sample gesture recognition

In human-computer interaction, gesture recognition based on physiological signals offers advantages such as a more natural and fast interaction mode and less constrained by the environment than visual-based. Surface electromyography-based gesture recognition has significantly progressed. However, since individuals have physical differences, researchers must collect data multiple times from each user to train the deep learning model. This data acquisition process can be particularly burdensome for non-healthy users. Researchers are currently exploring transfer learning and data augmentation techniques to enhance the accuracy of small-sample gesture recognition models. However, challenges persist, such as negative transfer and limited diversity in training samples, leading to suboptimal recognition performance. Therefore, We introduce motion information into sEMG-based recognition and propose a multimodal optimal matching and augmentation method for small sample gesture recognition, achieving efficient gesture recognition with only one acquisition per gesture. Firstly, this method utilizes the optimal matching signal selection module to select the most similar signals from the existing data to the new user as the training set, reducing inter-domain differences. Secondly, the similarity calculation augmentation module enhances the diversity of the training set. Finally, the Modal-type embedding enhances the information interaction between each mode signal. We evaluated the effectiveness on Self-collected Stroke Patient, the Ninapro DB1 dataset and the Ninapro DB5 dataset and achieved accuracies of 93.69%, 91.65% and 98.56%, respectively. These results demonstrate that the method achieved performance comparable to traditional recognition models while significantly reducing the collected data.

Sequential sEMG Recognition With Knowledge Transfer and Dynamic Graph Network Based on Spatio-Temporal Feature Extraction Network

Surface electromyography (sEMG) signals are electrical signals released by muscles during movement, which can directly reflect the muscle conditions during various actions. When a series of continuous static actions are connected along the temporal axis, a sequential action is formed, which is more aligned with people's intuitive understanding of real-life movements. The signals acquired during sequential actions are known as sequential sEMG signals, including an additional dimension of sequence, embodying richer features compared to static sEMG signals. However, existing methods show inadequate utilization of the signals' sequential characteristics. Addressing these gaps, this paper introduces the Spatio-Temporal Feature Extraction Network (STFEN), which includes a Sequential Feature Analysis Module based on static-sequential knowledge transfer, and a Spatial Feature Analysis Module based on dynamic graph networks to analyze the internal relationships between the leads. The effectiveness of STFEN is tested on both modified publicly available datasets and on our acquired Arabic Digit Sequential Electromyography (ADSE) dataset. The results show that STFEN outperforms existing models in recognizing sequential sEMG signals. Experiments have confirmed the reliability and wide applicability of STFEN in analyzing complex muscle activities. Furthermore, this work also suggests STFEN's potential benefits in rehabilitation medicine, particularly for stroke recovery, and shows promising future applications.

Hand gestures classification of sEMG signals based on BiLSTM-metaheuristic optimization and hybrid U-Net-MobileNetV2 encoder architecture

Surface electromyography (sEMG) data has been extensively utilized in deep learning algorithms for hand movement classification. This paper aims to introduce a novel method for hand gesture classification using sEMG data, addressing accuracy challenges seen in previous studies. We propose a U-Net architecture incorporating a MobileNetV2 encoder, enhanced by a novel Bidirectional Long Short-Term Memory (BiLSTM) and metaheuristic optimization for spatial feature extraction in hand gesture and motion recognition. Bayesian optimization is employed as the metaheuristic approach to optimize the BiLSTM model's architecture. To address the non-stationarity of sEMG signals, we employ a windowing strategy for signal augmentation within deep learning architectures. The MobileNetV2 encoder and U-Net architecture extract relevant features from sEMG spectrogram images. Edge computing integration is leveraged to further enhance innovation by enabling real-time processing and decision-making closer to the data source. Six standard databases were utilized, achieving an average accuracy of 90.23% with our proposed model, showcasing a 3-4% average accuracy improvement and a 10% variance reduction. Notably, Mendeley Data, BioPatRec DB3, and BioPatRec DB1 surpassed advanced models in their respective domains with classification accuracies of 88.71%, 90.2%, and 88.6%, respectively. Experimental results underscore the significant enhancement in generalizability and gesture recognition robustness. This approach offers a fresh perspective on prosthetic management and human-machine interaction, emphasizing its efficacy in improving accuracy and reducing variance for enhanced prosthetic control and interaction with machines through edge computing integration.

Leveraging Motor Unit Spatial Activation Patterns for Channel Selection in Finger Force Regression

Individual finger forces can be predicted by regression of high-density surface electromyography (sEMG) signals. This is promising for applications in human-machine interfaces, specifically prosthesis control, although the large number of electrodes imposes high computational requirements. In this study, we present strategies for a-priori channel selection guided by motor unit spatial activation patterns to reduce computational costs without compromising decoding accuracy. In contrast to subject-specific data-driven selection, we test the hypothesis that pre-selecting sEMG channels for finger-specific force estimation can still generalize across subjects. We show that a subset of 32 channels, out of a total of 256, achieves an RMSE of 6.32 ± 2.34 % of the Maximum Voluntary Contraction (MVC) on the HYSER RANDOM dataset, competitive with the state-of-the-art baseline model, using all channels, which attains an RMSE of 5.57 ± 1.94 % MVC. These results highlight the potential of simple, a-priori channel selection strategies in decoding finger forces from sEMG, which would be particularly suited for applications with limited computational resources.

Hand gesture recognition with deep residual network using Semg signal

OBJECTIVES

To design and develop a classifier, named Sewing Driving Training based Optimization-Deep Residual Network (SDTO_DRN) for hand gesture recognition.

METHODS

The electrical activity of forearm muscles generates the signals that can be captured with Surface Electromyography (sEMG) sensors and includes meaningful data for decoding both muscle actions and hand movement. This research develops an efficacious scheme for hand gesture recognition using SDTO_DRN. Here, signal pre-processing is done through Gaussian filtering. Thereafter, desired and appropriate features are extracted. Following that, effective features are chosen using SDTO. At last, hand gesture identification is accomplished based on DRN and this network is effectively fine-tuned by SDTO, which is a combination of Sewing Training Based Optimization (STBO) and Driving Training Based Optimization (DTBO). The datasets employed for the implementation of this work are MyoUP Dataset and putEMG: sEMG Gesture and Force Recognition Dataset.

RESULTS

The designed SDTO_DRN model has gained superior performance with magnificent results by delivering a maximum accuracy of 0.943, True Positive Rate (TPR) of 0.929, True Negative Rate (TNR) of 0.919, Positive Predictive Value (PPV) of 0.924, and Negative Predictive Value (NPV) of 0.924.

CONCLUSIONS

The hand gesture recognition using the proposed model is accurate and improves the effectiveness of the recognition.

Dual Stream Long Short-Term Memory Feature Fusion Classifier for Surface Electromyography Gesture Recognition

Gesture recognition using electromyography (EMG) signals has prevailed recently in the field of human-computer interactions for controlling intelligent prosthetics. Currently, machine learning and deep learning are the two most commonly employed methods for classifying hand gestures. Despite traditional machine learning methods already achieving impressive performance, it is still a huge amount of work to carry out feature extraction manually. The existing deep learning methods utilize complex neural network architectures to achieve higher accuracy, which will suffer from overfitting, insufficient adaptability, and low recognition accuracy. To improve the existing phenomenon, a novel lightweight model named dual stream LSTM feature fusion classifier is proposed based on the concatenation of five time-domain features of EMG signals and raw data, which are both processed with one-dimensional convolutional neural networks and LSTM layers to carry out the classification. The proposed method can effectively capture global features of EMG signals using a simple architecture, which means less computational cost. An experiment is conducted on a public DB1 dataset with 52 gestures, and each of the 27 subjects repeats every gesture 10 times. The accuracy rate achieved by the model is 89.66%, which is comparable to that achieved by more complex deep learning neural networks, and the inference time for each gesture is 87.6 ms, which can also be implied in a real-time control system. The proposed model is validated using a subject-wise experiment on 10 out of the 40 subjects in the DB2 dataset, achieving a mean accuracy of 91.74%. This is illustrated by its ability to fuse time-domain features and raw data to extract more effective information from the sEMG signal and select an appropriate, efficient, lightweight network to enhance the recognition results.

Cross-User Electromyography Pattern Recognition Based on a Novel Spatial-Temporal Graph Convolutional Network

With the goal of promoting the development of myoelectric control technology, this paper focuses on exploring graph neural network (GNN) based robust electromyography (EMG) pattern recognition solutions. Given that high-density surface EMG (HD-sEMG) signal contains rich temporal and spatial information, the multi-view spatial-temporal graph convolutional network (MSTGCN)is adopted as the basic classifier, and a feature extraction convolutional neural network (CNN) module is designed and integrated into MSTGCN to generate a new model called CNN-MSTGCN. The EMG pattern recognition experiments are conducted on HD-sEMG data of 17 gestures from 11 subjects. The ablation experiments show that each functional module of the proposed CNN-MSTGCN network has played a more or less positive role in improving the performance of EMG pattern recognition. The user-independent recognition experiments and the transfer learning-based cross-user recognition experiments verify the advantages of the proposed CNN-MSTGCN network in improving recognition rate and reducing user training burden. In the user-independent recognition experiments, CNN-MSTGCN achieves the recognition rate of 68%, which is significantly better than those obtained by residual network-50 (ResNet50, 47.5%, p < 0.001) and long-short-term-memory (LSTM, 57.1%, p=0.045). In the transfer learning-based cross-user recognition experiments, TL-CMSTGCN achieves an impressive recognition rate of 92.3%, which is significantly superior to both TL-ResNet50 (84.6%, p = 0.003) and TL-LSTM (85.3%, p = 0.008). The research results of this paper indicate that GNN has certain advantages in overcoming the impact of individual differences, and can be used to provide possible solutions for achieving robust EMG pattern recognition technology.

TraHGR: Transformer for Hand Gesture Recognition via Electromyography

Deep learning-based Hand Gesture Recognition (HGR) via surface Electromyogram (sEMG) signals have recently shown considerable potential for development of advanced myoelectric-controlled prosthesis. Although deep learning techniques can improve HGR accuracy compared to their classical counterparts, classifying hand movements based on sparse multichannel sEMG signals is still a challenging task. Furthermore, existing deep learning approaches, typically, include only one model as such can hardly extract representative features. In this paper, we aim to address this challenge by capitalizing on the recent advances in hybrid models and transformers. In other words, we propose a hybrid framework based on the transformer architecture, which is a relatively new and revolutionizing deep learning model. The proposed hybrid architecture, referred to as the Transformer for Hand Gesture Recognition (TraHGR), consists of two parallel paths followed by a linear layer that acts as a fusion center to integrate the advantage of each module. We evaluated the proposed architecture TraHGR based on the commonly used second Ninapro dataset, referred to as the DB2. The sEMG signals in the DB2 dataset are measured in real-life conditions from 40 healthy users, each performing 49 gestures. We have conducted an extensive set of experiments to test and validate the proposed TraHGR architecture, and compare its achievable accuracy with several recently proposed HGR classification algorithms over the same dataset. We have also compared the results of the proposed TraHGR architecture with each individual path and demonstrated the distinguishing power of the proposed hybrid architecture. The recognition accuracies of the proposed TraHGR architecture for the window of size 200ms and step size of 100ms are 86.00%, 88.72%, 81.27%, and 93.74%, which are 2.30%, 4.93%, 8.65%, and 4.20% higher than the state-of-the-art performance for DB2 (49 gestures), DB2-B (17 gestures), DB2-C (23 gestures), and DB2-D (9 gestures), respectively.

On the Distribution of Muscle Signals: A Method for Distance-Based Classification of Human Gestures

We investigate the distribution of muscle signatures of human hand gestures under Dynamic Time Warping. For this we present a k-Nearest-Neighbors classifier using Dynamic Time Warping for the distance estimate. To understand the resulting classification performance, we investigate the distribution of the recorded samples and derive a method of assessing the separability of a set of gestures. In addition to this, we present and evaluate two approaches with reduced real-time computational cost with regards to their effectiveness and the mechanics behind them. We further investigate the impact of different parameters with regards to practical usability and background rejection, allowing fine-tuning of the induced classification procedure.

Feasibility study on the application of a spiking neural network in myoelectric control systems

In recent years, the effectiveness of a spiking neural network (SNN) for Electromyography (EMG) pattern recognition has been validated, but there is a lack of comprehensive consideration of the problems of heavy training burden, poor robustness, and high energy consumption in the application of actual myoelectric control systems. In order to explore the feasibility of the application of SNN in actual myoelectric control systems, this paper investigated an EMG pattern recognition scheme based on SNN. To alleviate the differences in EMG distribution caused by electrode shifts and individual differences, the adaptive threshold encoding was applied to gesture sample encoding. To improve the feature extraction ability of SNN, the leaky-integrate-and-fire (LIF) neuron that combines voltage-current effect was adopted as a spike neuron model. To balance recognition accuracy and power consumption, experiments were designed to determine encoding parameter and LIF neuron release threshold. By conducting the gesture recognition experiments considering different training test ratios, electrode shifts, and user independences on the nine-gesture high-density and low-density EMG datasets respectively, the advantages of the proposed SNN-based scheme have been verified. Compared with a Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM) and Linear Discriminant Analysis (LDA), SNN can effectively reduce the number of repetitions in the training set, and its power consumption was reduced by 1-2 orders of magnitude. For the high-density and low-density EMG datasets, SNN improved the overall average accuracies by about (0.99 ~ 14.91%) under different training test ratios. For the high-density EMG dataset, the accuracy of SNN was improved by (0.94 ~ 13.76%) under electrode-shift condition and (3.81 ~ 18.95%) in user-independent case. The advantages of SNN in alleviating the user training burden, reducing power consumption, and improving robustness are of great significance for the implementation of user-friendly low-power myoelectric control systems.

Empowering Hand Rehabilitation with AI-Powered Gesture Recognition: A Study of an sEMG-Based System

Stroke is one of the most prevalent health issues that people face today, causing long-term complications such as paresis, hemiparesis, and aphasia. These conditions significantly impact a patient's physical abilities and cause financial and social hardships. In order to address these challenges, this paper presents a groundbreaking solution-a wearable rehabilitation glove. This motorized glove is designed to provide comfortable and effective rehabilitation for patients with paresis. Its unique soft materials and compact size make it easy to use in clinical settings and at home. The glove can train each finger individually and all fingers together, using assistive force generated by advanced linear integrated actuators controlled by sEMG signals. The glove is also durable and long-lasting, with 4-5 h of battery life. The wearable motorized glove is worn on the affected hand to provide assistive force during rehabilitation training. The key to this glove's effectiveness is its ability to perform the classified hand gestures acquired from the non-affected hand by integrating four sEMG sensors and a deep learning algorithm (the 1D-CNN algorithm and the InceptionTime algorithm). The InceptionTime algorithm classified ten hand gestures' sEMG signals with an accuracy of 91.60% and 90.09% in the training and verification sets, respectively. The overall accuracy was 90.89%. It showed potential as a tool for developing effective hand gesture recognition systems. The classified hand gestures can be used as a control command for the motorized wearable glove placed on the affected hand, allowing it to mimic the movements of the non-affected hand. This innovative technology performs rehabilitation exercises based on the theory of mirror therapy and task-oriented therapy. Overall, this wearable rehabilitation glove represents a significant step forward in stroke rehabilitation, offering a practical and effective solution to help patients recover from stroke's physical, financial, and social impact.

sEMG-Based Hand Gesture Recognition Using Binarized Neural Network

Recently, human-machine interfaces (HMI) that make life convenient have been studied in many fields. In particular, a hand gesture recognition (HGR) system, which can be implemented as a wearable system, has the advantage that users can easily and intuitively control the device. Among the various sensors used in the HGR system, the surface electromyography (sEMG) sensor is independent of the acquisition environment, easy to wear, and requires a small amount of data. Focusing on these advantages, previous sEMG-based HGR systems used several sensors or complex deep-learning algorithms to achieve high classification accuracy. However, systems that use multiple sensors are bulky, and embedded platforms with complex deep-learning algorithms are difficult to implement. To overcome these limitations, we propose an HGR system using a binarized neural network (BNN), a lightweight convolutional neural network (CNN), with one dry-type sEMG sensor, which is implemented on a field-programmable gate array (FPGA). The proposed HGR system classifies nine dynamic gestures that can be useful in real life rather than static gestures that can be classified relatively easily. Raw sEMG data collected from a dynamic gesture are converted into a spectrogram with information in the time-frequency domain and transferred to the classifier. As a result, the proposed HGR system achieved 95.4% classification accuracy, with a computation time of 14.1 ms and a power consumption of 91.81 mW.

Best Research Papers in the Field of Sensors, Signals, and Imaging Informatics 2021

OBJECTIVES

In this synopsis, we identify and highlight research papers representing noteworthy developments in signals, sensors, and imaging informatics in 2021.

METHODS

A broad literature search was conducted on PubMed and Scopus databases. We combined Medical Subject Heading (MeSH) terms and keywords to construct particular queries for sensors, signals, and imaging informatics. Except for the sensor section, we only consider papers that have been published in journals providing at least three articles in the query response. Using a three-point Likert scale (1=not include, 2=maybe include, and 3=include), we reviewed the titles and abstracts of all database returns. Only those papers which reached two times three points were further considered for full paper review using the same Likert scale. Again, we only considered works with two times three points and provided these for external reviews. Based on the external reviews, we selected three best papers, as it happens that the three highest ranked papers represent works from all three parts of this section: sensors, signals, and imaging informatics.

RESULTS

The search for papers was executed in January 2022. After removing duplicates and conference proceedings, the query returned a set of 88, 376, and 871 papers for sensors, signals, and imaging informatics, respectively. For signals and images, we filtered out journals that had less than three papers in the query results, reducing the number of papers to 215 and 512, respectively. From this total of 815 papers, the section co-editors identified 35 candidate papers with two times three Likert points, from which nine candidate best papers were nominated after full paper assessment. At least three external reviewers then rated the remaining papers and the three best-ranked papers were selected using the composite rating of all external reviewers. By accident, these three papers represent each of the three fields of sensor, signal, and imaging informatics. They were approved by consensus of the International Medical Informatics Association (IMIA) Yearbook editorial board. Deep and machine learning techniques are still a dominant topic as well as concepts beyond the state-of-the-art.

CONCLUSIONS

Sensors, signals, and imaging informatics is a dynamic field of intense research. Current research focuses on creating and processing heterogeneous sensor data towards meaningful decision support in clinical settings.

Compound motion decoding based on sEMG consisting of gestures, wrist angles, and strength

This study aimed to highlight the demand for upper limb compound motion decoding to provide a more diversified and flexible operation for the electromyographic hand. In total, 60 compound motions were selected, which were combined with four gestures, five wrist angles, and three strength levels. Both deep learning methods and machine learning classifiers were compared to analyze the decoding performance. For deep learning, three structures and two ways of label encoding were assessed for their training processes and accuracies; for machine learning, 24 classifiers, seven features, and a combination of classifier chains were analyzed. Results show that for this relatively small sample multi-target surface electromyography (sEMG) classification, feature combination (mean absolute value, root mean square, variance, 4th-autoregressive coefficient, wavelength, zero crossings, and slope signal change) with Support Vector Machine (quadric kernel) outstood because of its high accuracy, short training process, less computation cost, and stability (p &lt; 0.05). The decoding result achieved an average test accuracy of 98.42 ± 1.71% with 150 ms sEMG. The average accuracy for separate gestures, wrist angles, and strength levels were 99.35 ± 0.67%, 99.34 ± 0.88%, and 99.04 ± 1.16%. Among all 60 motions, 58 showed a test accuracy greater than 95%, and one part was equal to 100%.

Toward Improving the Reliability of Discrete Movement Recognition of sEMG Signals

Currently, the classification accuracy of surface electromyography (sEMG) signals is high in literature, but the conventional recognition system may classify untrained movements or the trained movements of low reliability to one of its target classes by mistake. If such a system is used for prosthetic control, sometimes it may cause a disaster. A two-layer classifier that fuses the Gaussian mixture model (GMM) and k-nearest neighbor (kNN) in a sequential structure is proposed in this study. The proposed algorithm can reject the trained movements with low reliability and is efficient in rejecting the untrained movements, thus enhancing the reliability of the myoelectric control system. The results show that the proposed algorithm can produce 95.7% active accuracy in recognizing 12 trained movements and a 30.3% error rate for rejecting 12 untrained movements. When the movement number is six, the active accuracy for trained movements can reach 99.2%, and the error rate of untrained movement is only 17.4%, which is much better than previous studies. Therefore, the proposed classifier can accurately recognize the trained movements and reject untrained movement patterns effectively.

Towards Robust, Adaptive and Reliable Upper-limb Motion Estimation Using Machine Learning and Deep Learning--A Survey in Myoelectric Control

To develop multi-functional human-machine interfaces that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) and deep learning (DL) techniques have been widely implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, due to the high complexity of upper-limb movements and the inherent non-stable characteristics of sEMG, the usability of ML/DL based control schemes is still greatly limited in practical scenarios. To this end, tremendous efforts have been made to improve model robustness, adaptation, and reliability. In this article, we provide a systematic review on recent achievements, mainly from three categories: multi-modal sensing fusion to gain additional information of the user, transfer learning (TL) methods to eliminate domain shift impacts on estimation models, and post-processing approaches to obtain more reliable outcomes. Special attention is given to fusion strategies, deep TL frameworks, and confidence estimation. \textcolor{red}{Research challenges and emerging opportunities, with respect to hardware development, public resources, and decoding strategies, are also analysed to provide perspectives for future developments.

A Residual Movement Classification Based User Interface for Control of Assistive Devices by Persons with Complete Tetraplegia

OBJECTIVE

Complete tetraplegia can deprive a person of hand function. Assistive technologies may improve autonomy but needs for ergonomic interfaces for the user to pilot these devices still persist. Despite the paralysis of their arms, people with tetraplegia may retain residual shoulder movements. In this work we explored these movements as a mean to control assistive devices.

METHODS

We captured shoulder movement with a single inertial sensor and, by training a support vector machine based classifier, we decode such information into user intent.

RESULTS

The setup and training process take only a few minutes and so the classifiers can be user specific. We tested the algorithm with 10 able body and 2 spinal cord injury participants. The average classification accuracy was 80% and 84%, respectively.

CONCLUSION

The proposed algorithm is easy to set up, its operation is fully automated, and achieved results are on par with state-of-the-art systems.

SIGNIFICANCE

Assistive devices for persons without hand function present limitations in their user interfaces. Our work present a novel method to overcome some of these limitations by classifying user movement and decoding it into user intent, all with simple setup and training and no need for manual tuning. We demonstrate its feasibility with experiments with end users, including persons with complete tetraplegia without hand function.

No Interface, No Problem: Gesture Recognition on Physical Objects Using Radar Sensing

Physical objects are usually not designed with interaction capabilities to control digital content. Nevertheless, they provide an untapped source for interactions since every object could be used to control our digital lives. We call this the missing interface problem: Instead of embedding computational capacity into objects, we can simply detect users’ gestures on them. However, gesture detection on such unmodified objects has to date been limited in the spatial resolution and detection fidelity. To address this gap, we conducted research on micro-gesture detection on physical objects based on Google Soli’s radar sensor. We introduced two novel deep learning architectures to process range Doppler images, namely a three-dimensional convolutional neural network (Conv3D) and a spectrogram-based ConvNet. The results show that our architectures enable robust on-object gesture detection, achieving an accuracy of approximately 94% for a five-gesture set, surpassing previous state-of-the-art performance results by up to 39%. We also showed that the decibel (dB) Doppler range setting has a significant effect on system performance, as accuracy can vary up to 20% across the dB range. As a result, we provide guidelines on how to best calibrate the radar sensor.